Coverity builds broken

[richardlau]

Two most recent node-daily-coverity builds have failed.
Some error about the agent going offline, no obvious other error.

e.g. https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/3010/console

make[1]: *** Deleting file '/home/iojs/build/workspace/node-daily-coverity/out/Release/obj.target/v8_compiler/deps/v8/src/compiler/graph.o'
make[1]: *** Deleting file '/home/iojs/build/workspace/node-daily-coverity/out/Release/obj.target/v8_compiler/deps/v8/src/compiler/graph-visualizer.o'
make[1]: *** Deleting file '/home/iojs/build/workspace/node-daily-coverity/out/Release/obj.target/v8_compiler/deps/v8/src/compiler/graph-reducer.o'
make[1]: *** Deleting file '/home/iojs/build/workspace/node-daily-coverity/out/Release/obj.target/v8_compiler/deps/v8/src/compiler/frame-states.o'
FATAL: Unable to delete script file /tmp/jenkins5048327983852336012.sh
java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:155)
	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:143)
	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:789)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@67405022:JNLP4-connect connection from 147.75.72.255/147.75.72.255:58626": Remote call on JNLP4-connect connection from 147.75.72.255/147.75.72.255:58626 failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:996)
	at hudson.FilePath.act(FilePath.java:1230)
	at hudson.FilePath.act(FilePath.java:1219)
	at hudson.FilePath.delete(FilePath.java:1766)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:163)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
	at hudson.model.Build$BuildExecution.build(Build.java:199)
	at hudson.model.Build$BuildExecution.doRun(Build.java:164)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:526)
	at hudson.model.Run.execute(Run.java:1895)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:442)
[Agent went offline during the build](https://ci.nodejs.org/computer/test%2Dequinix%2Dubuntu2204%2Dx64%2D1/log)
ERROR: Connection was broken

[targos]

When I installed the new Jenkins workspace, I downloaded a more recent version of Coverity that I deployed on the existing machines too (and I updated the Jenkins job).

This is probably the compilation going out of memory. I saw the same symptoms on Fedora hosts. The kernel kills the entire process tree when it happens, including the Jenkins agent.

[richardlau]

The Equinix machines (where the job was running) have more CPU/RAM (#3597 (comment)) than the IBM machine (#3597 (comment)) which I put back online a few hours ago.

The job is running

V=1 cov-build --dir cov-int make -j $(getconf _NPROCESSORS_ONLN)

which for the Equinix machine was 16 (which would appear to be 2 threads per each of the 8 cores). We could possibly set server_jobs in the inventory to a lower number, which should set the JOBS environment variable and then change the job to use JOBS. I'm a bit wary of touching test-equinix-ubuntu2204-x64-1 at the moment as it's the machine that also runs the binary temp git repository used in the fanned jobs and the other Equinix workspace machine is currently down (#3721).

The IBM machine is 2 vCPUs/4 GB RAM, which is more like the regular test machines -- maybe adding 2GB swap like we did for the test machines would be sufficient, although the job is tending to prefer running on test-equinix-ubuntu2204-x64-1.

Or maybe we can be more drastic and shift the job to the Hetzner benchmark machines? I forget if there's a reason these had to run on the jenkins-workspace machines other than having to have the Coverity build tool installed, which I've now automated in #3722.

[targos]

Here's a run with hardcoded -j 6: https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/3013/

[targos]

That build passed. I suggest to keep the hardcoded value until a better solution is implemented.
+1 on using the Hetzner machines.