Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jenkins java.io.IOException: remote file operation failed #173

Closed
orangemocha opened this issue Sep 1, 2015 · 22 comments
Closed

Jenkins java.io.IOException: remote file operation failed #173

orangemocha opened this issue Sep 1, 2015 · 22 comments
Labels
ci-public

Comments

@orangemocha
Copy link

This is an intermittent failure that I have seen in Jenkins for a long time. One instance here: https://ci.nodejs.org/job/node-test-commit-arm/nodes=pi1-raspbian-wheezy-1_of_2/407/console

Started by upstream project "node-test-commit-arm" build number 407
originally caused by:
 Started by upstream project "node-test-commit" build number 402
 originally caused by:
  Started by upstream project "node-test-pull-request" build number 214
  originally caused by:
   Started by user Jeremiah Senkpiel
[EnvInject] - Loading node environment variables.
Building remotely on iojs-nodesource-raspbian-wheezy-pi1p-4 (pi1-raspbian-wheezy pi1-raspbian-wheezy-2_of_2 pi1-raspbian-wheezy-1_of_2) in workspace /home/iojs/build/workspace/node-test-commit-arm/nodes/pi1-raspbian-wheezy-1_of_2
java.io.IOException: remote file operation failed: /home/iojs/build/workspace/node-test-commit-arm/nodes/pi1-raspbian-wheezy-1_of_2 at hudson.remoting.Channel@c92e156:iojs-nodesource-raspbian-wheezy-pi1p-4: java.io.IOException: Remote call on iojs-nodesource-raspbian-wheezy-pi1p-4 failed
    at hudson.FilePath.act(FilePath.java:987)
    at hudson.FilePath.act(FilePath.java:969)
    at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:131)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:705)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:697)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1037)
    at hudson.scm.SCM.checkout(SCM.java:485)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1277)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
    at hudson.model.Run.execute(Run.java:1741)
    at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:408)
Caused by: java.io.IOException: Remote call on iojs-nodesource-raspbian-wheezy-pi1p-4 failed
    at hudson.remoting.Channel.call(Channel.java:786)
    at hudson.FilePath.act(FilePath.java:980)
    ... 14 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy9
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at java.lang.reflect.Proxy.newInstance(Proxy.java:764)
    at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:755)
    at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:111)
    at hudson.remoting.Channel.export(Channel.java:593)
    at hudson.remoting.Channel.export(Channel.java:562)
    at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:201)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1075)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at hudson.remoting.UserRequest._serialize(UserRequest.java:158)
    at hudson.remoting.UserRequest.serialize(UserRequest.java:167)
    at hudson.remoting.UserRequest.perform(UserRequest.java:129)
    at hudson.remoting.UserRequest.perform(UserRequest.java:49)
    at hudson.remoting.Request$2.run(Request.java:325)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at hudson.remoting.Engine$1$1.run(Engine.java:69)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to iojs-nodesource-raspbian-wheezy-pi1p-4(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
    at hudson.remoting.Channel.call(Channel.java:778)
    ... 15 more
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: test.tap
Did not find any matching files. Setting build result to FAILURE.
Build step 'Publish TAP Results' marked build as failure
Checking ^not ok
Jenkins Text Finder: File set 'test.tap' is empty
Notifying upstream projects of job completion
Finished: FAILURE
@orangemocha
Copy link
Author

Could potentially be addressed by restarting Jenkins, like #169

@orangemocha
Copy link
Author

@nodejs/jenkins-admins

@rvagg
Copy link
Member

rvagg commented Sep 1, 2015

happening a lot recently, perhaps we need to update slave.jar

@orangemocha orangemocha mentioned this issue Sep 1, 2015
Closed
@jbergstroem
Copy link
Member

Haven't seen these in a good while. Fwiw, we've updated slave.jar's throughout our fleet but is likely unrleated.

@rvagg rvagg closed this as completed Nov 26, 2015
@jbergstroem
Copy link
Member

This just occured again (spotted by @Trott): https://ci.nodejs.org/job/node-test-binary-arm/159/RUN_SUBSET=1,nodes=pi2-raspbian-wheezy/console

@Trott
Copy link
Member

Trott commented Nov 28, 2015

And again:

Should this be re-opened?

@jbergstroem jbergstroem reopened this Nov 28, 2015
@Trott
Copy link
Member

Trott commented Nov 29, 2015

Is it useful to keep reporting these here as they occur or is that just noise?

@rvagg
Copy link
Member

rvagg commented Nov 30, 2015

interesting they are just pi2-9 and pi2-2 so far

@Trott feel free to keep on reporting if they keep showing up, a pattern may be interesting

@jbergstroem
Copy link
Member

@Trott - seen anything of this recently?

@Trott
Copy link
Member

Trott commented Feb 2, 2016

Haven't noticed it lately, but I haven't been looking for it either.

@jbergstroem
Copy link
Member

Speaking of the devil: https://ci.nodejs.org/job/node-test-binary-arm/RUN_SUBSET=addons,nodes=pi2-raspbian-wheezy/902/console

Edit: again just on ARM it seems. Are these perhaps trying to all pull at the same time?

@jpsim
Copy link

jpsim commented Sep 8, 2016

Sorry to bump into this thread, this is a last resort... was this ever resolved? If so how?

I'm seeing the same exception stack trace in my Jenkins jobs, though running in OS X, so not ARM:

java.io.IOException: remote file operation failed: /Users/realm/workspace/objc_pr at hudson.remoting.Channel@4ec66dd4:sf_host1_3: java.io.IOException: Remote call on sf_host1_3 failed
    at hudson.FilePath.act(FilePath.java:1014)
    at hudson.FilePath.act(FilePath.java:996)
    at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:131)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:741)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:733)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1080)
    at hudson.scm.SCM.checkout(SCM.java:485)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
    at hudson.model.Run.execute(Run.java:1741)
    at hudson.matrix.MatrixBuild.run(MatrixBuild.java:301)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOException: Remote call on sf_host1_3 failed
    at hudson.remoting.Channel.call(Channel.java:789)
    at hudson.FilePath.act(FilePath.java:1007)
    ... 14 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy11
    at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
    at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:143)
    at hudson.remoting.Channel.export(Channel.java:621)
    at hudson.remoting.Channel.export(Channel.java:590)
    at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:201)
    at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1118)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at hudson.remoting.UserRequest._serialize(UserRequest.java:157)
    at hudson.remoting.UserRequest.serialize(UserRequest.java:166)
    at hudson.remoting.UserRequest.perform(UserRequest.java:128)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at hudson.remoting.Engine$1$1.run(Engine.java:62)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to sf_host1_3(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
    at hudson.remoting.Channel.call(Channel.java:781)
    ... 15 more

@jbergstroem
Copy link
Member

Haven't seen these in a long while, but others; "remote host closed connection"

@huzhijiang
Copy link

huzhijiang commented Nov 2, 2016 •
edited by gibfahn
Loading

Again:

Wiping out workspace first.
java.io.IOException: remote file operation failed: /home/jenkins-ci/opnfv/slave_root/workspace/daisy4nfv-verify-build-master at hudson.remoting.Channel@61f2a4c1:lf-build1: java.nio.file.AccessDeniedException: /home/jenkins-ci/opnfv/slave_root/workspace/daisy4nfv-verify-build-master/daisycloud-core/.git/refs/heads/master
    at hudson.FilePath.act(FilePath.java:986)
    at hudson.FilePath.act(FilePath.java:968)
    at hudson.FilePath.deleteContents(FilePath.java:1183)
    at hudson.plugins.git.extensions.impl.WipeWorkspace.beforeCheckout(WipeWorkspace.java:28)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1013)
    at hudson.scm.SCM.checkout(SCM.java:485)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
    at hudson.model.Run.execute(Run.java:1738)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:410)
Caused by: java.nio.file.AccessDeniedException: /home/jenkins-ci/opnfv/slave_root/workspace/daisy4nfv-verify-build-master/daisycloud-core/.git/refs/heads/master
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1126)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at hudson.Util.deleteFile(Util.java:255)
    at hudson.FilePath.deleteRecursive(FilePath.java:1203)
    at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212)
    at hudson.FilePath.deleteRecursive(FilePath.java:1194)
    at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212)
    at hudson.FilePath.deleteRecursive(FilePath.java:1194)
    at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212)
    at hudson.FilePath.deleteRecursive(FilePath.java:1194)
    at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212)
    at hudson.FilePath.deleteRecursive(FilePath.java:1194)
    at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212)
    at hudson.FilePath.access$1100(FilePath.java:190)
    at hudson.FilePath$15.invoke(FilePath.java:1186)
    at hudson.FilePath$15.invoke(FilePath.java:1183)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
    at hudson.remoting.UserRequest.perform(UserRequest.java:120)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at hudson.remoting.Engine$1$1.run(Engine.java:62)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to lf-build1(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
    at hudson.remoting.Channel.call(Channel.java:781)
    at hudson.FilePath.act(FilePath.java:979)
    ... 13 more
Finished: FAILURE

@joaocgreis joaocgreis mentioned this issue Apr 1, 2017
Merged
2 tasks
@chr0n1x
Copy link

chr0n1x commented Jun 22, 2017

bump

Did you guys ever figure out what was going on here? Any Jenkins JIRA tickets?

@gibfahn
Copy link
Member

gibfahn commented Jun 22, 2017

@chr0n1x afraid not, there's also not much we can do about this, you'd need to raise (or find) an issue in Jenkins.

@chr0n1x
Copy link

chr0n1x commented Jun 22, 2017 •
edited
Loading

@gibfahn one thing that I tried was this. I seems to have helped with what I was seeing on my instance. In my case it was many many many slaves spinning up and the connections to them not terminating fast enough, getting stuck in the TIME_WAIT state and causing further connections that the master attempted to stall. Full resource that I read up on for TIME_WAIT here.

@mmarchini mmarchini mentioned this issue Nov 6, 2017
Closed
2 tasks
@maclover7
Copy link
Contributor

Has this happened recently? Otherwise I think this can be closed

@maclover7
Copy link
Contributor

Closing due to inactivity. Please reopen if this is still needed :)

@refack
Copy link
Contributor

refack commented Apr 9, 2018

I've found a possible workaround — restarting the java worker (slave.jar).
It just solved this issue for me on two workers, so it's worth giving it a shot.

@refack
Copy link
Contributor

refack commented Apr 9, 2018

A bit more info (for posterity). Looking at these seemingly key lines from the most recent trace:

java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy11
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
  1. The class failing is com.sun.proxy.$Proxy11 while in the OP it's $Proxy9, and in other traces, here and on the internet, the class numarator continues to vary.
  2. Seeing as this looks like code that auto-generates a remoting proxy by reflection, it seems likely that code version mismatch is the culprit, hence updating the client (or even restarting it) could solve this issue.

@refack refack mentioned this issue Apr 9, 2018
Closed
@rvagg
Copy link
Member

rvagg commented Apr 10, 2018

I had to take one of the smartos16 machines offline yesterday because of this, every build failed the same. No matter what I did on the node fixed the problem, including updating slave.jar. I got nuttin'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ci-public
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@refack @jbergstroem @jpsim @rvagg @Trott @chr0n1x @orangemocha @maclover7 @huzhijiang @gibfahn