Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sudo responder hangs #311

Closed
gschueler opened this issue Feb 22, 2013 · 14 comments
Closed

sudo responder hangs #311

gschueler opened this issue Feb 22, 2013 · 14 comments
Assignees
Labels
Milestone

Comments

@gschueler
Copy link
Member

using secondary sudo authentication can cause a hang in the output, and the job doesn't seem to finish

@ghost ghost assigned gschueler Feb 28, 2013
@ryanch
Copy link

ryanch commented Mar 11, 2013

I'm seeing this issue on 1.4.4-1.3, we are going to try on 1.5.

To repo this problem we did:
sudo ls -R /usr/bin << this was slow
ls -R /usr/bin << this was fast

@gschueler
Copy link
Member Author

this fix is not in 1.5 yet.

fyi: a potential workaround for 1.4 is to increase use the "sudo-response-max-lines" and "sudo-response-max-timeout" to some very large values, but this will only work if you are not using sudo2-command-enabled=true

@ryanch
Copy link

ryanch commented Mar 11, 2013

We just found another work around for 1.4- in our xml for our node we removed "sudo-command-enabled" and now it works without delay. We don't need that because we don't need a password for sudo.

Maybe this is another problem?

@gschueler
Copy link
Member Author

my guess is since you had enabled the sudo responder mechanism but don't use a password, the delay you were seeing is the timeout looking for the password prompt.

if you don't use a password for sudo then there is no need for the sudo response functionality

@gschueler
Copy link
Member Author

There were a few additional bugs in the thread code:

  1. Responder output stream needs to be closed when done, otherwise the SSH input reader thread can hang waiting for more output
  2. There is a race condition around the "disconnect" handler: if the sudo prompt timeout happens before the SSH connection is established, it can cause the execution to hang
  3. The ExecutorService should be shutdown after the Sudo logic is done, otherwise an extra thread lingers until the entire execution is done.

gschueler added a commit that referenced this issue Mar 16, 2013
Cleanup sudo threadpool after it is done.

Fix race condition when sudo input prompt timeout happens before
the SSH connection begins.

for #311
@connaryscott
Copy link

I am having trouble with a rundeck 1.5 upgrade that might be related to this....
I am getting an SSH hang using following sudo params:

sudo-command-enabled="true" sudo-fail-on-prompt-timeout="false"

which allows us to ssh in as one user and sudo to another and execute the defined rundeck command line.
This resource model setup was applied in 1.3.1 and am now attempting a rundeck 1.5 upgrade.

if the remote command produces lots of output, the session hangs. If I reduce the output to minimal, it does not hang and completes successfully. Perhaps there are additional sudo parameters that can resolve this? And if this is not related, i can file a new issue accordingly.

Thanks,

Chuck

@gschueler
Copy link
Member Author

yes, the only workaround right now is to increase the "sudo-response-max-lines" and "sudo-response-max-timeout" to very large values.

@connaryscott
Copy link

I updated my rundeck 1.5 GA rpm install with:

[cscott2@centgoldrk1v ~]$ rpm -q rundeck
rundeck-1.5.1-0.204.SNAPSHOT.noarch

from http://build.rundeck.org/job/rundeck-development/

and was able to resolve my ssh hang issue issue.

@gschueler
Copy link
Member Author

Another bug could cause NPE in sudo responder thread. won't stop a job that is already running, but could potentially prevent sudo password from response from working.

@andysteady
Copy link

With 1.5.1 GA by following the reproduce steps for this defect : #347 but doing it as sudo I have reproduced the hang.
With the snapshot 209 this does not happen.

@gschueler
Copy link
Member Author

@andysteady I'm not sure if that comment was meant for this issue or 347? Also, what do you mean by 1.5.1 GA (it hasn't been released yet)

@andysteady
Copy link

Hi,
Sorry I meant it does happen on 1.5 but it doesn't happen on 1.5.1 snapshot.
Originally I discussed rundeck hanging in this thread: https://groups.google.com/forum/?fromgroups=#!searchin/rundeck-discuss/observations/rundeck-discuss/yMhpDW8NnN8/o1BnGAlq0tkJ and said I would try to reproduce the case more reliably. This issue (311) seems to be very similar or the same, and I found I could reproduce 311 by using the steps in 347 modified to run as sudo.

Hope that makes sense.

Thanks,

Andy

@gschueler
Copy link
Member Author

Ok I see, so sounds like you are confirming the fix in 1.5.1 dev branch then?

@andysteady
Copy link

Yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants