Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process interactive commands and stream output in logs #2059

Closed
wants to merge 10 commits into from
54 changes: 41 additions & 13 deletions opendevin/runtime/docker/ssh_box.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from glob import glob

import docker
from pexpect import exceptions, pxssh
from pexpect import EOF, TIMEOUT, ExceptionPexpect, exceptions, pxssh
from tenacity import retry, stop_after_attempt, wait_fixed

from opendevin.core.config import config
Expand Down Expand Up @@ -45,7 +45,11 @@ def exit_code(self):
return -1

_exit_code = self.ssh.before.strip()
return int(_exit_code)
try:
return int(_exit_code)
except ValueError:
logger.error(f'Invalid exit code: {_exit_code}')
return -1
Comment on lines 47 to +52
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to check the error rather than use try catch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got Jupyter kernel ready in _exit_code and the program stopped when I set timeout to 3 for testing. There some bugs in that like #1915. Also, it is handled in https://github.com/OpenDevin/OpenDevin/blob/d8d17aae03c8c0987cbce625103cd15020fb05da/opendevin/runtime/plugins/mixin.py#L55-L58

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CC: @neubig, I am waiting for the reply.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agree with @yufansong that we should fix the underlying problem, rather than catching the error here.


def read_output(self):
st = time.time()
Expand Down Expand Up @@ -419,7 +423,7 @@ def _send_interrupt(
command_output += '\n' + self.ssh.before
return (
-1,
f'Command: "{cmd}" timed out. Sending SIGINT to the process: {command_output}',
f'Command: "{cmd}" timed out. Sent SIGINT to the process: {command_output}',
)

def execute(
Expand All @@ -441,11 +445,38 @@ def execute(
self.ssh.sendline(cmd)
if stream:
return 0, SSHExecCancellableStream(self.ssh, cmd, self.timeout)
success = self.ssh.prompt(timeout=timeout)
if not success:
logger.exception('Command timed out, killing process...', exc_info=False)
return self._send_interrupt(cmd)
command_output = self.ssh.before
prompts = [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we inherit from pxssh.pxssh, and override their .prompt method to implement this? It can make the code much more readable?

Copy link
Contributor Author

@SmartManoj SmartManoj Jun 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will we move to Paramiko for windows support?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the next week, I'm thinking about completely get rid of SSH, and spin up a websocket client inside sandbox, connect it to the backend, and do things from there -- this will allow us to use ONE event stream for everything which can be cleaner. I can start a draft PR and we can discuss from there i guess.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that would be better for connecting the frontend terminal.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we inherit from pxssh.pxssh, and override their .prompt method to implement this? It can make the code much more readable?

In which file, do you want this to be implemented?

Copy link
Contributor Author

@SmartManoj SmartManoj Jul 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still need to inherit as it is reaching the end of life?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so! We can leave it as is for now probably?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Copy link
Contributor Author

@SmartManoj SmartManoj Jul 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you reopen it?

r'Do you want to continue\? \[Y/n\]',
self.ssh.PROMPT,
EOF,
TIMEOUT,
]
command_output = ''
timeout_counter = 0
while True:
try:
# Wait for one of the prompts
index = self.ssh.expect(prompts, timeout=1)
line = self.ssh.before
logger.info(line)
command_output += line
if index == 0:
self.ssh.sendline('Y')
elif index == 1:
break
elif index == 2:
logger.debug('End of file')
break
elif index == 3:
timeout_counter += 1
if timeout_counter > timeout:
logger.exception(
'Command timed out, killing process...', exc_info=False
)
return self._send_interrupt(cmd)
except ExceptionPexpect as e:
logger.exception(f'Unexpected exception: {e}')
break
Comment on lines +473 to +504
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems brittle to me. Auto-confirm based on prompt text is kind of a hack

It looks like you have another PR which warns the agent about commands like this...will that suffice?

I'm generally concerned that this file is filling up with little hacks for the agent, instead of just being a thin wrapper around SSH.

Copy link
Contributor Author

@SmartManoj SmartManoj Jun 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean #2034 (I'm not the author)? We can warn them but they are not AGI models yet to follow that always. So need some little tweaks. Also helps weaker LLMs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is introducing several changes in the way that prompts are handled.

  1. Automatic entering of "Y" -- I think this is a bad idea... Usually a confirmation dialog is there because you need to check before doing something dangerous, and having the system automatically choose to proceed seems dangerous.
  2. This does nothing, so it can probably just be removed?
  3. Adding "end of file" may be good, but what was the behavior before?
  4. This seems like a good idea, and I actually like it better than the solution in fix: codeact bug [If running a command that never returns, it gets stuck #1895] #2034, which expands the prompt for something that can probably be done automatically like we do it here.


# once out, make sure that we have *every* output, we while loop until we get an empty output
while True:
Expand All @@ -462,7 +493,7 @@ def execute(
)
if isinstance(output, str) and output.strip() == '':
break
command_output += output
command_output += str(output)
command_output = command_output.removesuffix('\r\n')

# get the exit code
Expand Down Expand Up @@ -719,11 +750,9 @@ def close(self):
except Exception as e:
logger.exception('Failed to start Docker container: %s', e)
sys.exit(1)

logger.info(
"Interactive Docker container started. Type 'exit' or use Ctrl+C to exit."
)

# Initialize required plugins
ssh_box.init_plugins([AgentSkillsRequirement(), JupyterRequirement()])
logger.info(
Expand All @@ -733,7 +762,7 @@ def close(self):
)

bg_cmd = ssh_box.execute_in_background(
"while true; do echo -n '.' && sleep 10; done"
"while true; do echo -n '.' && sleep 20; done"
)

sys.stdout.flush()
Expand All @@ -753,7 +782,6 @@ def close(self):
continue
exit_code, output = ssh_box.execute(user_input)
logger.info('exit code: %d', exit_code)
logger.info(output)
if bg_cmd.pid in ssh_box.background_commands:
logs = ssh_box.read_logs(bg_cmd.pid)
logger.info('background logs: %s', logs)
Expand Down