[eval] add evaluation workflow

[xingyaoww]

End-user friendly description of the problem this fixes or functionality that this introduces

• Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

Add nightly evaluation workflow to monitor the quality of OpenHands during development

---

Give a summary of what the PR does, explaining any non-trivial design decisions

• Add nightly evaluation workflow to monitor the quality of OpenHands during development
• Fix some minor parsing issues with deepseek parsing on the new editing action.

---

Link of any specific issues this addresses

[github-actions]

Hi! I started running the evaluation on your PR. You will receive a comment with the results shortly.

[github-actions]

Evaluation outputs:

----------------------------------------------------------------------------------------------------
# of resolved: 2 / 10 (20.00%)
# of empty patch: 3 / 10 (30.00%)
# of error lines: 0 / 10 (0.00%)
# of loop: 3 / 10 (30.00%)
----------------------------------------------------------------------------------------------------
Detailed error breakdown:
Agent got stuck in a loop: 3 (30.00%)

You can download the full evaluation outputs here.

[github-actions]

Hi! I started running the evaluation on your PR. You will receive a comment with the results shortly.

[github-actions]

Trigger by: Pull Request (eval-this label on PR #4489)
SWE-Bench Evaluation Report

---

Integration Tests Evaluation Report
Success rate: 80.00% (4/5)

instance_id	success	reason
t03_jupyter_write_file	True
t01_fix_simple_typo	True
t05_simple_browsing	True
t04_git_staging	True
t02_add_bash_hello	False	Failed to execute /workspace/hello.sh: hello
		/workspace/hello.sh: line 3: /file_edit: No such file or directory
		/workspace/hello.sh: line 4: /file_edit: No such file or directory.

---

You can download the full evaluation outputs here.

[github-actions]

Hi! I started running the evaluation on your PR. You will receive a comment with the results shortly.

[github-actions]

Hi! I started running the evaluation on your PR. You will receive a comment with the results shortly.

[github-actions]

Trigger by: Pull Request (eval-this label on PR #4489)
SWE-Bench Evaluation Report

of resolved: 0 / 10 (0.00%)

of empty patch: 1 / 10 (10.00%)

of error lines: 0 / 10 (0.00%)

of loop: 6 / 10 (60.00%)

Avg. num of turns per instance: 31.90
Avg. agent cost per instance: 0.11 USD
Avg. editor cost per instance: 0.00 USD
Avg. total cost per instance: 0.11 USD

Detailed error breakdown:
Agent got stuck in a loop: 6 (60.00%)
Agent reached maximum iteration in headless mode, task stopped. Current iteration: 50.00, max iteration: 50.00: 1 (10.00%)

---

Integration Tests Evaluation Report
Success rate: 80.00% (4/5)

instance_id	success	reason
t03_jupyter_write_file	True
t01_fix_simple_typo	True
t04_git_staging	True
t05_simple_browsing	True
t02_add_bash_hello	False	Failed to execute /workspace/hello.sh: hello
		/workspace/hello.sh: line 3: /file_edit: No such file or directory
		/workspace/hello.sh: line 4: /file_edit: No such file or directory.

---

You can download the full evaluation outputs here.

[github-actions]

Hi! I started running the evaluation on your PR. You will receive a comment with the results shortly.

[github-actions]

Trigger by: Pull Request (eval-this label on PR #4489)
SWE-Bench Evaluation Report

of resolved: 0 / 10 (0.00%)

of empty patch: 0 / 10 (0.00%)

of error lines: 0 / 10 (0.00%)

of loop: 7 / 10 (70.00%)

Avg. num of turns per instance: 28.90
Avg. agent cost per instance: 0.10 USD
Avg. editor cost per instance: 0.00 USD
Avg. total cost per instance: 0.10 USD

Detailed error breakdown:
Agent got stuck in a loop: 7 (70.00%)
Agent reached maximum iteration in headless mode, task stopped. Current iteration: 50.00, max iteration: 50.00: 1 (10.00%)

---

Integration Tests Evaluation Report
Success rate: 80.00% (4/5)

instance_id	success	reason
t04_git_staging	True
t03_jupyter_write_file	True
t01_fix_simple_typo	True
t02_add_bash_hello	True
t05_simple_browsing	False	The answer is not found in any message. Total messages: 1. Messages: [MessageAction(content='Browse localhost:8000, and tell me the ultimate answer to life.', images_urls=None, wait_for_response=False, action='message', security_risk=None)]

---

You can download the full evaluation outputs here.

[github-actions]

Trigger by: Pull Request (eval-this label on PR #4489)
SWE-Bench Evaluation Report
Number of resolved: 1 / 10 (10.00%)
Number of empty patch: 0 / 10 (0.00%)
Number of error lines: 0 / 10 (0.00%)
Number of agent stuck in loop: 8 / 10 (80.00%)

Statistics

Avg. num of turns per instance: 33.30
Avg. agent cost per instance: 0.11 USD
Avg. editor cost per instance: 0.00 USD
Avg. total cost per instance: 0.11 USD

Detailed error breakdown:

Agent got stuck in a loop: 8 (80.00%)
Agent reached maximum iteration in headless mode, task stopped. Current iteration: 50.00, max iteration: 50.00: 1 (10.00%)

---

Integration Tests Evaluation Report
Success rate: 80.00% (4/5)

instance_id	success	reason
t03_jupyter_write_file	True
t02_add_bash_hello	True
t04_git_staging	True
t05_simple_browsing	False	The answer is not found in any message. Total messages: 1. Messages: [MessageAction(content='Browse localhost:8000, and tell me the ultimate answer to life.', images_urls=None, wait_for_response=False, action='message', security_risk=None)]
t01_fix_simple_typo	True

---

You can download the full evaluation outputs here.