Create workflow on how to run OpenDevin on SWE-bench evaluation

[code2graph]

How to run OpenDevin with the SWE-Bench evaluation?

I don't understand how to capture the patch?
Also how to test the patch against the benchmark test suite?

Can we have this documentation?

[dorbanianas]

So basically right now the current SWE-BENCH in the repo is deprecated because it has a lot of issues but we are trying to make a new SWE-BENCH that would be in a separate repo

[dorbanianas]

if there's any other questions let me know 👌

[neubig]

This is really important to be able to evaluate different design decisions and changes that we make to OpenDevin. Hopefully @libowen2121 's work on setting up SWE-bench evaluation, and @mlejva 's implementation within the e2b sandbox will make a dent in this.

[SmartManoj]

In SWE-bench, generally (or for instance_id sympy__sympy-14774), how environment_setup_commit was chosen?

[libowen2121]

Hey guys, we have a private stabilized version of SWE-bench evaluation pipeline but it is now behind the official SWE-bench repo. We will push the changes to the forked SWE-bench repo and make sure it aligns with the official repo.

[neubig]

This is (mostly) finished, thanks @libowen2121 and @xingyaoww ! https://xwang.dev/blog/2024/opendevin-codeact-1.0-swebench/