Benchmark to resolve Real-world issues labelled with "good-first-issue".
On-boarding a new open-source repository/community is really difficult.
The common approach to get started is to go though small issues for new commers.
Maintainers tag easy-to-resolve issues with the following lavels; good-first-issue or help-wanted.
OSS bounty platforms(e.g. Quira.sh and OnlyDust.com) feature good-first-issues mostly.
However, the actual compretion ratio of resolving the issues and submissing PRs by new commers is less than 5%.
Most of the applicants who comminted to try resolving issues ghost without asking any questions.
Build a AI buddy to assist understing the code base, and suggest how to resolve the issues.
To generate the right guide for each issues, the following step is required.
- LLM knows the answer.
- LLM can break down the process of problem solving.
- LLM can generate a step-by-step guidance for each issues.
To raise the raio of PR completion to 50%.