Projects with injected bugs

The challenge binaries, valid test inputs, and sample vulnerabilities create an industry standard benchmark suite for evaluating:

• Bug-finding tools
• Program-analysis tools (e.g. automated test coverage generation, value range analysis)
• Patching strategies
• Exploit mitigations
• Learning software testing techniques

Software projects with injected defects:

• C/C++ FuzzBench https://google.github.io/fuzzbench/
• C/C++ Memory sanitizer benchmark
• C/C++, Java Juliet Test Suite
• Java droixbench is a benchmark that contains 24 reproducible crashes in open source Android apps.
• Java Defects4J is a benchmark of 341 Java bugs from 5 open-source projects. It contains the corresponding patches, which cover a variety of patch type.
• Java Bears is a benchmark of continuous integration build failures focusing on test failures.
• Java Bugs.jar — a large-scale, diverse dataset of bugs for Java program repair.
• Java REST service (https://martinfowler.com/articles/practical-test-pyramid.html)
• Java Juliet Test Suite for Java
• Java tarpit-java - a web application seeded with vulnerabilities, rootkits, backdoors and data leaks.
• Java Web EasyBuggy is a broken web application in order to understand behavior of bugs and vulnerabilities, for example, memory leak, deadlock, JVM crash, SQL injection and so on.
• Python tarpit-python - a web application seeded with vulnerabilities, rootkits, backdoors and data leaks.
• C C++ IntroClass — automated program repair benchmark that consists of 998 defects in small student-written programming assignments.
• C C++ DBGBench — 291 (in)correct patches from real software professionals for 27 real bugs in C for the qualitative evaluation of automated repair techniques.
• C C++ Codeflaws — 3902 bugs from Codeforces programming competition for evaluating program repair tools across different defect classes.
• C Space
• C "Siemens" programs were assembled by Tom Ostrand and colleagues at Siemens Corporate Research for a study of the fault detection capabilities of control-flow and data-flow coverage criteria [Hutchins94], and were made available to us by Tom Ostrand.
• C C++ ManyBugs is a benchmark of 185 C bugs in nine open-source programs.
• C tarpit-c - a set of C/C++ code snippets seeded with vulnerable conditions.
• C Program Bug Examples
• Web ParkCalc - Agile Testing Challenge
• Multilingual BugSwarm — a dataset of thousands of real software bugs and their fixes.
• Multilingual QuixBugs — a parallel corpus of 40 programs in both Python and Java, each with a bug on one line.
• DARPA Challenge Binaries are custom-made programs specifically designed to contain vulnerabilities that represent a wide variety of crashing software flaws.
• DARPA Cyber Grand Challenge Sample Challenges - What is the Cyber Grand Challenge?

---

• BugZoo - is a decentralised platform for distributing, reproducing, and interacting with historical software bugs.
• https://samate.nist.gov/SARD/testsuite.php
• Software-artifact Infrastructure Repository