Writing a JQF test

JQF tests are JUnit tests that take one or more arguments. JQF generates the values for these arguments many times, in what is called a fuzzing loop.

JQF builds on junit-quickcheck, so do checkout the junit-quickcheck documentation for tips on writing generators for complex types. Let's summarize some of the differences between junit-quickcheck and JQF:

• Test methods must be annotated with @Fuzz instead of @Property. The @Property annotation allowed setting several configuration parameters such as the number of trials or the random seed to use; in JQF, we defer the running of the fuzzing loop to a separately configurable front-end. For example, when using AFL as the front-end, the fuzzing loop runs infinitely unless the AFL process is killed by an external user.
• Test classes must be annotated with @RunWith(JQF.class) instead of @RunWith(JunitQuickcheck.class). However, class JQF extends JunitQuickcheck, so a JQF test class can have a mix of JQF @Fuzz targets, Quickcheck @Property tests and classic Junit @Test methods without any arguments.

All JQF tests can be part of your standard unit test suite. By default, JQF uses NoGuidance for 100 trials, which is basically the same as standard quickcheck (i.e. random values without any feedback), and therefore runs very fast. The same tests can later be run with feedback (such as AFLGuidance -- see fuzzing with AFL) if you have more time.

Adding Dependencies to JQF

You'll of course need to add JQF as a dependency to your test classes. You can either add that manually or using Maven.

Command-line

Use the handy script classpath.sh that expands to the JQF classpath needed to compile your tests.

# Assume you have a test class in `MyTest.java`
javac -cp $(jqf/scripts/classpath.sh) MyTest.java

Maven

If you mvn install JQF from the JQF directory then you'll have the snapshot in your local repository which you can now import in your custom project as follows:

    <dependencies>
        <dependency>
            <groupId>edu.berkeley.cs</groupId>
            <artifactId>jqf-fuzz</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>edu.berkeley.cs</groupId>
            <artifactId>jqf-instrument</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
    </dependencies>

We are still working on making a release to Maven central so that you won't have to install locally.

Sample tests

Look at some examples bundled with JQF for sample tests.

Let's walk through some examples here.

JDK DateTimeFormatter

@RunWith(JQF.class)
public class DateFormatterTest {
    @Fuzz
    public void fuzzLocalDateTime(String date, String pattern) throws IllegalArgumentException, DateTimeParseException {
        LocalDateTime.parse(date, DateTimeFormatter.ofPattern(pattern));
    }
}

JQF will generate the date and pattern automatically. Anything listed in the throws clause is assumed to be normal, i.e. we expect those exceptions to be thrown by our code, so they will not be marked as failures. Any other exceptions (e.g. ArrayIndexOutOfBoundsException or NullPointerException) will be marked as failures.

JDK ImageIO - PNG parsing

@RunWith(JQF.class)
public class PngReaderTest {
    @Fuzz
    public void read(ImageInputStream input) throws IOException {
        // Decode image from input stream
        reader.setInput(input);
        // Bound dimensions
        Assume.assumeTrue(reader.getHeight(0) < 1024);
        Assume.assumeTrue(reader.getWidth(0) < 1024);
        // Parse PNG
        reader.read(0);
    }
}

JQF generates the ImageInputStream automatically (using the registered ImageInputStreamGenerator).

If a generated input stream leads to a very large input, the assume statements will fail and JQF will discard those inputs. Any exceptions other than IOException thrown from the parser are considered failures.

Maven - ModelReader

@RunWith(JQF.class)
public class ModelReaderTest {
    @Fuzz
    public void testWithGenerator(@From(XmlDocumentGenerator.class) @Dictionary("dictionaries/maven-model.dict") Document dom) throws IOException {
        testWithSequence(XmlDocumentGenerator.documentToInputStream(dom));
    }
}

We explicitly tell JQF to use the XmlDocumentGenerator to generate XML DOM objects and then test Maven's pom.xml parsing routine. We also provide a dictionary file to the generator that contains many tag names scraped from the Maven Model description.