Understanding the Test results

Let's assume you have created the following partial implementation of learntris in python, and saved it in a file called learntris.py:

# learntris.py version 0.1
pass

On windows, you can now test it with the command:

c:\learntris> c:\python27\python testris.py c:\python.exe learntris.py

It should produce the following message. (You will probably need to scroll up in the console to see it all, or pipe the results through a paging program such as less or more):

Running test 1: io.q
q : quit
---- sending commands ----
q
---- awaiting results ----
Test 1 passed


Running test 2: io.p
p : print
---- sending commands ----
p
q
---- awaiting results ----
: The 'p' command instructs learntris to print the state
: of the Matrix, the rectangular array of cells in which
: blocks can appear.
:
: The Matrix is 10 cells wide and 22 cells deep, although
: the top two rows are used only for spawning new Tetraminos.
:
: At the start of the game, the Matrix should be empty.
: The 'p' command should indicate empty cells with the
: '.' character.
:
: Cells should be separated by spaces.
:
: Lines should be separated by the standard end of line
: sequence on your operating system (python's "\n").
---- expected results ----
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
Test 2 failed: output mismatch:
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .

Let's look through these results section by section:

Running test 1: io.q
q : quit
---- sending commands ----
q
---- awaiting results ----
Test 1 passed

The first two lines indicate the internal name for test 1 (io.q) and a human friendly title. (This one is indicating that 'q' is the command to quit the program.)

The ---- sending commands ---- section demonstrates the input that testris.py will send. This one sends the command 'q' on a line by itself.

Since there's no expected output and our implementation produces no output, the test passes.

Running test 2: io.p
p : print
---- sending commands ----
p
q
---- awaiting results ----

This says that the second test is called io.p, testing the p command, which means "print". The input sent to learntris.py will be letter p on one line, followed by the letter q on the next line.

: The 'p' command instructs learntris to print the state
: of the Matrix, the rectangular array of cells in which
: blocks can appear.
:
: The Matrix is 10 cells wide and 22 cells deep, although
: the top two rows are used only for spawning new Tetraminos.
:
: At the start of the game, the Matrix should be empty.
: The 'p' command should indicate empty cells with the
: '.' character.
:
: Cells should be separated by spaces.
:
: Lines should be separated by the standard end of line
: sequence on your operating system (python's "\n").

This section just describes what the test is asking for, in more detail. Not all tests have explanations like this yet. All of these explanations, and the tests themselves, are stored in a text file file called testplan.org You can open this file in any text editor, but if you happen to use emacs, you can open the file in org-mode and browse or edit it as an outline (folding tree structure).

---- expected results ----
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .

This section demonstrates the exact output that testris.py is expecting your implementation to produce. Since it's expecting output, and our implementation so far produces no output, the test is obviously going to fail.

Test 2 failed: output mismatch:
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .
+ . . . . . . . . . .

This section shows the result of running a diff algorithm to compare the actual output with the expected output. You can think of it as a list of instructions telling you what you need to change: the plus signs at the start of the lines indicate that you need to add the corresponding lines.

Responding to failing tests

Suppose we change our implementation so that it produces some output:

# learntris.py version 0.2
print 'hello world!'

Now when we run the tests, we get a different result:

Running test 1: io.q
q : quit
---- sending commands ----
q
---- awaiting results ----
: The 'q' command instructs learntris to quit.
:
: Learntris should not produce any output unless
: explicitly instructed to do so.
---- expected results ----

Test 1 failed: output mismatch:
- hello world

You can see that the test for the q command now has a description: the description is only shown for the first failing test, and the test runner immediately stops running the test suite so that it can show you the error. Unlike unit testing, where each test is independent and the entire suite always runs, these tests are designed to run in sequence.

The results of each test are still independent though, because testris.py launches a new instance of the implementation for each test. (Of course if you stored some kind of persistent state in an external storage system between runs, you would have to take care that it didn't interfere with the results.)

Anyway, this time around, the problem isn't that we're missing lines, but that we have too many. The report here is saying that the expected output should be empty, but that it saw the line hello world. The minus sign indicates that this line should be removed.

Getting the tests to pass

Learntris is not a tutorial, but a challenge. This page was intended to help you run the tests and understand the feedback they produce, but the rest is up to you.

So now, your task is to figure out how to produce the desired output to pass the second test, without causing the first test to fail in the process.

If you're even a moderately experienced programmer, this should be rather easy to accomplish.

If you're a complete beginner and find this daunting, you may want to put learntris on hold for a bit and come back once you've learned the basics. Alternatively, you could use it as a way to direct your own personal study of programming.

Either way, good luck, and remember if you get stuck, you can always come visit us in the #learnprogramming channel on irc.freenode.net , or at http://reddit.com/r/learnprogramming/