-
Notifications
You must be signed in to change notification settings - Fork 9
How to write a reporter
PLEASE NOTE: Tap Out is still in early stages of development and the reporter API is likely to change some before all is said and done. Not really a big deal as the overall structure will remain the same. Just head up that you may need to spare a few minutes to update your custom reporter as we move toward a 1.0 release. Also note that this tutorial itself isn't quite complete yet either.
Writing a reporter is fairly straight forward. You are basically writing a listener class, base on an abstract base class that acts a kind of interface class to help you construct a reporter with some basic defaults and helper methods. It's a good idea to look at that code to get an idea of what you will be creating on top of it.
http://github.com/rubyworks/tapout/blob/master/lib/tapout/reporters/abstract.rb
Okay, so now lets start out by creating an empty subclass.
require 'tapout/reporters/abstract'
module TapOut
module Reporters
# My cool new TapOut reporter.
class MyReporter < Abstract
...
end
end
end
When the TapOut::Reporters::Abstract is inherited the subclass gets added to a list of reporter classes. This is how Tap Out tracks available reporters.
Now with out skeleton setup, lets add some methods to get it to actually do something. Let's start with the very first things that happens when a TAP-Y/J document stream is being consumed, we should receive a suite
entry. This entry is passed to the #start_suite
method. The methods is simply passed a single argument, a Hash object of the entry data. By looking at the specification for TAP-Y/J we see that the suite entry offers a few different fields. For the fun of it lest be very explicit about all of it.
# Handle header.
def start_suite(entry)
puts "There are #{entry['count']} tests to be tested!"
puts "The randomization seed is #{entry['seed']}." if entry['seed']
puts "Testing began at exactly #{entry['start']}."
puts "This document stream is using revision ##{entry['rev']} of the TAP-Y/J specification."
end
That wasn't so hard was it? Let's knock out the next type, the case
entry.
# At the start of a new test case.
def start_case(entry)
puts "This is a #{entry['subtype']} kind of test case."
puts "The case is called, '#{entry['label']}'."
puts "And the case is at level #{entry['level']}."
end
We might also get a note entry. We'll should output that as given.
# Handle an arbitray note.
def note(entry)
puts "NOTE: #{entry['text']}"
end
Just before a test is run, the #start_test method is called.
# This is run before the status handlers.
def start_test(entry)
end
After a test is run, one of five status handlers will be called depending on the result
of running the test. If the the test passed than the #pass
method is called.
# Handle test with pass status.
def pass(entry)
puts "Yeah! The #{entry['label']} test passed!"
super(entry)
end
Notice we called super
here. The abstract class will keep a running list of each test for each status which we can use later. So that can ave us the trouble of doing it ourselves. It also makes the #tally_message
helper method useful when we finally get to the end of the stream.
# Handle test with fail status.
def fail(entry)
super(entry)
end
# Handle test with error status.
def error(entry)
super(entry)
end
The next two status handlers are #omit
and #skip
. Now, you might wonder what the hell is the difference. It's fairly basic. A test is omitted by design. That is to say, there is a test but it was not actually run for some specific reason that the actual test runner determined. A good example might be if the test only applies to a particular platform.
# Handle test with omit status.
def omit(entry)
puts "The #{entry['label']} test was omitted."
super(entry)
end
Skip on the other hand is a sort of place holder. There are pending tests, that for one reason or another weren't fully executed because they are not finished being implemented, but ultimately they need to be.
# Handle test with skip or pending status.
def skip(entry)
puts "The #{entry['label']} test is pending completion."
super(entry)
end
After the status method is called, the #finish_test
method is called.
# This is run before the status handlers.
def finish_test(entry)
end
When a test case is completed the the #finish_case
method gets called. The entry argument here is the same as it was for #start_case
.
# When a test case is complete.
def finish_case(entry)
puts "The #{entry['label']} case is complete!"
end
Now to finish up completely when the stream runs out.
# Handle footer.
def finish_suite(entry)
puts "Finished running tests in #{entry['time']} seconds."
puts tally_message(entry)
end
Notice that we use the #tally_message
helper method to print out the total counts for each status. You can of course handle this yourself, but the helper is convenient for common usage.