Reporting results with NAB

NAB is designed as a community tool for testing anomaly detection algorithms on streaming data. As such we welcome any researcher to test other algorithms using the NAB dataset and report their results.

In order for a researcher to report NAB results on an algorithm, it must meet the following additional criteria:

• Any sort of look ahead is not allowed. Detections by the algorithm must be made on the fly, one record at a time.

• There must be no batch, or per data file, manual parameter tuning. The algorithm must be fully automated with a single set of parameters across all data files. Any further parameter tuning required by the algorithm must be done on the fly. For example, it is not possible to look through a dataset, set a threshold, and then run through the data again. (One exception to this rule is that we allow the algorithm to know the range of data values for each data file. In the NAB detector code, the min/max value of each data file is passed in automatically when each algorithm is initialized.)

• Each file is treated independently. The algorithm is initialized from scratch for each stream using a single constant set of parameters.

• Although you don't have to use NAB code to run your algorithm, you must use the NAB scoring code to compute the final scores. (If this is difficult to do please let us know.)

These rules are designed to simulate real-world real-time streaming scenarios. It is easy to "game" the system without following the above rules. Anyone reporting NAB results needs to convince the reader that the above criteria are met. A great way is to release your code as open source, or by documenting your process sufficiently so that someone else can replicate it. If you fail to do so, we will not include your algorithm's scores on the official NAB leaderboard.

Please cite the ICMLA paper if you report your own NAB results in a paper or presentation. Please also send an email (ideally with PDF attached) to nab@numenta.org as we would appreciate hearing about it.

Please also see [NAB Entry Points](NAB Entry Points) for instructions on how to run your algorithms on NAB.

Listing your results on the NAB leaderboard

Results that meet the above criteria will be listed on the NAB leaderboard. If a NAB user wants to have the results of their NAB runs posted in the NAB repository along with the other algorithms, the user should send an email to nab@numenta.org, attaching their final score results file, their name and/or affiliation, a link to the algorithm source code, and any other relevant information. A link or citation to any publication would be appreciated and will be included on the leaderboard.