Test Infrastructure

Test Environment Provisioning

• https://github.com/DSpeichert/netbootd
• Pixie
• Waitron (see https://jhaals.se/2015/11/14/replacing-foreman-with-waitron.html)
• Foreman
• FAI
• Cobbler, Terraform + Cobbler
• OpenStack Ironic
• bacio
• iPXE https://gist.github.com/robinsmidsrod/2234639
• Stamper
• cloud-init

Power Management

• pxe-pilot
• pdu-gateway
• pdudaemon

Performance Evaluation Services

Tool	Known users	Resources
ResultsDB	Fedora Linux (Red Hat)	Demo
Test Report Center	Intel	Demo, Documentation
Perfomatic	Mozilla	-
Perfherder	Mozilla	Demo
Treeherder	Mozilla	Demo, Documentation
Codespeed	Python Community	Demo
JMeter-Control-Center

Test Case Management Systems

• Nitrate
• testcasedb
• Tarantula
• Kiwi TCMS
• testicase

Test Results Systems

Tool	Known users	Resources
Subunit2SQL	OpenStack	Формат: SubUnit. Видео, слайды, Using SubUnit2SQL with the gate
Badger	2ГИС	Форматы: JUnit. Видео: CodeFest, SQAdays
Allure	Яндекс и другие	Форматы: JUnit. Расширение для Jenkins CI.
Tapper	Амазон	Формат: TAP (Test Anything Protocol). Разработка приостановлена.
Jenkins: Test Results Analyzer Plugin	?	Форматы: JUnit, TAP (Test Anything Protocol). Расширение для Jenkins CI.
testrepository	?	Форматы: SubUnit.
Report Portal	EPAM
Treeherder	Mozilla	Demo, Документация
ResultsDB	Fedora Linux (Red Hat)	Demo
Test Report Center	Intel	Demo
SQUAD	Linaro	Demo
Taskotron	Fedora Project, RedHat	Demo
SystemTestPortal
QA Reports	Linaro	Demo
Avocado Server	RedHat
app--prove--history	?
TAP-DOM-Db	?

Code coverage tools

• https://github.com/uber/uberalls
• Compare Coverage https://github.com/zachriggle/covpare
• coverage of patch - dgcov (source)
• coverage2sql
• https://github.com/gabrielcsapo/lcov-server
• https://github.com/tuvistavie/opencov
• https://github.com/ddunbar/zcov
• https://github.com/Telmo/thong
• https://www.sonarqube.org/
• coverage-report-server.py
• https://git.esheavyindustries.com/esell/hoptocopter
• https://pypi.python.org/pypi/pycobertura
• https://github.com/xing/hardcover
• parallel lcov https://github.com/mozilla/grcov
• https://github.com/Telmo/thong
• https://github.com/mavak/trucov
• (!) https://github.com/xaizek/uncov

Test Runners

• https://cmake.org/cmake/help/latest/manual/ctest.1.html
• https://github.com/ligurio/openbsd-tests/tree/master/tools/runtest
• https://perldoc.perl.org/prove.html
• https://github.com/exclave/exclave
• https://git.yoctoproject.org/cgit.cgi/patchtest/
• https://git.yoctoproject.org/cgit.cgi/ptest-runner2/
• https://github.com/labgrid-project/labgrid

Distributed Bug Tracking

• https://sny.no/2014/04/dbts
• Mark Shuttleworth http://www.markshuttleworth.com/archives/145
• LWN https://lwn.net/Articles/281849/
• http://tychoish.com/post/supporting-distributed-bug-tracking/
• https://erlangish.blogspot.ru/2007/06/distributed-bug-tracking-again.html
• http://ericsink.com/entries/dbts_fossil.html
• https://blog.liw.fi/posts/distributed-bug-tracking/
• https://www.youtube.com/watch?v=ysgMlGHtDMo
• Fossil SCM - sqlite
• Artemis
• https://dist-bugs.branchable.com/

Tools:

• List http://www.cs.unb.ca/~bremner/blog/posts/git-issue-trackers/
• List https://dist-bugs.branchable.com/software/
• Over VCS
  • https://github.com/droundy/entomonitor
  • http://docs.bugseverywhere.org/master/tutorial.html#introduction
  • https://github.com/jwiegley/git-issues/ (Git)
  • https://github.com/schacon/ticgit/wiki (unmaintained, and subsequently deprecated)
  • http://www.ditrack.org/ (Subversion)
  • SD http://syncwith.us/sd/ (LWN review and https://blog.liw.fi/posts/experiences-with-sd/)
• Hg https://www.mercurial-scm.org/wiki/BugtrackerExtension
• Fossil SCM https://www.fossil-scm.org/xfer/doc/trunk/www/bugtheory.wiki
• https://github.com/driusan/bug

Logs Processing

• Improving Log-Based Fault Diagnosis by Log Classification
• openstack
  • https://github.com/openstack-infra/puppet-log_processor/blob/master/files/classify-log.crm
  • https://github.com/openstack-infra/elastic-recheck
  • Openstack Elastic-recheck: Powered by the Elk Stack
  • https://docs.openstack.org/infra/elastic-recheck/readme.html
  • https://github.com/mtreinish/ciml
  • https://github.com/afrittoli/ciml_talk/blob/master/ciml.pdf
  • https://docs.openstack.org/infra/publications/processing-ci-log-events/
  • http://crm114.sourceforge.net/ (openstack)
• bmf?
• https://github.com/mcholste/elsa
• http://osbf-lua.luaforge.net/
• logreduce
  • https://opensource.com/article/18/9/quiet-log-noise-python-and-machine-learning
  • https://blogs.rdoproject.org/2017/11/anomaly-detection-in-ci-logs/
  • logreduce
  • https://github.com/TristanCacqueray/presentation-berlin-log-classify/blob/gh-pages/presentation.md
  • https://www.cio.com/article/2382644/big-data/anomaly-detection-lets-you-find-patterns-in-log-data.html
• cibuglog https://intel-gfx-ci.01.org/cibuglog/

logreduce uses a model to learn successful logs and detect novelties in failed logs:

Random words are manually removed using regular expression Then lines are converted to a matrix of token occurrences (using HashingVectorizer), An unsupervised learner implements neighbor searches (using NearestNeighbors).