Microservice test: Does the CNF have a reasonable startup time?

[lixuna]

Acceptance Criteria

Microservice test: Does the CNF have a reasonable startup time?

EPIC: CNF Conformance Test Suite

Release: alpha

Test Category: Microservice

Type of test (static or runtime) = ?

• example code: POC ISSUE_NUMBER

---

Environment set up tasks:

Upstream tool set up tasks: (conformance suite + upstream tools)

• N/A

CNF setup Tasks

Sample CNF tasks:

Code implementation tasks:

• [ ]

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update USAGE md if needed
• Update installation instructions if needed

QA tasks

Dev Review:

• walk through A/C
• do you get the expected result?
• if yes,
  • move to Needs Peer Review column
  • create Pull Request and follow check list
  • Assign 1 or more people for peer review
• if no, document what additional tasks will be needed

Peer review:

• walk through A/C
• do you get the expected result?
• if yes,
  • move to Reviewer Approved column
  • Approve pull request
• if no,
  • document what did not go as expected, including error messages and screenshots (if possible)
  • Add comment to pull request
  • request changes to pull request

[lixuna]

Acceptance Criteria for peer review:

--

Common steps

• Find or create a K8s cluster to use
  • Steps for creating will be in another ticket
• Talk with team for access to an existing cluster
• Set the KUBECONFIG environment variable for accessing the target K8s cluster
• As a developer, Pull down latest version of CNF conformance test code from develop branch
• As a peer reviewer, Pull down latest version of CNF conformance test code from master branch
• Run shards install
• Run crystal spec
• I expect to see PASSED for the tests

When using the test crystal src/cnf-conformance.cr reasonable_startup_time command the sample "good" CNF should PASS test successfully:

• Run crystal src/cnf-conformance.cr setup
• Run crystal src/cnf-conformance.cr sample_coredns_setup
• Run crystal src/cnf-conformance.cr reasonable_startup_time
• I expect to see the output and results from the test
• I expect to see PASSED for the test
• fast emoji from emoji

When using the test crystal spec --tag reasonable_startup_time -v command the sample "good" CNF should PASS and the "unreasonable startup" CNF example should FAIL test:

• Run crystal spec --tag reasonable_startup_time -v
• I expect to see the output and results from the test
• I expect to see "PASSED: Reasonable startup time"
• I expect to see "FAILURE: Unreasonable startup time"

I would expect the CNF to pass when I run the entire test category suite (microservice)

• Follow common steps above
• Run crystal src/cnf-conformance.cr setup
• Run crystal src/cnf-conformance.cr sample_coredns_setup
• Run crystal src/cnf-conformance.cr microservice
• I expect to see the output and results from all tests in the test category
• I expect to see PASSED for the test and test category
• fast emoji from emoji

I would expect the CNF to pass when I run the entire test suite (all)

• Follow common steps above
• Run crystal src/cnf-conformance.cr setup
• Run crystal src/cnf-conformance.cr sample_coredns_setup
• Run crystal src/cnf-conformance.cr all
• I expect to see the output from the test
• I expect to see PASSED for the test
• fast emoji from emoji

[lixuna]

Level of effort in pts: 0,1,2,3,5,8,13,21 pts: 3 pts

@taylor 3
@williscool 3
@denverwilliams 3
@wavell 5
@lixuna 3

[taylor]

Example emoji set for PASS / FAIL:

PASS:

• ✔️ CNF has a reasonable startup time ⏲️ 🚀
  FAIL:
• ❌ CNF does NOT have a reasonable startup time ⏲️ 🐢

Reference:

timer clocks:

• https://emojipedia.org/timer-clock/ ⏲️

Fast

• https://emojipedia.org/rocket/ 🚀
• https://emojipedia.org/rabbit/ 🐇
• https://emojipedia.org/dashing-away/ 💨

Slow

• https://emojipedia.org/turtle/ 🐢
• https://emojipedia.org/snail/ 🐌

[wvwatson]

crystal spec --tag reasonable_startup_time -v
Error: uninstall: Release not loaded: coredns: release: not found
Error: uninstall: Release not loaded: bad-helm-coredns-coredns: release: not found
Error: uninstall: Release not loaded: privileged-coredns: release: not found
Error: uninstall: Release not loaded: privileged-coredns: release: not found
Error: uninstall: Release not loaded: bad-liveness: release: not found
Error: uninstall: Release not loaded: coredns: release: not found
Error: uninstall: Release not loaded: coredns: release: not found
Microservice
'reasonable_startup_time' should pass if the cnf has a reasonable startup time(helm_directory)Error: uninstall: Release not loaded: coredns: release: not found
'reasonable_startup_time' should pass if the cnf has a reasonable startup time(helm_directory)
'reasonable_startup_time' should fail if the cnf doesn't has a reasonable startup time(helm_directory)

Finished in 2:04 minutes
2 examples, 0 failures, 0 errors, 0 pending

---

crystal src/cnf-conformance.cr microservice
PASSED: Image size is good
Error: cannot re-use a name that is still in use
NAME READY UP-TO-DATE AVAILABLE AGE
coredns-coredns 1/1 1 1 21s
dozing-turkey-coredns 1/1 1 1 8d
gatekeeper 1/1 1 1 78d
PASSED: CNF had a reasonable startup time 🚀

---

crystal src/cnf-conformance.cr all
Do you wish to overwrite the results.yml file? If so, your previous results.yml will be lost.
(Y/N) (Default N): > Y
PASSED: No privileged containers
PASSED: Replicas increased to 3
PASSED: Replicas decreased to 1
FAILURE: IP addresses found
PASSED: Helm liveness probe found
PASSED: Helm readiness probe found
PASSED (by default): No install script provided
Successfully created directories for cnf-conformance
PASSED: Helm Chart cnfs/coredns/helm_chart/coredns Lint Passed
PASSED: Published Helm Chart Repo added
PASSED: Image size is good
Error: cannot re-use a name that is still in use
NAME READY UP-TO-DATE AVAILABLE AGE
coredns-coredns 1/1 1 1 50s
dozing-turkey-coredns 1/1 1 1 8d
gatekeeper 1/1 1 1 78d
PASSED: CNF had a reasonable startup time 🚀
Final score: 59
Results have been saved to cnf-conformance-results-20200507-134110-409.yml

[lixuna]

@wavell this is a new test category. Has it been added to the documentation?

in the description:

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update USAGE md if needed
• Update installation instructions if needed