Installable test: Can the CNF perform a rolling version upgrade?

[lixuna]

Acceptance Criteria

Installable test: Can the CNF perform a rolling version upgrade?

Short description:

• to check if a CNF version can be upgraded through a rolling update
• reference https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment

EPIC: CNF Conformance Test Suite

Release: vN.N.N

Test Category: Installable + Upgradeable

Type of test (static or runtime) = Runtime

• example code: Proof of concept: Test if the CNF can perform a rolling update #7

• similar to Scalability Test: Can CNF increase and decrease capacity? #24

---

Environment set up tasks:

Upstream tool set up tasks: (conformance suite + upstream tools)

• N/A

CNF setup Tasks

Sample CNF tasks:

Code implementation tasks:

• [ ]

Documentation tasks:

• Update Test Categories md if needed > crystal src/cnf-conformance.cr rolling_deployment_update
• Update Pseudo Code md if needed > crystal src/cnf-conformance.cr rolling_deployment_update
• Update USAGE md if needed > crystal src/cnf-conformance.cr rolling_deployment_update
• Update installation instructions if needed > crystal src/cnf-conformance.cr rolling_deployment_update

QA tasks

Dev Review:

• walk through A/C
• do you get the expected result?
• if yes,
  • move to Needs Peer Review column
  • create Pull Request and follow check list
  • Assign 1 or more people for peer review
• if no, document what additional tasks will be needed

Peer review:

• walk through A/C
• do you get the expected result?
• if yes,
  • move to Reviewer Approved column
  • Approve pull request
• if no,
  • document what did not go as expected, including error messages and screenshots (if possible)
  • Add comment to pull request
  • request changes to pull request

[lixuna]

Acceptance Criteria for peer review:

---

Common steps

• Find or create a K8s cluster to use
  • Steps for creating will be in another ticket
  • Talk with team for access to an existing cluster
• Set the KUBECONFIG environment variable for accessing the target K8s cluster
• As a developer, Pull down latest version of CNF conformance test code from develop branch
• As a peer reviewer, Pull down latest version of CNF conformance test code from master branch
• Run shard install
• Run crystal spec
• I expect to see PASSED for the tests

When using the test crystal src/cnf-conformance.cr rolling_update command the sample "good" CNF should PASS test successfully:

• Run crystal src/cnf-conformance.cr setup
• Run crystal src/cnf-conformance.cr sample_coredns_setup
• Run crystal src/cnf-conformance.cr rolling_update
• I expect to see the output and results from the test
• I expect to see PASSED for the test

N/A - a "bad" CNF should FAIL test:

• N/A - decision to defer "failing" test from this issue

When using the test crystal spec --tag 'rolling_update' command the sample "good" CNF should PASS and the bad version argument test should FAIL

• I expect to see the output and results from the test
• I expect to see "PASSED: CNF coredns-coredns Rolling Update Passed"
• I expect to see "FAILURE: CNF coredns-coredns Rolling Update Failed"

I would expect the CNF to pass when I run the entire test category suite (installability)

• Run crystal src/cnf-conformance.cr cleanup
• Run crystal src/cnf-conformance.cr setup
• Run crystal src/cnf-conformance.cr sample_coredns_setup
• Run crystal src/cnf-conformance.cr installability
• I expect to see the output and results from all tests in the test category
• I expect to see PASSED for the test category

I would expect the CNF to pass when I run the entire test suite (all)

• Follow common steps above
• Run crystal src/cnf-conformance.cr cleanup
• Run crystal src/cnf-conformance.cr setup
• Run crystal src/cnf-conformance.cr sample_coredns_setup
• Run crystal src/cnf-conformance.cr all
• I expect to see the output from the test
• I expect to see PASSED for the test

---

Steps after A/C fails

If all A/C fails for peer reviewer

• Add a comment to the PR summarizing the problem tagging the developer
• Attach any console output or logs that can help the developer fix the issue

Steps after A/C Passes

If all A/C passes for developer

• open a Pull Request to master branch and mark as "developer approved"
• Tag team members for peer review

If all A/C passes for peer reviewer

• Mark the PR as approved and merge the PR into master

[taylor]

Update of CNF will use Deployments as seen in https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment vs directly using a ReplicationController (https://kubernetes.io/docs/tasks/run-application/rolling-update-replication-controller/)

[lixuna]

Level of effort in pts: 0, 1, 2, 3, 5, 8, 13, 21 = 3 pts

@taylor 3 pts
@denverwilliams 3 pts
@lixuna 3 pts

[taylor]

See the Scalability Test, lfn-cnti/certification#24, for test which updates the Deployment.

[williscool]

merged into master re #120

[lixuna]

@williscool please check the tasks that have been completed and update the documentation for this new test

[taylor]

Testing with envoy and coredns

pair@cnfdev03:~/workspace/taylor/qa/cnf-conformance$ ./cnf-conformance rolling_update
error: timed out waiting for the condition
Waiting for deployment "envoy" rollout to finish: 1 old replicas are pending termination...
FAILURE: CNF envoy Rolling Update Failed

pair@cnfdev03:~/workspace/taylor/qa/cnf-conformance$ ./cnf-conformance rolling_update
Waiting for deployment "coredns-coredns" rollout to finish: 0 of 1 updated replicas are available...
deployment "coredns-coredns" successfully rolled out                                                                                    
PASSED: CNF coredns-coredns Rolling Update Passed  

[taylor]

A/C passes for envoy and coredns.

the error in the last comment is probably a timeout for the test.