Figure out an end-to-end testing approach

[hbagdi]

decK v0.5.0 had a very simple bug that rendered the release unusable.
This could have been easily could with an automated end-to-end testing setup, which we don't have at the moment.
As more changes are introduced, manual testing is becoming more expesive and proving not to be sufficient.
Figure out a path forward for how do we want to test decK in an automated fashion.

[teunis90]

Going over your open tickets, I saw this one.

What about creating a full-featured kong.yaml file, provision Kong:

• Dump the kong file, and diff
• Call all HTTP endpoints manually.

That would not cover everything but would be a good start.

[hbagdi]

That will cover part of it but not all. We need to test that decK is smart enough to handle cases where it can detect differences and out of sync cases, do reverse syncs too.

If you wish to contribute what you are suggesting, that would be a good addition. PR welcome!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hbagdi]

Some thoughts/questions?

• What do we test and what should take priority? There are numerous things that we need to include tests for:
  • Diffing logic: (mainly the diff package) This involves giving the diff engine two states and let decK traverse those states and output diff events. Verification involves ensure that the events generated are correct.
  • e2e: Given an state file, run decK and see what it outputs. This verifies if decK correct generates the right output on stdout. The verification trusts decK to do the right thing in this case if the text on stdout is right.
    • Something to add here would be to make API calls against Kong to ensure that the state that decK assumed that it successfully created actually matches the reality. This is meant to detect bugs inside the solver package and wiring of components inside decK.
    • This can be done against different versions of Kong.