ngclient testing: client workflow sequences

[sechkova]

Description of issue or feature request:

If needed, narrow down the scope of this issue and spawn separate ones for top-level metadata update, delegations metadata update, Fetching targets.

edit: This issue deals with top-level-metadata updates. Delegated roles and targets will be covered #1641 #1642 #1643 .

Following the high-level steps described in #1579 implement tests for the various client workflow sequences:

• Tests scenarios from specification (Can we mark these to be our conformance tests?)
• Specific sequences the client should be able to handle go here
• Extracted useful scenarios from legacy tests
• Specific python-tuf sequences (for ex. loading and validating local metadata)

Decide what we want to test sometimes and what should be always included:

• errors / no errors
• what metadata is downloaded?
• what metadata is read from disk?
• what metadata is on disk afterwards?
• are the currently loaded metadata versions correct?
• (some of the same bullet points repeated for targets)
• ? more ?

The tests should be RepositorySimulator based.

[MVrachev]

Decide what we want to test sometimes and what should be always included:
errors / no errors
what metadata is downloaded?
what metadata is read from disk?
what metadata is on disk afterwards?
are the currently loaded metadata versions correct?
(some of the same bullet points repeated for targets)
? more ?

When we decide on that we can use it as a start for capturing an ADR resolving #1129?

[jku]

what metadata is downloaded?
what metadata is read from disk?

I think these are worth testing (because it's so easy to regress into loading something twice or something), but I think making all tests do this would be ugly... I would suggest a specific set of tests that e.g. runs a few refresh(), get_targetinfo() and download_target() calls with a reasonably complex repository setup, and ensures only the expected downloads and file reads are done.

what metadata is on disk afterwards?
are the currently loaded metadata versions correct?

These should be checked on more tests than just a specific one... at least the root key rotation tests should ensure that local trusted metadata is the expected file.

[sechkova]

After experimenting in #1636 with top-level-roles update:

• Tests scenarios from specification (Can we mark these to be our conformance tests?)

These are probably the most straight-forward test to implement, they tests step by step the detailed client workflow (6565365). In case of a specific need to generate many variations of a test, like in the case of keys rotation, this can be expanded in a seperate test class: #1607

• Specific sequences the client should be able to handle go here

These are still to be defined

• Extracted useful scenarios from legacy tests

To be added

• Specific python-tuf sequences (for ex. loading and validating local metadata)

Testing if local metadata is loaded is not impossible eaeb091 but as described in #1636, Testing if expired metadata from the local cache is loaded and used to verify the new version requires peeking as deep as
Updater._trusted_metadata_set["timestamp] != None I don't find it fir for these tests, it seems like TrustedMetadataSet tests should accommodate it.

[sechkova]

Decide what we want to test sometimes and what should be always included:

If we want to define a common rule, it seems like positive tests that do not test for some exception or error need to include some of these checks to confirm that the expected result is achieved.
What exactly to test is very case-dependent. I think I agree with you that checking what metadata is on disk afterwards as well as its version is the most common need.

[jku]

The new set of tests (test_updater_top_level_update.py) are in my opinion the state of the art now:

• tests are reasonably short and readable
• it's easy to see what individual tests actually check

If we can document the decision making process that lead to these choices (situation X -> should test for Y) maybe in the test module docstring or somewhere that would be good... but if there are no generic rules and we just need to look at individual tests, then I guess the current examples are good examples to look at.

Can we close this or are there more sub issues that need to be filed first?

[sechkova]

I opened two follow up issues for testing the rest of the steps following the client workflow from the specification: #1641 #1642. We can close this one.