Automate API design tests to ensure CAMARA compliance

[Kevsy]

This proposal is to create a pipeline of three tests to ensure that the API definition has been defined according to CAMARA API design principles and is 'secure by design'. These tests are against the API definition itself and not the implementation.

Part of this topic has been raised before - see e.g. Commonalties camaraproject/QualityOnDemand#6 - but this proposal extends the concept with a test pipeline. The tests should be run as part of every release and act as a gate for release publication.

The tests are:

1. Check compliance against OAS:
Using e.g. Spectral lint GitHub action.

2. Check compliance against CAMARA API design guidelines:
Spectral lint again, but with a custom ruleset based on the rules in the API design guidelines. This ruleset would need to be derived from the API design guidelines

3. Check Security by Design:
Fuzz testing using Schemathesis . Detects errors/vulnerabilities at design level. Produces a report at the end too! Available as a GitHub app.

4. Check API meets business requirements
BDD (Gherkin .feature) needs to be present, although I'm not sure if the .feature file itself can be tested to determine CAMARA compliance.

Tests 1 and 3 are 'out of the box' and require no further configuration. Test 2 would require a linting ruleset to be derived from the API Design Guidelines.

[hdamker]

@Kevsy I'm supportive of automating tests as far as possible.

Part of this topic has been raised before - see e.g. Commonalties camaraproject/QualityOnDemand#6 - but this proposal extends the concept with a test pipeline. The tests should be run as part of every release and act as a gate for release publication.

Actually, linting should run before every merge of changes to the OAS. That's the way it's today implemented in QualityOnDemand sub project (can be used as a blueprint).

The tests are:

1. Check compliance against OAS: Using e.g. Spectral lint GitHub action.

Established in QualityOnDemand with mega linter as umbrella. You can fork the repo and test it.
The main initial effort was to correct the OAS file so that no errors were raised finally (e.g. spaces etc).

2. Check compliance against CAMARA API design guidelines: Spectral lint again, but with a custom ruleset based on the rules in the API design guidelines. This ruleset would need to be derived from the API design guidelines

That's what #15 is about.

3. Check Security by Design: Fuzz testing using Schemathesis . Detects errors/vulnerabilities at design level. Produces a report at the end too! Available as a GitHub app.

I'm not sure, but from first view the tool is generating tests based on the OAS but then they supposed to be executed against an API implementation? But maybe I need to spend more time on it.

4. Check API meets business requirements BDD (Gherkin .feature) needs to be present, although I'm not sure if the .feature file itself can be tested to determine CAMARA compliance.

Tests 1 and 3 are 'out of the box' and require no further configuration. Test 2 would require a linting ruleset to be derived from the API Design Guidelines.

[hdamker]

Transfer to https://github.com/camaraproject/ReleaseManagement

[tanjadegroot]

@hdamker To transfer this issue back to Commonalities as agreed within https://wiki.camaraproject.org/display/CAM/2024-05-14+Release+WG+Minutes, as this concerns linting & tests to be defined in Commonalities.

[rartych]

I propose to to close this issue and continue in #188