Feature/integration tests

[wandmagic]

Committer Notes

This PR introduces a baseline for integration tests using cucumber and oscal cli.

this will allow a github action to verify that the upcoming version of oscal will still validate content as expected.

Additionally, this can be used as the groundwork for styleguides and other checks we may need to introduce into the build process to maintain the integrity of the OSCAL project.

All Submissions:

• Have you selected the correct base branch per Contributing guidance?
• Have you set "Allow edits and access to secrets by maintainers"?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?
• Have you squashed any non-relevant commits and commit messages? [instructions]
• Do all automated CI/CD checks pass?

By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.

(For reviewers: The wiki has guidance on code review and overall issue review for completeness.)

Changes to Core Features:

• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your core changes, as applicable?
• Have you included examples of how to use your new feature(s)?
• Have you updated the OSCAL website and readme documentation affected by the changes you made? Changes to the OSCAL website can be made in the OSCAL-Pages and OSCAL_Reference repositories.

[wandmagic]

@brian-ruf @Rene2mt See this pull request for how we can be sure that direction prop is allowed in components.

Currently the tests fail because direction prop is not in alignment. ssp.xml has an added direction prop

See test run:
https://github.com/usnistgov/OSCAL/actions/runs/12162062858/job/33918108617?pr=2084

[wandmagic]

@iMichaela @wendellpiez kindly let me know if you have a chance to look at this PR, would love to hear your feedback

[iMichaela]

@wandmagic - Hi Paul - thank you for the proposed harness. I am abroad on PTO until mid Dec and will only be able to provide a complete feedback upon my return and gain access to my dev environment. Sorry for the delay.

[iMichaela]

@wandmagic -- All I can tell so far, from browsing the changed files is that the test harness is installing and using oscal - a very confusing name (I strongly objected to in the past and still insist to be changed !) a JS wrapper of oscal-cli version used by FedRAMP and not by NIST under usnistgov ) & OSCAL Server (also not managed by NIST).

Also, this PR is barcoding "valid OSCAL content " which is obsolete and not aligned with the oscal-content set of examples which should be used as we update them to latest OSCAL schemas (today: 1.1.3)

For interested reviewers, see: https://github.com/GSA/oscal-js . All baselines used are FeRAMP baselines. All examples used are GSA examples.

This is not something NIST can accept as proposed, despite its great intent of enhancing OSCAL testing methodology. Changes are needed.

Major ask: remove/rename the oscal npm package https://www.npmjs.com/package/oscal wrongly and confusingly named after the OSCAL (language) - a set of schemas. This is unacceptable to NIST, as documented in the issue GSA/oscal-js#4 - an issue already closed by @aj-stein-gsa which stated that oscal is used by the community in different situations and no confusion was created. I find no substance and no proof provided by this simple opinionated statement (aka "no confusion is created") . NIST is not endorsing the use on oscal name for the OSCAL schema validation because coming from NIST will cause confusion. I also recall Dave's opinion aligned with mine since both of us fought for years community members' misunderstanding that OSCAL is a tool they can install. Providing an oscal tool that can be installed but which is not OSCAL the language (the set of schemas) will bring back the confusion, especially when coming from NIST.

[wendellpiez]

Notwithstanding the difficulties cited by @iMichaela, I am very much liking the general direction here. A PR along these lines would be a small but significant step forward. We manage the risks of posting bad data by not posting bad data, so of course any concerns by @iMichaela or others need to be addressed. But we also need a way that third parties can judge data quality for themselves without having to ask us - we do this (in part) by posting useful examples even if only mockups.

[iMichaela]

Notwithstanding the difficulties cited by @iMichaela, I am very much liking the general direction here. A PR along these lines would be a small but significant step forward. We manage the risks of posting bad data by not posting bad data, so of course any concerns by @iMichaela or others need to be addressed. But we also need a way that third parties can judge data quality for themselves without having to ask us - we do this (in part) by posting useful examples even if only mockups.

Totally agree with @wendellpiez general direction. However, this PR proposes to 'augment' OSCAL schemas validation using the harness FedRAMP is using. NIST OSCAL Makefile already includes validation with usnistgov\oscal-cli but suffers when usnistgov\oscal-cli is not updated to the latest OSCAL version. A chicken and an egg problem which will not be addressed by this PR. This PR is not proposing a tool for others to use, but rather an additional way for using an oscal-cli which is not usnistgov\oscal-cli + OSCAL Server developed by GSA for their purpose.

[wandmagic]

this PR no longer uses oscal-server or oscal-js, please have a second look. @iMichaela the contents of valid-content are copy pasted from oscal-content repo. we can integrate this data as a submodule or query them directly, i added them directly under the impression that they would have the newest up to date for an upcoming version, and oscal-content would be updated upon release of a new version. I understand reservations about naming and confusion regarding oscal-js, and I feel that the library is not needed here as it is largely a convenience library. oscal-server is only for performance of running hundreds of oscal-cli commands in parallel and is not needed here either.

[wandmagic]

It would be good to have an itemized list of things we need to have to get this type of check integrated into the repo, if it is possible.

[aj-stein-gsa]

@iMichaela since there are a variety of concerns and claims that are separate of the PR and out of scope given @wandmagic's changes described in #2084 (comment), I moved the discussion to #2087 and not sidetrack from the PR at hand. I hope that helps.

[wandmagic]

I tried switching to the NIST managed oscal CLI from here: https://repo1.maven.org/maven2/gov/nist/secauto/oscal/tools/oscal-cli/cli-core/1.0.3/
Which leads me to this error when running make test:

An uncaught runtime error occured. An error occurred while evaluating the expression 'prop[has-oscal-namespace('http://csrc.nist.gov/ns/oscal')]/@name'.

To me this indicates that the nist managed CLI is no longer maintained enough to be leveraged as part of CI/CD.

[iMichaela]

It would be good to have an itemized list of things we need to have to get this type of check integrated into the repo, if it is possible.

We can work on it together when I return to US