Add oval-graph support for the test suite #7750
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jan-cerny
requested changes
Oct 18, 2021
There was a problem hiding this comment.
My biggest problem is that when I run some rule like selinux_state which has 3 scenarios I can see graph file only for one of the scenarios.
[jcerny@thinkpad scap-security-guide{pr/7750}]$ python3 tests/test_suite.py rule --libvirt qemu:///system ssgts_rhel8 selinux_state
Setting console output to log level INFO
INFO - The base image option has not been specified, choosing libvirt-based test environment.
INFO - Logging into /home/jcerny/work/git/scap-security-guide/logs/rule-custom-2021-10-18-0954/test_suite.log
INFO - xccdf_org.ssgproject.content_rule_selinux_state
INFO - Script selinux_enforcing.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script selinux_missing.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
INFO - Script selinux_permissive.fail.sh using profile xccdf_org.ssgproject.content_profile_ospp OK
[jcerny@thinkpad scap-security-guide{pr/7750}]$ ls /home/jcerny/work/git/scap-security-guide/logs/rule-custom-2021-10-18-0954
env-preparation.log selinux_state.prescripts.log test_suite.log
results.json selinux_state-selinux_enforcing.pass.sh-initial-graph.html
[jcerny@thinkpad scap-security-guide{pr/7750}]$
| @@ -427,7 +428,7 @@ def _get_formatting_dict_for_remediation(self): | |||
|
|
|||
| class ProfileRunner(GenericRunner): | |||
| def _get_arf_file(self): | |||
| return '{0}-initial-arf.xml'.format(self.profile) | |||
| return '{0}-{self.stage}-arf.xml'.format(self.profile, self.stage) | |||
There was a problem hiding this comment.
Suggested change
| return '{0}-{self.stage}-arf.xml'.format(self.profile, self.stage) | |
| return '{0}-{1}-arf.xml'.format(self.profile, self.stage) |
| html_filename = re.sub(r"\barf\b", "graph", arf_filename) | ||
| html_filename = re.sub(r".xml", ".html", html_filename) | ||
|
|
||
| cmd = ['arf-to-graph', '--all-in-one', '--output', html_filename, arf_filename, '.'] |
There was a problem hiding this comment.
Please document that the graph is used in the test suite readme.
|
@matejak @Honny1 I have discovered that it's because arf-to-graph refuses to consume the ARF. @matejak It would be better to consume the stderr/stdout of the arf-to-graph command in the @Honny1 What do you think? |
|
@jan-cerny It seems you have replicated the issue in the |
|
@jan-cerny The issue with |
jan-cerny
added a commit
to jan-cerny/scap-security-guide
that referenced
this pull request
Dec 1, 2021
During review of ComplianceAsCode#7750 we have discovered that for this test scenario the rule fails not because the `SELINUX=permissive` but because no `SELINUX=` entry exists. The missing `SELINUX=` entry was caused by a typo in the sed expression which effectively replaced `SELINUX=enforcing` by just `permissive`. As a result, oscap didn't find the OVAL object `oval:ssg-object_etc_selinux_config:obj:1`.
|
@Honny1 Thanks for finding the root cause which was that the ARF didn't contain any system_data because no OVAL object has been collected. Thanks to that, I found it suspicous that oscap didn't collect any object and I have discovered that there is a problem in the test scenario |
jan-cerny
approved these changes
Dec 1, 2021
|
@matejak As the problem is not in this PR but in oval-graph I think we can happily merge this unless you know about something that would prevent that. Just please rebase it on master so that we have the F34 gating job. |
|
@matejak bump |
This PR flips the result format to ARF, and uses oval-graph to generate HTML graphs locally. HTML graphs are not cleaned from the folder.
matejak
force-pushed
the
oval-graph_ssgts
branch
from
0a159be to
a816f2e
Compare
|
The new version of |
matejak
added a commit
to matejak/scap-security-guide
that referenced
this pull request
Jan 4, 2022
In the past, ARFs were used exclusively as a source of Bash/Ansible remediations. However, it is more practical to have them available as test artifacts, as they contain more information. In order to finish what ComplianceAsCode#7750 just started, following needed to be done: - Rename ..._file to ..._basename, so it is clear what the variable represents. - Always generate ARFs during initial stage of tests even if reports are not requested - those files are still needed to act as sources of remediations. - Keep those ARFs that could be used in remediations around for the next stage of tests. - Make test runners a context manager, which allows them to clean remaining files after the test run finishes.
Connect2naga
pushed a commit
to Connect2naga/content
that referenced
this pull request
Jan 24, 2022
In the past, ARFs were used exclusively as a source of Bash/Ansible remediations. However, it is more practical to have them available as test artifacts, as they contain more information. In order to finish what ComplianceAsCode#7750 just started, following needed to be done: - Rename ..._file to ..._basename, so it is clear what the variable represents. - Always generate ARFs during initial stage of tests even if reports are not requested - those files are still needed to act as sources of remediations. - Keep those ARFs that could be used in remediations around for the next stage of tests. - Make test runners a context manager, which allows them to clean remaining files after the test run finishes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR flips the result format to ARF, and uses oval-graph to generate HTML graphs locally.
If
oval-graphis available on the system that runs the tests, the test suite generates the interactive HTML visualization of the OVAL check based on the ARF. Generated HTML graphs are small, so they are currently not cleaned from the folder (in contrast to reports).It looks like that the ARF that oscap produces when called with
--remediaterefers to the state before the fix, so those graphs aren't useful.