Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(oscal): link remapper method component defn #879

Open
wants to merge 19 commits into
base: main
Choose a base branch
from

Conversation

meganwolf0
Copy link
Collaborator

@meganwolf0 meganwolf0 commented Jan 15, 2025

Description

Added a new method to component definition - RewritePaths

Intent of this method is to facilitate aggregation of data in component definitions (e.g., via Compose functionality) where the references contained therein may be relative to said component definition. This method should also have extensibility to be used for a future packaging type workflow where we may wish to pull disaggregated OSCAL models into a centralized, opinionated structure. Hence, this method may make sense to add to the interface when it's been implemented across models.

Notes to reviewers

  • I updated the network function GetLocalFileDir -> before it was doing a file existence check which I think is probably not needed in this function itself. Removal allowed for easier testing. Also needed to update this function to check for the opaque structure (before was just checking host) - see net/url for reference

Related Issue

Fixes #878

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Other (security config, docs update, etc)

Checklist before merging

@meganwolf0 meganwolf0 changed the title 878 link remapper method component defn feat(oscal): link remapper method component defn Jan 15, 2025
@meganwolf0 meganwolf0 marked this pull request as ready for review January 21, 2025 14:03
@meganwolf0 meganwolf0 requested a review from a team as a code owner January 21, 2025 14:03
@meganwolf0 meganwolf0 marked this pull request as draft January 21, 2025 14:56
@meganwolf0 meganwolf0 marked this pull request as ready for review January 21, 2025 17:24
Copy link
Contributor

@mildwonkey mildwonkey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel a bit uncomfortable with this implementation, but not so much that I'd block it - it's partly based on not entirely understanding what's going on in here and partly based on my entirely fuzzy mental image of a future refactor, which isn't exactly a valid consideration 😂

src/pkg/common/common.go Outdated Show resolved Hide resolved
src/pkg/common/common.go Outdated Show resolved Hide resolved
src/pkg/common/common_test.go Outdated Show resolved Hide resolved
src/pkg/common/oscal/component.go Outdated Show resolved Hide resolved
@meganwolf0
Copy link
Collaborator Author

I feel a bit uncomfortable with this implementation, but not so much that I'd block it

Yeah I feel this too 😆 - I really wasn't sure how to do this optimally/reusably...

So I think the bigger picture idea is that you can have an OSCAL model (this is just for component definition, but really any OSCAL Model) that can have links to other resources (remote or local). If we want to sort of package up all this data we may need to re-write the paths. Additionally, in the case of a component definition, it can import other component definitions (which contain these links) and those internal references will need to be re-written when the data gets imported to the importer model (ala, what we do with Compose right now, although those don't resolve paths, that function just puts all validations in the backmatter, which is not great for some of the use cases we've discussed with the SSP, etc).

The initial thought on this was to brute force look through all the spots where you'd possibly need to change a reference, and just do it. Although you can see from the list of items in pathsToRemap that it's a little on the long side. So totally agreed that reflect is not optimal, but that was my solution to not brute forcing it. Maybe there's another option that I'm overlooking if you have an idea on how to do it differently, I'm all ears!

@mildwonkey
Copy link
Contributor

My discomfort is because this feels like an implementation for iterating over raw json, but we have concrete types, so I'd expect something that iterated over each type.

Here's some probably-awful pseudo code:

func (c ComponentDefinition) RewritePaths (paths, anythingelse) error {
for r := range c.Resources {
if r.Href != "" {
rewritePath(r.Href)
}
for s := range c.OtherField {
// something like this for everything
}
}

Though for the actual implementation (and this is where my handwavey future ideas come into play, so again THIS MIGHT NOT MAKE SENSE for right now/ever) I'd probably have each type with links (every field inside the ComponentDefinition) fulfill an interface that includes a RewritePaths method, so we could handle each separately.

@meganwolf0
Copy link
Collaborator Author

Ok fair enough, yeah agreed maybe it would be better to just write a super long function and have all the types checked by type and not with reflect. This does get a little tedious as well because of all the pointers... To be honest, this is probably something I could coerce chat-gipity to give me so I didn't have to write it manually 🤣

For the future idea, I truly don't think I know enough about go-oscal to know if that's do-able. But maybe worth investigating more...

I suppose for the near term we're probably trading the two options: using the reflect implementation or write a big function. (selfishly/lazily I do think this implementation is easier to test/understand)

@meganwolf0
Copy link
Collaborator Author

@brandtkeller if you have any weigh-ins on this TIA!

@brandtkeller
Copy link
Member

Reading back through the issue and then this implementation - I am weighing the "what problem does this solve now" with "are the surrounding assumptions correct".

Yes the solutions complexity obfuscates some of the operations - I had to spend some time with debugging to get a clear-ish picture.

But we have an original problem statement - composing component definitions that import other component definitions and remapping paths. Relevant problem for us - as each imported component-definition has potential links to Lula Validations or potentially other artifacts (which is why I appreciated the breadth of support in this PR).

Apologies in advance. I believe we see the intent of a package operation for sake of transport - which allows us to dictate the artifact retrieval and replacement for re-mapping (evidence for a solution).

Do we see ourselves having any mechanism of getting away from a compose like operation? for component-definitions and allowing the import of other component-defintions - it feels like the answer is no and that this will be tied intimately to the execution of validate functionality - whether directly with Lula or with LaaL. Gut feeling here is that i'd default to a verbose solution that is easily traceable over a minimal solution that is less traceable.

@meganwolf0 meganwolf0 marked this pull request as draft January 23, 2025 14:25
@meganwolf0 meganwolf0 marked this pull request as ready for review January 23, 2025 16:05
@meganwolf0
Copy link
Collaborator Author

@brandtkeller @mildwonkey - Updated this PR to remove the reflect method (as that sounded broadly unpopular 😅 ) in favor of the verbose model walk/rewrite in place

@meganwolf0 meganwolf0 requested a review from mildwonkey January 23, 2025 16:08
src/pkg/common/oscal/component.go Outdated Show resolved Hide resolved
src/pkg/common/oscal/component_test.go Outdated Show resolved Hide resolved
brandtkeller
brandtkeller previously approved these changes Jan 23, 2025
Copy link
Member

@brandtkeller brandtkeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the re-write - debugging traceability really improved my perspective.

I'm a bit of a mess - this is all notional work not tied into execution currently - do we have issue(s) that add this to current execution?

@brandtkeller
Copy link
Member

I'm a bit of a mess - this is all notional work not tied into execution currently - do we have issue(s) that add this to current execution?

#896 may be the execution implementation?

@meganwolf0
Copy link
Collaborator Author

#896 may be the execution implementation

That's right - trying to decide if that PR should also include instrumenting the method in the generate SSP function or if that should be separate. But that's really the endgame here: enable the SSP having the right links to pass through to the Assessment Plan (so then we can do a thing with that).

mildwonkey
mildwonkey previously approved these changes Jan 24, 2025
@meganwolf0 meganwolf0 dismissed stale reviews from mildwonkey and brandtkeller via 6d4a4ad January 24, 2025 15:39
@meganwolf0
Copy link
Collaborator Author

meganwolf0 commented Jan 24, 2025

Alright, I know you all already approved this but I think @mildwonkey was right about refactoring that huge function (ultimately I do see reuse across models, so the goal was to make that more reusable/clear/testable) - with that, I decided to not be lazy and made the following updates:

  • Broke down the ComponentDefinition RewritePaths method into smaller functions (that I put in common.go because they aren't all specific to the component definition model (I didn't break out every type, the intent was to break into types that I saw reused and/or ones that were nested to facilitate testing)
  • Updated the test files for ^^ method - honestly it seemed unnecessarily confusing with the abs paths in there and hopefully the point still stands. I think import/validate will use real paths but mock paths I believe are sufficient to test this. After going through the above exercise, I also wanted to address gaps in testing all the elements of ComponentDefinition
  • Added a fuzz to RemapPath, hopefully it's a valid implementation 😅

@@ -549,3 +656,11 @@ func FuzzReadValidationsFromYaml(f *testing.F) {
common.ReadValidationsFromYaml(a)
})
}

func FuzzRemapPath(f *testing.F) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 nice!!!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 you may be able to round-trip this: (If this is valid) you could re-map back to the original path and add a check that last returned path is equal to the original test input.

Something like this?
gotPath := remap path (a, b, c)
reversePath := (gotPath, c, b)

Copy link
Contributor

@mildwonkey mildwonkey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love the changes! I left one incredibly minor comment on handlePath but I think this is looking great.

return handlePath(filepath.Join(url.Host, url.RequestURI()), baseDir)
}

func handlePath(path, baseDir string) string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method name isn'g telling me anything, and it's not obvious to me why it returns an empty string if given an abs path but a full path to a directory if given a relative path. Can you add some comments that explains why this is the behavior? And maybe rename it - returnBaseDirIfNotAbsPath seems like a more accurate description of what's happening (though I still wonder why)

@@ -102,6 +102,74 @@ func (c *ComponentDefinition) HandleExisting(path string) error {
return nil
}

// RewritePaths finds all the paths in the component definition relative to the baseDir and updates them to be relative to the newDir
// baseDir and newDir must be absolute paths
func (c *ComponentDefinition) RewritePaths(baseDir string, newDir string) (err error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love the changes here ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

Link re-mapper method on Component Definition
3 participants