File.macho create

[peasead]

RFC Preview

Resolves #1096

• Have you signed the contributor license agreement? Yes
• Have you followed the contributor guidelines? Yes
• For proposing substantial changes or additions to the schema, have you reviewed the RFC process? Yes
• If submitting code/script changes, have you verified all tests pass locally using make test? NA at this stage
• If submitting schema/fields updates, have you generated new artifacts by running make and committed those changes? NA at this stage
• Is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed. Yes
• Have you added an entry to the CHANGELOG.next.md? NA at this stage

[andrewstucki]

Not that I'm super excited about it, but--how do you envision working with fat object files with multiple architectures? Just wondering because I would think you'd almost need to structure the entire macho field sets as nested if you want to handle them... @webmat what are your thoughts on making something that could be as large as a macho field set entirely nested?

[peasead]

@ebeahan @dcode has made the call on the nested field type for macho.headers.

Are there any hanging questions for this PR stage?

[ebeahan]

I made a pass on the macho YAML field definition.

The only major change I made was removing the sub-fields from macho.segments.sections, due to using type: flattened. I also touch on this in another comment below.

[ebeahan]

I think we're nearly there with this one. Here's a summary of what's already completed and what's outstanding for advancing:

• Identified sponsor. @devonakerr is sponsoring and has already reviewed ✅
• High-level description of examples of usage
• High-level description of example sources of data
• Identified potential concerns and implementation challenges/complexity
• Subject matter experts identified and weighed in
• Outlined initial field definitions.*

*I believe the thread here is the last open discussion point, but I'll make another pass at the current field definitions in the proposal.

[ebeahan]

@andrewstucki I did some cross-checking between the proposed field list here and your work in elastic/beats#24195. I pulled those mappings for macho into a gist for easier reference.

100% alignment isn't necessary at all at this point. However, there are enough differences between that approach and what's captured here I think it's worth raising.

@peasead @dcode @devonakerr

[dcode]

I have since updated this PR with a proposed layout that aligns sections and segments with ELF and PE (which doesn't have segments).

[ebeahan]

Thanks, @dcode.

There's still some large differences between the implementation in elastic/beats#24195 and what's proposed here.

Again, not something we need to completely resolve now. At a minimum, let's capture the difference in approaches as a Concern to make sure we revisit the topic in the next stage's PR.

[dcode]

@andrewstucki the strategy I'm suggesting here is splitting out fat binary components that can be unified by the same file metadata (namely hashes or other source unique id's). That will allow analyzing all binaries by their sections, segments, symbols, etc in a seamless way, regardless of platform or if their multi-platform. Do you think that can work for your implementation?

[peasead]

Just pining to see if there was a resolution here.

[andrewstucki]

@dcode sorry, just caught up on this. To clarify, you're talking about essentially emitting an event per cpu architecture? I'm not completely against that, but it does seem like a pretty artificial constraint to model something that is, by its nature, a single entity (i.e. a multi-arch mach-o file).

[andrewstucki]

Just as a use-case example: let's say we're triggering an alert on a malicious binary that has been compiled for both the new m1 as well as older, intel-based chipsets and we want to add in object-level data about the file. Would we emit two separate alerts, one for each architecture? Seems kind of odd.

[peasead]

Moved to new PR #1346