OSSEM Data model integration with MSTICPy

[ianhelle]

Some initial thoughts on how this might work (be modified from current implementation).

1. Module/function to import and unpack file-based events - e.g. JSON - extract EventData fields and output pandas DataFrame
   • in cases where multiple events are in the source file, we might return a dictionary of dataframes, indexed by event ID
2. Download/access OSSEM yaml files
   • one option is to add the OSSEM repos as submodules of msticpy
   • another option is a GitHub actions that downloads the files (and probably reads and indexes them)
3. Module/subpackage to read OSSEM data models and serialize to indexed form. Pickled dictionaries is a possibility (current implementation but unpickling stuff is a potential security issue - esp when it comes from outside package). Maybe some other serialization format like feather would work.
4. function to map fields to OSSEM data model based on event type
   • it would be nice to have logic that could infer the event type from the data - we can create some set of rules that can do this for common event formats
   • We def need to preserve the system fields in the translated event
   • Output would be a DF
5. Map data to entities
   • We need to sync on the entity definitions. MP currently has entity schema that follows this (more or less) https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/sentinel/entities-reference.md
   • We can probably make them so that they will instantiate from and have properties that work with both the OSSEM and MS schemas - e.g. do some funky stuff overriding getattr and setattr.
   • There are also many more entities in the OSSEM schema than we currently have - we can prob create some data classes or something that auto-defines these additional classes from the OSSEM schema.
6. Build a graph
   • our entities have built-in graph support - e.g. they are each nodes and have a edges collection.
   • They also support
     • outputting/converting a collection to a networkx graph.
     • equality resolution (same attribs), equivalence resolution (overlapping set of non-conflicting attrib values) and merging methods
   • Seems like networkx would be a good, standardized way of storing the entire graph

[Cyb3rPandaH]

Meeting August 18, 2022: Sarguel, Ian, Jose
Next Meeting September 1, 2022: 5 Pm Eastern Time

Regarding 2:

• Review the implementation of a version file for every component (DD, DM, CDM) or maybe the main repo. (Jose)
• Implement version files caching in msticpy (Ian)

Regarding 3:

• On Developer branch, implement GitHub actions (DD, DM, CDM) to generate standard SQL-like database files OR a single yaml file. (Sarguel)

Regarding 4:

• We now have a way to identify dictionaries: platform, log_source, evetn_id, event_version
• But we need to review the implementation of dictionary masks
• Implement code with some ideas to review later:
  - First, implementation of event_version (Sarguel)
  - Second, review implementation of dictionary masks. Do we want a black box (Direct transformation) or have conditions to allow parsing (Jose)

[Cyb3rPandaH]

Meeting September 01, 2022: Sarguel, Jose, Ian
Next Meeting September 15, 2022: 5 Pm Eastern Time

Regarding 2:

• We will test managing versions using Git/Tags. This will we considered under point 3. (Okay)
• Implement version files caching in msticpy (Ian): Version Caching checking.

Regarding 3:

• Format proposed by Sarguel: JSON. We will use this format for initial testing (Okay)
• We need to create GitHub action on the main OSSEM repo to create JSON files using content from each Sub-Repository (3 JSON files). This might include the update of sub-repos as part of GitHub actions. Also, the GitHub actions will manage the versions of each repo using Git/Tags, and It will create a simple JSON file with just the versions that we could access with a HTTP request. (Sarguel)

Regarding 4: STANDBY: We need to define the implementation of additional field (Channel for Windows events)
We now have a way to identify dictionaries: platform, log_source, evetn_id, event_version
But we need to review the implementation of dictionary masks
Implement code with some ideas to review later:

• First, implementation of event_version (Sarguel)
• Second, review implementation of dictionary masks. Do we want a black box (Direct transformation) or have conditions to allow parsing (Jose)

Azure data request:

• Getting some samples of Azure data to test the parser. (Ian)

[Cyb3rPandaH]

Meeting September 15, 2022: Cancelled
Nest meeting Thursday October 6, 2022 (5pm Eastern Time)

I met with Sarguel on Saturday September 17, 2022 to review some updates he has been working on:

Regarding 2:

• I need to define a procedure to manage OSSEM DD, CDM and DM versions and tags. These tags will be used by github actions from point 3. (Jose)
• Implement version files caching in msticpy (Ian): Version Caching checking.

Regarding 3:

• Github action initial draft has been created in OSSEM repo (Sarguel).
  - It creates JSON files and versions numbers files (access via http reques) for CDM, DD, and DM
  - Sarguel is currently updating the github action to update the OSSEM web files.
  - The github action does not update sub-repos content within main OSSEM repo. We might need to implement this procedure on each repo so the content is updated after every PR is merged.

Regarding 4: STANDBY -> I am currently adding the Channel field to OSSEM DM
We now have a way to identify dictionaries: platform, log_source, evetn_id, event_version
But we need to review the implementation of dictionary masks
Implement code with some ideas to review later:

• First, implementation of event_version (Sarguel)
• Second, review implementation of dictionary masks. Do we want a black box (Direct transformation) or have conditions to allow parsing (Jose)

Azure data request:

• Getting some samples of Azure data to test the parser. (Ian)

[rcobb-scwx]

I am posting here at Ian's request.

As part of the Feb hackathon month, I was considering writing a code generator for the OSSEM-CDM that would generate both pydantic and pandera models (basically fancy dataclasses) for OSSEM-CDM entities and tables.

For example, the Device entity in the OSSEM-CDM would generate Python classes like this:

# Relatively close pseudo-code
from typing import Annotated

import pandera as pa

from pandera.typing import Series
from pydantic import BaseModel, Field


class Device(BaseModel):
    """
    Events used to normalize events for the device or endpoint that generated the event (source or destination).
    """
    action: Annotated[str, "If reported by an intermediary device such as a firewall, the action taken by device."] = Field(alias="Action")
    inbound_interface: Annotated[str, "If reported by an intermediary device such as a firewall, the network interface used by it for the connection to the source device"] = Field(alias="InboundInterface")
    outbound_interface: Annotated[str, "If reported by an intermediary device such as a firewall, the network interface used by it for the connection to the destination device."] = Field(alias="OutboundInterface")
    hostname: Annotated[str, "The host name from which the event/log came from. There may be multiple host names in an event (i.e. syslog could have forwarder host name), this field is to be the most true log host name (i.e. NOT the forwarders name)."] = Field(alias="Hostname")
    domain: Annotated[str, "Name of the domain the device is part of."] = Field(alias="Domain")
    fqdn: Annotated[str, "The fully qualified domain name of the host"] = Field(alias="Fqdn")
    interface_guid: Annotated[str, "GUID of the network interface which was used for authentication request"] = Field(alias="InterfaceGuid")
    interface_name: Annotated[str, "the name (description) of the network interface that was used for authentication request. You can get the list of all available network adapters using ipconfig /all command"] = Field(alias="InterfaceName")
    os: Annotated[str, "The OS of the device"] = Field(alias="Os")
    model_name: Annotated[str, "The model name of the device"] = Field(alias="ModelName")
    model_number: Annotated[str, "The model number of the device"] = Field(alias="ModelNumber")
    _type: Annotated[str, "The type of the device"] = Field(alias="Type")


class DeviceSchema(pa.SchemaModel):
    """
    Events used to normalize events for the device or endpoint that generated the event (source or destination).
    """
    action: Series[str]
    inbound_interface: Series[str]
    outbound_interface: Series[str]
    hostname: Series[str]
    domain: Series[str]
    fqdn: Series[str]
    interface_guid: Series[str]
    interface_name: Series[str]
    os: Series[str]
    model_name: Series[str]
    model_number: Series[str]
    _type: Series[str]

The benefit is that these classes would give you better auto-complete in your IDE (through pylance/jedi) and provide runtime validation of objects to ensure they conform to the specification in OSSEM-CDM. It would still be up to the developer to use the data dictionaries from OSSEM-DD to populate the generated class models for the OSSEM entities. The only coercion that pydantic or pandera would do is type enforcement (ie, normalize an int PID as a string PID, if that is how the OSSEM entity is defined).

Here is a you example of how we could define an analytical function that takes a dataframes of OSSEM-CDM devices and returns a list of malicious devices based on some detection logic:

import pandas as pd

from typing import Any, Dict, List

from pandera.typing import DataFrame

# Dictionary representation of the devices.

device_dump: List[Dict[str, Any]] = dump_devices_from_somewhere()

# The pydantic model will try to parse the dicts into
# the class model. If it is wildly different, then it
# will throw an exception. If there are subtle type
# mis-matches, it will attempt to coerce the values
# into whatever types are defined on the class model
# OSSEM-CDM entity.

# These are now tidy models, so you can access attributes
# using dot notation instead of itemgetter. You can also
# leverage pydantic to compose multiple entities into
# the OSSEM-CDM tables.

corporate_devices = [Device(**device) for device in device_dump]

# But we love our dataframes for analytics. So how can
# we get similar sanity checking/type annotations on
# of OSSEM-CDM entities in dataframes? We can use
# pandera.

device_df = pd.DataFrame(corporate_devices)

@pa.check_types(with_pydantic=True)
def detect_compromised_devices(df: DataFrame[DeviceSchema]) -> List[Device]:
    """
    This function knows nothing about where these devices
    came from or how they were populated. But if you pass
    it a dataframe with the appropriate columns as defined
    in the pandera schema model `DeviceSchema`, then the
    function should happily consume it.

    Think of this as a type annotation on a dataframe.
    The type annotation on the return value can also
    be one of these DataFrames, so if you are doing
    a bunch of munging/ETL, you can define what the output
    should be and pandera will complain if your function
    doesn't actually implement the contract.
    """
    raise NotImplementedError

# If `device_df` doesn't conform to the pandera schema model,
# then it will throw an exception. This makes it easier to
# reason about the internal logic of the function and MUCH
# easier to test.

compromised_devices = detect_compromised_devices(device_df)

Does this seem generally useful?

[ianhelle]

That does seem useful to me, although does pre-suppose that people have an easy way of getting data into OSSEM-CDM schema from the native schema.
Sargel and @Cyb3rPandaH were working on this capability in the OSSEM repo.

[rcobb-scwx]

Here is a brief update on my progress RE: OSSEM entity and table code generation into Pydantic models:

I used pypackage cookiecutter to boostrap the project and wrote parsers for both OSSEM entity and table definitions:

I then wrote jinja-based templates to convert the parsed entity/table definitions into Pydantic models.

The generated Pydantic models are syntactically valid and (as far as I can tell) semantically equivalent to the definitions in the YAML files.

And here is an example of the generated network session table model:

The generated table model properly adds the prefix and sub-selects the relevant fields from the specified entity:

My current challenges are with the tables. There are some tables that reference attributes that do not exist on their source entity. For example, the geo_city attribute listed here does not exist in the destination schema:

Additional work also needs to be done to ensure that all fields in the generated table models are actually unique. I have found a few name collisions.
The next steps would be:

• Work backwards through the OSSEM DD and create additional parsers that take a source log and attempt to coerce them into the generic OSSEM entity representation. I don’t think this is currently possible for many sources in OSSEM-DD because their standard names/types are still hard-coded as TBD.
• Create the wrapper classes to leverage the OSSEM entity models to validate/schematize pandas DataFrames (ie, pandera or pandas-dataclasses)
• Sketch out example msticpy analytic functions that work on the OSSEM DataFrame representations, rather than un-annotated DataFrames.

Hope some of this made sense :D