Generate entity/relationship documentation from well-defined entity/relationship metadata

[austinkelleher]

Problem

In large integrations, there are many steps that consume many different _type's. We currently have a _type tracker that will warn if an integration step ingests an entity or relationship with a _type that is not listed. For example:

export const mySteps: IntegrationStep<IntegrationConfig>[] = [
  {
    id: 'my-step',
    name: 'The step name',
    types: [
      // Entity _type
      'my_special_entity_type',
      // Relationship _type
      'my_special_entity_type_has_other_class'
    ],
    executionHandler: myHandler,
  },
];

Today, a developer will manually copy over documentation into the standard docs/jupiterone.md location. There is no safety that the information that they're copying over is accurate to what is actually described in the integration steps. Additionally, there is no information about which steps ingest which data or which order the data the ingested in. The standard documentation format resembles the following:

## Data Model

### Entities

Provide a table that maps concepts from the provider to the `_type` and `_class`
generated.

| Resources         | \_type of the Entity           | \_class of the Entity |
| ----------------- | ------------------------------ | --------------------- |
| My Resource       | `my_special_entity_type`       | `MyClass`             |
| My Other Resource | `my_other_special_entity_type` | `OtherClass`          |

...

### Relationships

The following relationships are created/mapped:

| From                     | Edge    | To                             |
| ------------------------ | ------- | ------------------------------ |
| `my_special_entity_type` | **HAS** | `my_other_special_entity_type` |

Proposal

We introduce a breaking change to IntegrationStep to remove types and introduce two new properties; entities and relationships. The data structures will look like this:

interface IntegrationStepGraphObjectMetadata {
  _type: string;
  description?: string;
}

interface IntegrationStepEntityMetadata extends IntegrationStepGraphObjectMetadata {
  _class: string | string[];
}

interface IntegrationStepRelationshipMetadata extends IntegrationStepGraphObjectMetadata {
  _class: string;
  sourceType: string;
  targetType: string;
}

interface IntegrationStep {
  ...
  entities: IntegrationStepEntityMetadata[];
  relationships: IntegrationStepRelationshipMetadata[];
}

A new j1-integration document command will be introduced that will auto-generate entity/relationship documentation based on the data defined in each step. The document command will update an existing Markdown file or create a new one if it does not already exist. The contents will resemble what is generated today and described above. The default location that the file will read/write from is {CWD}/docs/jupiterone.md. Similarly to j1-integration collect (among other commands), the j1-integration document command will read the src/index file to determine which steps are relevant.

Usage:

Usage: j1-integration document [options]

Options:

  --help -h Show this help message [string]

--location -l Location of the generate entity/relationship documentation. Defaults to docs/jupiterone.md. [string]

[austinkelleher]

Additional thoughts:

In the future, we can leverage this same data structure for additional metadata about a step for auto-generated documentation:

• Required permissions to ingest the data from the step
• Sample useful queries
• ...???

[ndowmon]

Nice Austin!!! One minor request for one of the proposed interfaces:

interface IntegrationStepRelationshipMetadata extends IntegrationStepGraphObjectMetadata {
  sourceType: string;
  targetType: string;
  _class: string;
}

[austinkelleher]

Thanks @ndowmon! Fixed and introduced IntegrationStepEntityMetadata for clarity.

[ndowmon]

Do we want to put any sort of character limit on the description property? Not sure how we want this to render in docs.