Skip to content

Commit

Permalink
[SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts im…
Browse files Browse the repository at this point in the history
…provements to lists (elastic#72337)

## Summary

* Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities.
* Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types
* Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things
* Removes extra types that were causing confusion which was an idiom from io-ts
* Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point.

NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted.


Output example of posting an exception list:

❯ ./post_exception_list.sh
```ts
{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "Wzk4NiwxXQ==",
  "created_at": "2020-07-17T18:59:22.872Z",
  "created_by": "yo",
  "description": "This is a sample endpoint type exception",
  "id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
  "type": "detection",
  "updated_at": "2020-07-17T18:59:22.891Z",
  "updated_by": "yo"
}
```

Output example of posting an exception list item
❯ ./post_exception_list_item.sh
```ts
{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "Wzk4NywxXQ==",
  "comments": [],
  "created_at": "2020-07-17T18:59:30.286Z",
  "created_by": "yo",
  "description": "This is a sample endpoint type exception",
  "entries": [
    {
      "field": "actingProcess.file.signer",
      "operator": "excluded",
      "type": "exists"
    },
    {
      "field": "host.name",
      "operator": "included",
      "type": "match_any",
      "value": [
        "some host",
        "another host"
      ]
    }
  ],
  "id": "a4f2b800-c85f-11ea-b1a6-c155df988a92",
  "item_id": "simple_list_item",
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f",
  "type": "simple",
  "updated_at": "2020-07-17T18:59:30.304Z",
  "updated_by": "yo"
}
```

Output example of when you get an exception list:

❯ ./get_exception_list.sh simple_list
```ts
{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "WzEwNzcsMV0=",
  "created_at": "2020-07-17T18:59:22.872Z",
  "created_by": "yo",
  "description": "Different description",
  "id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
  "type": "endpoint",
  "updated_at": "2020-07-17T20:01:24.958Z",
  "updated_by": "yo"
}
```

Example of the error you get if you do an update of an exception list and someone else has changed it:
```ts
{
  "message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }",
  "status_code": 409
}
```

Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts:

```ts
./post_list.sh
./post_list_item.sh
./find_list.sh
./find_list_item.sh
```



### Checklist

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
  • Loading branch information
FrankHassanabad committed Jul 20, 2020
1 parent 9ac0081 commit 18d251d
Show file tree
Hide file tree
Showing 128 changed files with 517 additions and 435 deletions.
1 change: 1 addition & 0 deletions x-pack/plugins/lists/common/constants.mock.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,4 @@ export const TAGS = [];
export const COMMENTS = [];
export const FILTER = 'name:Nicolas Bourbaki';
export const CURSOR = 'c29tZXN0cmluZ2ZvcnlvdQ==';
export const _VERSION = 'WzI5NywxXQ==';
4 changes: 4 additions & 0 deletions x-pack/plugins/lists/common/schemas/common/schemas.ts
Original file line number Diff line number Diff line change
Expand Up @@ -307,3 +307,7 @@ export type Deserializer = t.TypeOf<typeof deserializer>;

export const deserializerOrUndefined = t.union([deserializer, t.undefined]);
export type DeserializerOrUndefined = t.TypeOf<typeof deserializerOrUndefined>;

export const _version = t.string;
export const _versionOrUndefined = t.union([_version, t.undefined]);
export type _VersionOrUndefined = t.TypeOf<typeof _versionOrUndefined>;
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ export const createEsBulkTypeSchema = t.exact(
})
);

export type CreateEsBulkTypeSchema = t.TypeOf<typeof createEsBulkTypeSchema>;
export type CreateEsBulkTypeSchema = t.OutputOf<typeof createEsBulkTypeSchema>;
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,4 @@ export const indexEsListItemSchema = t.intersection([
esDataTypeUnion,
]);

export type IndexEsListItemSchema = t.TypeOf<typeof indexEsListItemSchema>;
export type IndexEsListItemSchema = t.OutputOf<typeof indexEsListItemSchema>;
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,4 @@ export const indexEsListSchema = t.exact(
})
);

export type IndexEsListSchema = t.TypeOf<typeof indexEsListSchema>;
export type IndexEsListSchema = t.OutputOf<typeof indexEsListSchema>;
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@ export const updateEsListItemSchema = t.intersection([
esDataTypeUnion,
]);

export type UpdateEsListItemSchema = t.TypeOf<typeof updateEsListItemSchema>;
export type UpdateEsListItemSchema = t.OutputOf<typeof updateEsListItemSchema>;
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,4 @@ export const updateEsListSchema = t.exact(
})
);

export type UpdateEsListSchema = t.TypeOf<typeof updateEsListSchema>;
export type UpdateEsListSchema = t.OutputOf<typeof updateEsListSchema>;
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import {
name,
tags,
} from '../common/schemas';
import { Identity, RequiredKeepUndefined } from '../../types';
import { RequiredKeepUndefined } from '../../types';
import { CreateCommentsArray, DefaultCreateCommentsArray, DefaultEntryArray } from '../types';
import { EntriesArray } from '../types/entries';
import { DefaultUuid } from '../../siem_common_deps';
Expand All @@ -44,20 +44,16 @@ export const createEndpointListItemSchema = t.intersection([
),
]);

export type CreateEndpointListItemSchemaPartial = Identity<
t.TypeOf<typeof createEndpointListItemSchema>
>;
export type CreateEndpointListItemSchema = RequiredKeepUndefined<
t.TypeOf<typeof createEndpointListItemSchema>
>;
export type CreateEndpointListItemSchema = t.OutputOf<typeof createEndpointListItemSchema>;

// This type is used after a decode since some things are defaults after a decode.
export type CreateEndpointListItemSchemaDecoded = Identity<
Omit<CreateEndpointListItemSchema, '_tags' | 'tags' | 'item_id' | 'entries' | 'comments'> & {
_tags: _Tags;
comments: CreateCommentsArray;
tags: Tags;
item_id: ItemId;
entries: EntriesArray;
}
>;
export type CreateEndpointListItemSchemaDecoded = Omit<
RequiredKeepUndefined<t.TypeOf<typeof createEndpointListItemSchema>>,
'_tags' | 'tags' | 'item_id' | 'entries' | 'comments'
> & {
_tags: _Tags;
comments: CreateCommentsArray;
tags: Tags;
item_id: ItemId;
entries: EntriesArray;
};
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import {
namespace_type,
tags,
} from '../common/schemas';
import { Identity, RequiredKeepUndefined } from '../../types';
import { RequiredKeepUndefined } from '../../types';
import {
CreateCommentsArray,
DefaultCreateCommentsArray,
Expand Down Expand Up @@ -53,24 +53,17 @@ export const createExceptionListItemSchema = t.intersection([
),
]);

export type CreateExceptionListItemSchemaPartial = Identity<
t.TypeOf<typeof createExceptionListItemSchema>
>;
export type CreateExceptionListItemSchema = RequiredKeepUndefined<
t.TypeOf<typeof createExceptionListItemSchema>
>;
export type CreateExceptionListItemSchema = t.OutputOf<typeof createExceptionListItemSchema>;

// This type is used after a decode since some things are defaults after a decode.
export type CreateExceptionListItemSchemaDecoded = Identity<
Omit<
CreateExceptionListItemSchema,
'_tags' | 'tags' | 'item_id' | 'entries' | 'namespace_type' | 'comments'
> & {
_tags: _Tags;
comments: CreateCommentsArray;
tags: Tags;
item_id: ItemId;
entries: EntriesArray;
namespace_type: NamespaceType;
}
>;
export type CreateExceptionListItemSchemaDecoded = Omit<
RequiredKeepUndefined<t.TypeOf<typeof createExceptionListItemSchema>>,
'_tags' | 'tags' | 'item_id' | 'entries' | 'namespace_type' | 'comments'
> & {
_tags: _Tags;
comments: CreateCommentsArray;
tags: Tags;
item_id: ItemId;
entries: EntriesArray;
namespace_type: NamespaceType;
};
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import {
namespace_type,
tags,
} from '../common/schemas';
import { Identity, RequiredKeepUndefined } from '../../types';
import { RequiredKeepUndefined } from '../../types';
import { DefaultUuid } from '../../siem_common_deps';
import { NamespaceType } from '../types';

Expand All @@ -43,17 +43,15 @@ export const createExceptionListSchema = t.intersection([
),
]);

export type CreateExceptionListSchemaPartial = Identity<t.TypeOf<typeof createExceptionListSchema>>;
export type CreateExceptionListSchema = RequiredKeepUndefined<
t.TypeOf<typeof createExceptionListSchema>
>;
export type CreateExceptionListSchema = t.OutputOf<typeof createExceptionListSchema>;

// This type is used after a decode since some things are defaults after a decode.
export type CreateExceptionListSchemaDecoded = Identity<
Omit<CreateExceptionListSchema, '_tags' | 'tags' | 'list_id' | 'namespace_type'> & {
_tags: _Tags;
tags: Tags;
list_id: ListId;
namespace_type: NamespaceType;
}
>;
export type CreateExceptionListSchemaDecoded = Omit<
RequiredKeepUndefined<t.TypeOf<typeof createExceptionListSchema>>,
'_tags' | 'tags' | 'list_id' | 'namespace_type'
> & {
_tags: _Tags;
tags: Tags;
list_id: ListId;
namespace_type: NamespaceType;
};
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import * as t from 'io-ts';

import { id, list_id, meta, value } from '../common/schemas';
import { Identity, RequiredKeepUndefined } from '../../types';
import { RequiredKeepUndefined } from '../../types';

export const createListItemSchema = t.intersection([
t.exact(
Expand All @@ -21,5 +21,7 @@ export const createListItemSchema = t.intersection([
t.exact(t.partial({ id, meta })),
]);

export type CreateListItemSchemaPartial = Identity<t.TypeOf<typeof createListItemSchema>>;
export type CreateListItemSchema = RequiredKeepUndefined<t.TypeOf<typeof createListItemSchema>>;
export type CreateListItemSchema = t.OutputOf<typeof createListItemSchema>;
export type CreateListItemSchemaDecoded = RequiredKeepUndefined<
t.TypeOf<typeof createListItemSchema>
>;
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import * as t from 'io-ts';

import { description, deserializer, id, meta, name, serializer, type } from '../common/schemas';
import { Identity, RequiredKeepUndefined } from '../../types';
import { RequiredKeepUndefined } from '../../types';

export const createListSchema = t.intersection([
t.exact(
Expand All @@ -20,5 +20,5 @@ export const createListSchema = t.intersection([
t.exact(t.partial({ deserializer, id, meta, serializer })),
]);

export type CreateListSchemaPartial = Identity<t.TypeOf<typeof createListSchema>>;
export type CreateListSchema = RequiredKeepUndefined<t.TypeOf<typeof createListSchema>>;
export type CreateListSchema = t.OutputOf<typeof createListSchema>;
export type CreateListSchemaDecoded = RequiredKeepUndefined<t.TypeOf<typeof createListSchema>>;
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import * as t from 'io-ts';

import { id, item_id } from '../common/schemas';
import { RequiredKeepUndefined } from '../../types';

export const deleteEndpointListItemSchema = t.exact(
t.partial({
Expand All @@ -17,7 +18,9 @@ export const deleteEndpointListItemSchema = t.exact(
})
);

export type DeleteEndpointListItemSchema = t.TypeOf<typeof deleteEndpointListItemSchema>;
export type DeleteEndpointListItemSchema = t.OutputOf<typeof deleteEndpointListItemSchema>;

// This type is used after a decode since some things are defaults after a decode.
export type DeleteEndpointListItemSchemaDecoded = DeleteEndpointListItemSchema;
export type DeleteEndpointListItemSchemaDecoded = RequiredKeepUndefined<
t.TypeOf<typeof deleteEndpointListItemSchema>
>;
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import * as t from 'io-ts';

import { id, item_id, namespace_type } from '../common/schemas';
import { NamespaceType } from '../types';
import { RequiredKeepUndefined } from '../../types';

export const deleteExceptionListItemSchema = t.exact(
t.partial({
Expand All @@ -19,11 +20,11 @@ export const deleteExceptionListItemSchema = t.exact(
})
);

export type DeleteExceptionListItemSchema = t.TypeOf<typeof deleteExceptionListItemSchema>;
export type DeleteExceptionListItemSchema = t.OutputOf<typeof deleteExceptionListItemSchema>;

// This type is used after a decode since some things are defaults after a decode.
export type DeleteExceptionListItemSchemaDecoded = Omit<
DeleteExceptionListItemSchema,
RequiredKeepUndefined<t.TypeOf<typeof deleteExceptionListItemSchema>>,
'namespace_type'
> & {
namespace_type: NamespaceType;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import * as t from 'io-ts';

import { id, list_id, namespace_type } from '../common/schemas';
import { NamespaceType } from '../types';
import { RequiredKeepUndefined } from '../../types';

export const deleteExceptionListSchema = t.exact(
t.partial({
Expand All @@ -19,9 +20,12 @@ export const deleteExceptionListSchema = t.exact(
})
);

export type DeleteExceptionListSchema = t.TypeOf<typeof deleteExceptionListSchema>;
export type DeleteExceptionListSchema = t.OutputOf<typeof deleteExceptionListSchema>;

// This type is used after a decode since some things are defaults after a decode.
export type DeleteExceptionListSchemaDecoded = Omit<DeleteExceptionListSchema, 'namespace_type'> & {
export type DeleteExceptionListSchemaDecoded = Omit<
RequiredKeepUndefined<t.TypeOf<typeof deleteExceptionListSchema>>,
'namespace_type'
> & {
namespace_type: NamespaceType;
};
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import * as t from 'io-ts';

import { id, list_id, valueOrUndefined } from '../common/schemas';
import { Identity, RequiredKeepUndefined } from '../../types';
import { RequiredKeepUndefined } from '../../types';

export const deleteListItemSchema = t.intersection([
t.exact(
Expand All @@ -20,5 +20,7 @@ export const deleteListItemSchema = t.intersection([
t.exact(t.partial({ id, list_id })),
]);

export type DeleteListItemSchemaPartial = Identity<t.TypeOf<typeof deleteListItemSchema>>;
export type DeleteListItemSchema = RequiredKeepUndefined<t.TypeOf<typeof deleteListItemSchema>>;
export type DeleteListItemSchema = t.OutputOf<typeof deleteListItemSchema>;
export type DeleteListItemSchemaDecoded = RequiredKeepUndefined<
t.TypeOf<typeof deleteListItemSchema>
>;
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,13 @@
import * as t from 'io-ts';

import { id } from '../common/schemas';
import { RequiredKeepUndefined } from '../../types';

export const deleteListSchema = t.exact(
t.type({
id,
})
);

export type DeleteListSchema = t.TypeOf<typeof deleteListSchema>;
export type DeleteListSchema = RequiredKeepUndefined<t.TypeOf<typeof deleteListSchema>>;
export type DeleteListSchemaEncoded = t.OutputOf<typeof deleteListSchema>;
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import * as t from 'io-ts';

import { list_id } from '../common/schemas';
import { RequiredKeepUndefined } from '../../types';

export const exportListItemQuerySchema = t.exact(
t.type({
Expand All @@ -17,5 +18,7 @@ export const exportListItemQuerySchema = t.exact(
})
);

export type ExportListItemQuerySchema = t.TypeOf<typeof exportListItemQuerySchema>;
export type ExportListItemQuerySchema = RequiredKeepUndefined<
t.TypeOf<typeof exportListItemQuerySchema>
>;
export type ExportListItemQuerySchemaEncoded = t.OutputOf<typeof exportListItemQuerySchema>;
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,19 @@
import { FILTER } from '../../constants.mock';

import {
FindEndpointListItemSchemaPartial,
FindEndpointListItemSchemaPartialDecoded,
FindEndpointListItemSchema,
FindEndpointListItemSchemaDecoded,
} from './find_endpoint_list_item_schema';

export const getFindEndpointListItemSchemaMock = (): FindEndpointListItemSchemaPartial => ({
export const getFindEndpointListItemSchemaMock = (): FindEndpointListItemSchema => ({
filter: FILTER,
page: '1',
per_page: '25',
sort_field: undefined,
sort_order: undefined,
});

export const getFindEndpointListItemSchemaDecodedMock = (): FindEndpointListItemSchemaPartialDecoded => ({
export const getFindEndpointListItemSchemaDecodedMock = (): FindEndpointListItemSchemaDecoded => ({
filter: FILTER,
page: 1,
per_page: 25,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import {
getFindEndpointListItemSchemaMock,
} from './find_endpoint_list_item_schema.mock';
import {
FindEndpointListItemSchemaPartial,
FindEndpointListItemSchema,
findEndpointListItemSchema,
} from './find_endpoint_list_item_schema';

Expand All @@ -29,7 +29,7 @@ describe('find_endpoint_list_item_schema', () => {
});

test('it should validate and empty object since everything is optional and has defaults', () => {
const payload: FindEndpointListItemSchemaPartial = {};
const payload: FindEndpointListItemSchema = {};
const decoded = findEndpointListItemSchema.decode(payload);
const checked = exactCheck(payload, decoded);
const message = pipe(checked, foldLeftRight);
Expand Down Expand Up @@ -98,7 +98,7 @@ describe('find_endpoint_list_item_schema', () => {
});

test('it should not allow an extra key to be sent in', () => {
const payload: FindEndpointListItemSchemaPartial & {
const payload: FindEndpointListItemSchema & {
extraKey: string;
} = { ...getFindEndpointListItemSchemaMock(), extraKey: 'some new value' };
const decoded = findEndpointListItemSchema.decode(payload);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,9 @@ export const findEndpointListItemSchema = t.exact(
})
);

export type FindEndpointListItemSchemaPartial = t.OutputOf<typeof findEndpointListItemSchema>;

// This type is used after a decode since some things are defaults after a decode.
export type FindEndpointListItemSchemaPartialDecoded = t.TypeOf<typeof findEndpointListItemSchema>;
export type FindEndpointListItemSchema = t.OutputOf<typeof findEndpointListItemSchema>;

// This type is used after a decode since some things are defaults after a decode.
export type FindEndpointListItemSchemaDecoded = RequiredKeepUndefined<
FindEndpointListItemSchemaPartialDecoded
>;

export type FindEndpointListItemSchema = RequiredKeepUndefined<
t.TypeOf<typeof findEndpointListItemSchema>
>;
Loading

0 comments on commit 18d251d

Please sign in to comment.