Update documentation

x-pack/plugins/event_log/README.md

@@ -6,13 +6,23 @@ actitivies.
 ## Overview
 
 This plugin provides a persistent log of "events" that can be used by other
-plugins to record their processing, for later acccess.  Currently it's only
-used by the alerts and actions plugins.
+plugins to record their processing, for later accces. It is used by:
 
-The "events" are ECS documents, with some custom properties for Kibana, and
-alerting-specific properties within those Kibana properties.  The number of
-ECS fields is limited today, but can be extended fairly easily.  We are being
-conservative in adding new fields though, to help prevent indexing explosions.
+- `alerting` and `actions` plugins
+- [work in progress] `security_solution` (detection rules execution log)
+
+The "events" are [ECS documents](https://www.elastic.co/guide/en/ecs/current/index.html)
+containing both standard ECS fields and some custom fields for Kibana.
+
+- Standard fields are those which are defined in the ECS specification.
+  Examples: `@timestamp`, `message`, `event.provider`. The number of ECS fields
+  supported in Event Log is limited today, but can be extended fairly easily.
+  We are being conservative in adding new fields though, to help prevent
+  indexing explosions.
+- Custom fields are not part of the ECS spec. We defined a top-level `kibana`
+  field set where we have some Kibana-specific fields like `kibana.server_uuid`
+  and `kibana.saved_objects`. Plugins added a few custom fields as well,
+  for example `kibana.alerting` field set.
 
 A client API is available for other plugins to:
 
@@ -47,16 +57,25 @@ The structure of the event documents can be seen in the
 generated via a script when the structure changes.  See the
 [README.md](generated/README.md) for how to change the document structure.
 
-Below is an document in the expected structure, with descriptions of the fields:
+Below is a document in the expected structure, with descriptions of the fields:
 
 ```js
 {
+  // Base ECS fields.
+  // https://www.elastic.co/guide/en/ecs/current/ecs-base.html
   "@timestamp": "ISO date",
   tags: ["tags", "here"],
   message: "message for humans here",
+
+  // ECS version. This is set by the Event Log and should not be specified
+  // by a client of Event Log.
+  // https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html
   ecs: {
     version: "version of ECS used by the event log",
   },
+
+  // Event fields. All of them are supported.
+  // https://www.elastic.co/guide/en/ecs/current/ecs-event.html
   event: {
     provider: "see below",
     action: "see below",
@@ -65,21 +84,50 @@ Below is an document in the expected structure, with descriptions of the fields:
     end: "ISO date of end time for events that capture a duration",
     outcome: "success | failure, for events that indicate an outcome",
     reason: "additional detail on failure outcome",
+    // etc
   },
+
+  // Error fields. All of them are supported.
+  // https://www.elastic.co/guide/en/ecs/current/ecs-error.html
   error: {
     message: "an error message, usually associated with outcome: failure",
+    // etc
+  },
+
+  // Log fields. Only a subset is supported.
+  // https://www.elastic.co/guide/en/ecs/current/ecs-log.html
+  log: {
+    level: "info | warning | any log level keyword you need",
+    logger: "name of the logger",
+  },
+
+  // Rule fields. All of them are supported.
+  // https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
+  rule: {
+    author: ["Elastic"],
+    id: "a823fd56-5467-4727-acb1-66809737d943",
+    // etc
   },
+
+  // User fields. Only user.name is supported.
+  // https://www.elastic.co/guide/en/ecs/current/ecs-user.html
   user: {
     name: "name of Kibana user",
   }, 
-  kibana: { // custom ECS field
+
+  // Custom fields that are not part of ECS.
+  kibana: {
     server_uuid: "UUID of kibana server, for diagnosing multi-Kibana scenarios",
     alerting: {
       instance_id: "alert instance id, for relevant documents",
       action_group_id: "alert action group, for relevant documents",
       action_subgroup_id: "alert action subgroup, for relevant documents",
       status: "overall alert status, after alert execution",
     },
+    detection_engine: {
+      rule_status: "detection rule execution status, e.g. warning",
+      rule_status_severity: 42,
+    },
     saved_objects: [
       {
         rel: "'primary' | undefined; see below",

x-pack/plugins/event_log/generated/README.md

@@ -1,11 +1,26 @@
+# Generating event schema
+
 The files in this directory were generated by manually running the script
-../scripts/create-schemas.js from the root directory of the repository.
+`../scripts/create-schemas.js` from the root directory of the repository.
 
-These files should not be edited by hand.
+**These files should not be edited by hand.**
 
 Please follow the following steps:
-1. clone the [ECS](https://github.com/elastic/ecs) repo locally so that it resides along side your kibana repo, and checkout the ECS version you wish to support (for example, the `1.6` branch, for version 1.6)
-2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to make th efollowing changes:
-    1. Update `EcsKibanaExtensionsMappings` to include the mapping of the fields you wish to add.
-    2. Update `EcsEventLogProperties` to include the fields in the generated mappings.json.
-3. cd to the `kibana` root folder and run: `node ./x-pack/plugins/event_log/scripts/create_schemas.js`
+
+1. Clone the [ECS](https://github.com/elastic/ecs) repo locally so that it
+  resides along side your kibana repo, and checkout the ECS version you wish to
+  support (for example, the `1.8` branch, for version 1.8).
+
+2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to
+  make the following changes:
+    - Update `EcsCustomPropertyMappings` to include the mapping of the custom
+      fields you wish to add.
+    - Update `EcsPropertiesToGenerate` to include the fields in the generated
+      `mappings.json`.
+    - Make sure to list all array fields in `EcsEventLogMultiValuedProperties`.
+
+3. Cd to the `kibana` root folder and run:
+
+    ```sh
+    node ./x-pack/plugins/event_log/scripts/create_schemas.js
+    ```