Merge pull request #36 from mdebarros/bug/#2135-ElasticSearch-ran-out…

…-of-disk-space

Added rollover policy, and updated readme to include how to manage the policy - fixes for mojaloop/project#2135

README.md

@@ -1,52 +1,54 @@
 # event-stream-processor
-** EXPERIMENTAL** Event Stream Processor for Event Stream (logs, audits, errors, trace, etc)
+** EXPERIMENTAL** Event Stream Processor (ESP) for Event Stream (logs, audits, errors, trace, etc)
 
 ## 1. Pre-requisites
 
 ### 1.1 Elasticsearch
 
-Ensure that you have created the `mojatemplate` based on the following config: [template](./config/template-mojaloop.json).
+Ensure that you have created the following resources prior to deploying the ESP component:
+1. [Create a Policy](#1111-create) with the desired Rollover configurations: [policy-rollover-mojaloop.json](./config/policy-rollover-mojaloop.json)
+2. [Create a Template](#1121-create) to associate the Policy Settings to the Logstash index based on the following config: [template-mojaloop.json](./config/template-mojaloop.json)
 
-#### 1.1.1 Create Template
- ```curl
- curl -X PUT "http://elasticsearch:9200/_template/mojatemplate?pretty" -H 'Content-Type: application/json' -d @config/template-mojaloop.json'
- ```
+> __NOTE__:<br/>
+> If ESP component is running prior to configuring the above template, the index mapping will be auto-created and will not take effect.
+
+<br/>
+
+#### 1.1.1 Mojaloop Index Rollover Policy
+
+##### 1.1.1.1 Create
+```curl
+curl -X PUT "http://elasticsearch:9200/_ilm/policy/mojaloop_rollover_policy?pretty" -H 'Content-Type: application/json' -d @config/policy-rollover-mojaloop.json
+```
+
+##### 1.1.1.2 Delete
+_Note: only needed if you need to remove the policy_
+```curl
+curl -X DELETE "http://elasticsearch:9200/_ilm/policy/mojaloop_rollover_policy?"
+```
+
+#### 1.1.1.3 Get
+_Note: useful for debugging issues_
+```curl
+curl -X GET "http://elasticsearch:9200/_ilm/policy/mojaloop_rollover_policy?"
+```
+
+#### 1.1.2 Mojaloop Index Template
+
+##### 1.1.2.1 Create
+```curl
+curl -X PUT "http://elasticsearch:9200/_template/moja_template?pretty" -H 'Content-Type: application/json' -d @config/template-mojaloop.json'
+```
 
-#### 1.1.2 Delete Template
+##### 1.1.2.2 Delete
 _Note: only needed if you need to remove the template_
  ```curl
- curl -X DELETE "http://elasticsearch:9200/_template/mojatemplate"
+ curl -X DELETE "http://elasticsearch:9200/_template/moja_template"
  ```
 
  #### 1.1.3 Get Template
  _Note: useful for debugging template issues_
  ```curl
- curl -X GET "http://elasticsearch:9200/_template/mojatemplate"
+ curl -X GET "http://elasticsearch:9200/_template/moja_template"
  ```
 
- #### 1.1.4 Known Issues
-
-  1. Elasticsearch returns field type error when document is tried to be insserted. If a custom template is not presented into Elasticsearch, when the first document is inserted, Elasticsearch assumes data model and creates index schema and won't work correctly with mojaloop. 
-
-  Error returned by Elasticsearch:
-  ```json
-  {
-    "error": "Content-Type header [text/plain] is not supported",
-    "status": 406
-  }
-  ```
-
-  To fix the issue, you need to delete all old indexes. Even the index might not appear on the interface
-
-     1. Get indexes starting with mojaloop
-
-     ```curl
-     curl -X GET 'elasticsearch:9200/mojaloop*'
-     ```
-
-     2. Delete index by name
-
-     ```curl
-     curl -X DELETE 'elasticsearch:9200/mojaloop-2020.04.30'
-     ```
-

audit-resolve.json

@@ -147,6 +147,34 @@
       "decision": "ignore",
       "madeAt": 1588341057822,
       "expiresAt": 1588945834871
+    },
+    "1654|@mojaloop/central-services-health>@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>yargs>y18n": {
+      "decision": "fix",
+      "madeAt": 1617883185605
+    },
+    "1654|@mojaloop/central-services-shared>@mojaloop/event-sdk>grpc>protobufjs>yargs>y18n": {
+      "decision": "fix",
+      "madeAt": 1617883185606
+    },
+    "1654|@mojaloop/event-sdk>grpc>protobufjs>yargs>y18n": {
+      "decision": "fix",
+      "madeAt": 1617883185606
+    },
+    "1654|@mojaloop/central-services-health>@mojaloop/central-services-shared>widdershins>yargs>y18n": {
+      "decision": "postpone",
+      "madeAt": 1617883195120
+    },
+    "1654|@mojaloop/central-services-shared>widdershins>yargs>y18n": {
+      "decision": "postpone",
+      "madeAt": 1617883195120
+    },
+    "1500|@mojaloop/central-services-health>@mojaloop/central-services-shared>widdershins>yargs>yargs-parser": {
+      "decision": "postpone",
+      "madeAt": 1617883196166
+    },
+    "1500|@mojaloop/central-services-shared>widdershins>yargs>yargs-parser": {
+      "decision": "postpone",
+      "madeAt": 1617883196166
     }
   },
   "rules": {},

config/policy-rollover-mojaloop.json

@@ -0,0 +1,24 @@
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "actions": {
+          "rollover": {
+            "max_age": "30d",
+            "max_size": "20GB"
+          },
+          "set_priority": {
+            "priority": 100
+          }
+        },
+        "min_age": "0ms"
+      },
+      "delete": {
+        "min_age": "30d",
+        "actions": {
+          "delete": {}
+        }
+      }
+    }
+  }
+}

config/template-mojaloop.json

@@ -2,7 +2,9 @@
   "index_patterns": ["mojaloop*"],
   "settings": {
     "number_of_shards": 1,
-    "number_of_replicas": 1
+    "number_of_replicas": 1,
+    "index.lifecycle.name": "mojaloop_rollover_policy", 
+    "index.lifecycle.rollover_alias": "mojaloop_rollover_policy_alias" 
   },
   "mappings": {
     "_source": {