full stack development development and test environment with docker-c…

…ompose

README.md

@@ -7,23 +7,126 @@
 
 WIP: query TXN2 data by account, model and index pattern. Save queries and execute saved queries.
 
+## Local Development
 
-## Examples
+The project includes a Docker Compose file with Elasticsearch, Kibana and Cerebro:
+```bash
+docker-compose up
+```
+
+Add test account with [txn2/provision]:
+```bash
+curl -X POST \
+  http://localhost:8070/account \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "id": "test",
+    "description": "This is a test account",
+    "display_name": "Test Organization",
+    "active": true,
+    "access_keys": [
+        {
+            "name": "test",
+            "key": "PDWgYr3bQGNoLptBRDkLTGQcRmCMqLGRFpXoXJ8xMPsMLMg3LHvWpJgDu2v3LYBA",
+            "description": "Generic access key 2",
+            "active": true
+        }
+    ],
+    "modules": [
+        "wx",
+        "data_science"
+    ]
+}'
+```
+
+Add an admin [User] with with [txn2/provision] given access to the test account:
+
+```bash
+curl -X POST \
+  http://localhost:8070/user \
+  -H 'Content-Type: application/json' \
+  -d '{
+	"id": "test",
+	"description": "Test User admin",
+	"display_name": "Test User",
+	"active": true,
+	"sysop": false,
+	"password": "eWidL7UtiWJABHgn8WA",
+	"sections_all": false,
+	"sections": ["api", "config", "data"],
+	"accounts": ["test"],
+	"admin_accounts": ["test"]
+}'
+```
+
+Get a user [Token] from [txn2/provision] with the [User]'s id and password:
+
+```bash
+TOKEN=$(curl -s -X POST \
+          http://localhost:8070/authUser?raw=true \
+          -d '{
+        	"id": "test_user",
+        	"password": "eWidL7UtiWJABHgn8WA"
+        }') && echo $TOKEN
+```
+
+Insert a sample [Model] into the **test** account using [txn2/tm] running on port **8085**:
 
-Start server:
 ```bash
-go run ./cmd/query.go --tokenKey=$TOKEN_KEY --esServer=http://elasticsearch:9200
+curl -X POST http://localhost:8085/model/test \
+ -H "Authorization: Bearer $TOKEN" \
+ -H 'Content-Type: application/json' \
+ -d '{
+  "machine_name": "some_metrics",
+  "display_name": "Some Metrics",
+  "description": "A sample model describing some metrics sent through rxtx",
+  "fields": [
+    {
+      "machine_name": "device_id",
+      "display_name": "Device ID",
+      "data_type": "keyword"
+    },
+    {
+      "machine_name": "random_number",
+      "display_name": "Random Number",
+      "data_type": "integer"
+    },
+    {
+      "machine_name": "another_number",
+      "display_name": "Another Number",
+      "data_type": "integer"
+    }
+  ]
+}'
 ```
 
-Run / Test a query:
+In Elastis search there is now a template `_template/test-data-some_metrics` for the test account describing [txn2/rxtx]/[txn2/rtbeat] inbound data matching index pattern `test-data-some_metrics-*`. Send some sample data to Elastic search through [txn2/rxtx] and wait for the batch interval (specified in the docker-compose) to complete:
+
 ```bash
 curl -X POST \
-  http://localhost:8080/run/xorg \
+  http://localhost:8090/rx/test/some_metrics/sample-data \
+  -H 'Content-Type: application/json' \
+  -d "{
+  \"device_id\": \"12345\",
+  \"random_number\": \"${RANDOM}\",
+  \"another_number\": 12345
+}"
+```
+
+The fields in the data sent to [txn2/rxtx] should match the fields described in the [txn2/tm] [Model]. Although the value for "random_number" is represented here as a string, the template mapping added with the [Model] instructs Elasticsearch to index it as an integer.
+
+
+## Example Queries
+
+Upsert a [Query]:
+```bash
+curl -X POST \
+  http://localhost:8080/upsert/test \
   -H 'Authorization: Bearer $TOKEN' \
   -d '{
     "machine_name": "all_los_angeles_parking_citations",
     "display_name": "All Los Angeles Parking Citations",
-    "description_brief": "Gets all Los Angeles parking citation records available.",
+    "description_brief": "Gets all Los Angeles parking citation records available",
     "description": "This is a dataset hosted by the city of Los Angeles. The organization has an open data platform found [here](https://data.lacity.org/)",
     "query_class": "table",
     "model": "los_angeles_parking_citations",
@@ -36,15 +139,15 @@ curl -X POST \
 }'
 ```
 
-Upsert a query:
+Run / Test a [Query]:
 ```bash
 curl -X POST \
-  http://localhost:8080/upsert/xorg \
+  http://localhost:8080/run/test \
   -H 'Authorization: Bearer $TOKEN' \
   -d '{
     "machine_name": "all_los_angeles_parking_citations",
     "display_name": "All Los Angeles Parking Citations",
-    "description_brief": "Gets all Los Angeles parking citation records available",
+    "description_brief": "Gets all Los Angeles parking citation records available.",
     "description": "This is a dataset hosted by the city of Los Angeles. The organization has an open data platform found [here](https://data.lacity.org/)",
     "query_class": "table",
     "model": "los_angeles_parking_citations",
@@ -60,7 +163,7 @@ curl -X POST \
 Search for queries:
 ```bash
 curl -X POST \
-  http://localhost:8080/search/xorg \
+  http://localhost:8080/search/test \
   -H 'Authorization: Bearer $TOKEN' \
   -d '{
   "size": 10,
@@ -77,6 +180,15 @@ curl -X GET \
   -H 'Authorization: Bearer $TOKEN'
 ```
 
+[Token]: https://github.com/txn2/token
+[txn2/provision]: https://github.com/txn2/provision
+[txn2/tm]: https://github.com/txn2/tm
+[txn2/rtbeat]: https://github.com/txn2/tm
+[txn2/rxtx]: https://github.com/txn2/rxtx
+[User]: https://godoc.org/github.com/txn2/provision#User
+[Query]: https://godoc.org/github.com/txn2/query#Query
+[Model]: https://godoc.org/github.com/txn2/tm#Model
+
 ## Release Packaging
 
 Build test release:

cmd/query.go

@@ -1,15 +1,14 @@
-/*
-   Copyright 2019 txn2
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-       http://www.apache.org/licenses/LICENSE-2.0
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
-*/
+// Copyright 2019 txn2
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//     http://www.apache.org/licenses/LICENSE-2.0
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 package main
 
 import (

dev_cfg/logstash.conf

@@ -0,0 +1,23 @@
+input {
+  beats {
+    port => 5044
+  }
+}
+
+filter {
+  mutate {
+    gsub => ["[rxtxMsg][label]", "^/", ""]
+    split => {"[rxtxMsg][label]" => "/"}
+  }
+}
+
+output {
+
+  if [fields][idx] == "ts" {
+    elasticsearch {
+        hosts => [ "elasticsearch:9200" ]
+        index => "%{[rxtxMsg][producer]}-data-%{[rxtxMsg][key]}-ts-%{+YYYY.MM.dd}"
+    }
+  }
+
+}

dev_cfg/rtbeat.yml

@@ -0,0 +1,20 @@
+rtbeat:
+  # HTTP server port
+  port: "8081"
+  # timeout
+  timeout: 5
+
+output.logstash:
+  enabled: true
+  # beats listener configured in logstash.conf
+  hosts: ["logstash:5044"]
+
+# internal metrics through a HTTP endpoint
+http.enabled: true
+http.port: 5066
+
+# used in the logstash pipeline to infer time-series
+# data without the use of a specific document id and
+# using -ts- in the index name.
+fields:
+  idx: ts