Query Library transfer script and cookbook article

docs/Home.md

@@ -19,6 +19,7 @@ effective use of the Brim desktop application and related tools.
 ## Cookbooks
 
 - [[Remote zqd]]
+- [[Query Library Transfer]]
 
 ## Developer Resources
 

docs/Query-Library-Transfer.md

@@ -0,0 +1,188 @@
+# Query Library Transfer
+
+- [Summary](#summary)
+- [About Cookbooks](#about-cookbooks)
+- [Limitations](#limitations)
+- [Background: `appState.json`](#background-appstatejson)
+- [Extracting/Replacing the Query Library](#extractingreplacing-the-query-library)
+- [Contact us!](#contact-us)
+
+# Summary
+
+As of Brim release [v0.22.0](https://github.com/brimsec/brim/releases/tag/v0.22.0),
+the contents of the Query Library are saved as part of the local app "state"
+alongside other persistent data such as the saved entries in the **History**
+panel and user **Preferences**. There are plans for future enhancements to
+allow Query Library contents to be shared directly between users. While
+awaiting these enhancements, this cookbook describes an approach that enables
+the manual transfer of Query Library contents between desktops in the current
+implementation.
+
+# About Cookbooks
+
+Brim cookbooks provide an opportunity to "test drive" new/experimental
+features in the Brim application and related [zq](https://github.com/brimsec/zq)
+tools. They also walk through details of how Brim and zq tools function and
+therefore may inspire other creative configurations.
+
+All efforts are made to disclose known caveats and limitations that are
+relevant to the configurations shown. However, due to the potential to
+encounter bugs in evolving functionality, it is recommended that you initially
+follow cookbooks in a non-production, lab-style setting. As such features
+become more complete and stable, cookbooks may be retired and replaced with
+regular [User Documentation](https://github.com/brimsec/brim/wiki#user-documentation).
+
+Please report any bugs or usability issues you find when working with cookbooks
+by [opening an issue](https://github.com/brimsec/brim/wiki/Troubleshooting#opening-an-issue)
+or reaching out on the [Brim public Slack](https://www.brimsecurity.com/join-slack/).
+We'd also love to hear your success stories and variations, so please don't be
+shy!
+
+# Limitations
+
+The approach described below effectively involves replacing the _entire_
+contents of the Query Library, which would be appropriate for a use case such
+as replacing an "out-of-the-box" Query Library with a customized one. No
+attempts are made to cover more sophisticated operations such as individual
+add/delete of entries.
+
+# Background: `appState.json`
+
+Brim maintains persistent user configuration in a file `appState.json` which
+is located in the following base directory on each platform:
+
+|**OS Platform**|**Location**|
+|---------------|------------|
+| **Windows**   | `%APPDATA%\Brim`                         |
+| **macOS**     | `$HOME/Library/Application Support/Brim` |
+| **Linux**     | `$HOME/.config/Brim`                     |
+
+If we peek inside it with a [JSON browser](http://jsonviewer.stack.hu/) or a
+tool like [`jq`](https://stedolan.github.io/jq/), we can see where different
+types of configuration are kept. For example, here we can see evidence of a
+change having been made to the presentation of time values in the
+**Preferences** settings.
+
+```
+$ cat appState.json | jq '.data.globalState.prefs'
+{
+  "jsonTypeConfig": "",
+  "timeFormat": "dddd, MMMM Do YYYY, h:mm:ss a",
+  "suricataRunner": "",
+  "suricataUpdater": "",
+  "zeekRunner": "",
+  "dataDir": ""
+}
+```
+
+And as is relevant for this article, here's the location of the Query Library
+entries.
+
+```
+$ cat appState.json | jq '.data.globalState.queries'
+{
+  "id": "root",
+  "name": "root",
+  "items": [
+    {
+      "id": "1",
+      "name": "Activity Overview",
+      "value": "count() by _path | sort -r",
+      "description": "This query shows a list of all Zeek streams in the data set, with a count of associated records",
+      "tags": [
+        "zeek",
+        "initial exploration"
+      ]
+    },
+...
+```
+
+# Extracting/Replacing the Query Library
+
+For ease of use, we've created a simple Python script
+[`qlib_util.py`](https://raw.githubusercontent.com/brimsec/brim/master/scripts/util/qlib_util.py)
+that can easily perform this.
+
+```
+$ python3 qlib_util.py --help
+usage: qlib_util.py [-h] (--extract EXTRACT | --replace REPLACE)
+                    [--statepath STATEPATH] [--backup]
+
+Extract/restore Query Library entries from Brim's appState.json
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --extract EXTRACT, -o EXTRACT
+                        Extract Query Library entries from appState.json into
+                        the specified output file (default: None)
+  --replace REPLACE, -r REPLACE
+                        Replace Query Library entries in appState.json with
+                        those from the specified input file (default: None)
+  --statepath STATEPATH, -s STATEPATH
+                        Pathname of appState.json file (default:
+                        ./appState.json)
+  --backup, -b          Backup appState.json before modifying it (default:
+                        True)
+```
+
+Let's walk through an example of its use on macOS. Once we've completely exited
+Brim, we download the script to the base config directory where our
+`appState.json` is located.
+
+```
+$ cd "$HOME/Library/Application Support/Brim"
+$ curl -O https://raw.githubusercontent.com/brimsec/brim/master/scripts/util/qlib_util.py
+```
+
+To extract the contents of the Query Library to a separate file `saved.json`:
+
+```
+$ python3 qlib_util.py --extract saved.json
+Successfully extracted Query Library from ./appState.json to saved.json
+```
+
+Inside the file, we can see a custom entry created previously.
+
+```
+$ cat saved.json 
+{
+    "id": "root",
+    "items": [
+...
+        {
+            "description": "My custom query",
+            "id": "0c7cda6670",
+            "name": "Super severe alerts",
+            "tags": [
+                "suricata"
+            ],
+            "value": "alert.severity < 2"
+        }
+    ],
+    "name": "root"
+}
+```
+
+Now let's sey we've got a separate system where another Brim user wants this
+to become their Query Library so they'll have this custom query as well. We
+transfer the `saved.json` file to that system, ensure Brim has been completely
+exited, and with the script downloaded as before:
+
+```
+$ cd "$HOME/Library/Application Support/Brim"
+$ curl -O https://raw.githubusercontent.com/brimsec/brim/master/scripts/util/qlib_util.py
+$ python3 qlib_util.py --replace saved.json 
+Backing up ./appState.json to /var/folders/yn/jbkxxkpd4vg142pc3_bd_krc0000gn/T/tmpnnfr164v
+Successfully replaced the Query Library in ./appState.json with the contents of saved.json
+```
+
+Relaunching Brim on the destination system, we can see our custom entry is
+present.
+
+![Query Library Replaced](media/Query-Library-replaced.png)
+
+# Contact us!
+
+If you have questions or feedback about this cookbook, we'd like to hear from
+you! Please join our [public Slack](https://www.brimsecurity.com/join-slack/) or
+[open an issue](https://github.com/brimsec/brim/wiki/Troubleshooting#opening-an-issue). Thanks!

docs/_Sidebar.md

@@ -14,6 +14,7 @@
 **Cookbooks**
 
 - [[Remote zqd]]
+- [[Query Library Transfer]]
 
 **Developer Resources**
 

scripts/util/qlib_util.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+
+import sys
+import os
+import json
+import argparse
+import tempfile
+import shutil
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='Extract/restore Query Library entries from Brim\'s appState.json',
+                                     formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    group = parser.add_mutually_exclusive_group(required=True)
+    group.add_argument('--extract', '-o', help='Extract Query Library entries from appState.json into the specified output file')
+    group.add_argument('--replace', '-r', help='Replace Query Library entries in appState.json with those from the specified input file')
+    parser.add_argument('--statepath', '-s', help='Pathname of appState.json file', default=os.path.join('.', 'appState.json'))
+    parser.add_argument('--backup', '-b', help='Backup appState.json before modifying it', action='store_false')
+    return parser.parse_args()
+
+if __name__ == '__main__':
+    args = parse_args()
+
+    if args.extract:
+        with open(args.statepath, 'rt') as f:
+            queries = json.load(f)['data']['globalState']['queries']
+            with open(args.extract, 'wt') as q:
+                json.dump(queries, q, sort_keys=True, indent=4)
+                print('Successfully extracted Query Library from ' + args.statepath + ' to ' + args.extract)
+
+    elif args.replace:
+        with open(args.replace, 'rt') as q:
+            queries = json.load(q)
+            if args.backup:
+                b = tempfile.NamedTemporaryFile(delete=False)
+                print('Backing up ' + args.statepath + ' to ' + b.name)
+                shutil.copyfile(args.statepath, b.name)
+                b.close()
+            with open(args.statepath, 'rt+') as f:
+                state = json.load(f)
+                state['data']['globalState']['queries'] = queries
+                f.seek(0)
+                json.dump(state, f)
+                f.truncate()
+                print('Successfully replaced the Query Library in ' + args.statepath + ' with the contents of ' + args.replace)