[PoC] Dashboard queries API

[drewdaemon]

Summary

This "move fast and break things" PoC adds an HTTP endpoint for reporting queries that will be run when a particular dashboard is loaded.

Example request

curl http://elastic:changeme@0.0.0.0:5601/ulo/api/dashboards/dashboard/b3367549-4b7c-468c-b565-a43268169d36/getQueries -H 'elastic-api-version: 2023-10-31'

Example response

{
   "panels": [
      {
         "title": "Records over time",
         "queries": [
            {
               "query": "from kibana_sample_data_logs | limit 10 | EVAL timestamp=DATE_TRUNC(30 second, @timestamp) | stats rows = count(*) by timestamp | rename timestamp as `@timestamp every 30 second`",
               "locale": "en",
               "filter": {
                  "bool": {
                     "must": [],
                     "filter": [],
                     "should": [],
                     "must_not": []
                  }
               }
            }
         ]
      },
      {
         "title": "Bytes by host",
         "queries": [
            {
               "aggs": {
                  "0": {
                     "terms": {
                        "field": "agent.keyword",
                        "order": {
                           "1": "desc"
                        },
                        "size": 2,
                        "shard_size": 25
                     },
                     "aggs": {
                        "1": {
                           "sum": {
                              "field": "bytes"
                           }
                        }
                     }
                  }
               },
               "size": 0,
               "fields": [
                  {
                     "field": "@timestamp",
                     "format": "date_time"
                  },
                  {
                     "field": "timestamp",
                     "format": "date_time"
                  },
                  {
                     "field": "utc_time",
                     "format": "date_time"
                  }
               ],
               "script_fields": {},
               "stored_fields": [
                  "*"
               ],
               "runtime_mappings": {
                  "hour_of_day": {
                     "type": "long",
                     "script": {
                        "source": "emit(doc['timestamp'].value.getHour());"
                     }
                  }
               },
               "_source": {
                  "excludes": []
               },
               "query": {
                  "bool": {
                     "must": [],
                     "filter": [],
                     "should": [],
                     "must_not": []
                  }
               }
            },
            {
               "aggs": {
                  "other-filter": {
                     "aggs": {
                        "1": {
                           "sum": {
                              "field": "bytes"
                           }
                        }
                     },
                     "filters": {
                        "filters": {
                           "": {
                              "bool": {
                                 "must": [],
                                 "filter": [
                                    {
                                       "exists": {
                                          "field": "agent.keyword"
                                       }
                                    }
                                 ],
                                 "should": [],
                                 "must_not": [
                                    {
                                       "match_phrase": {
                                          "agent.keyword": "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1"
                                       }
                                    },
                                    {
                                       "match_phrase": {
                                          "agent.keyword": "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24"
                                       }
                                    }
                                 ]
                              }
                           }
                        }
                     }
                  }
               },
               "size": 0,
               "fields": [
                  {
                     "field": "@timestamp",
                     "format": "date_time"
                  },
                  {
                     "field": "timestamp",
                     "format": "date_time"
                  },
                  {
                     "field": "utc_time",
                     "format": "date_time"
                  }
               ],
               "script_fields": {},
               "stored_fields": [
                  "*"
               ],
               "runtime_mappings": {
                  "hour_of_day": {
                     "type": "long",
                     "script": {
                        "source": "emit(doc['timestamp'].value.getHour());"
                     }
                  }
               },
               "_source": {
                  "excludes": []
               },
               "query": {
                  "bool": {
                     "must": [],
                     "filter": [],
                     "should": [],
                     "must_not": []
                  }
               }
            }
         ]
      }
   ]
}

Video

Screen.Recording.2023-12-14.at.11.43.43.AM.mov

Technical analysis

Key files

• Route contains logic to extract dashboard panel states and invoke the query collection routine.
• Lens setup method actually bootstraps the datasources, builds and runs the expressions for each visualization layer, and collects the requests from the search service via the RequestAdapter.
• Text-based and form-based common datasource classes.

Getting Lens code to run on the server

The most time-consuming part of this was preparing the Lens code that creates the expressions to run on the server. Currently, Lens liberally mixes browser-dependent code with env-agnostic code. For example, the datasource and visualization classes, as well as all the operation definitions for the form-based datasource contain expression building logic, state manipulation, and UI components as methods.

The most low-impact way I found to make some of this logic available on the server was to define paired-down interfaces in the common context that only include the things that need to be used server-side and client-side. For example, DatasourceCommon looks like this:

/**
 * A subset of datasource methods used on both client and server
 */
export interface DatasourceCommon<T = unknown, P = unknown> {
  id: string;

  initialize: (
    state?: P,
    savedObjectReferences?: SavedObjectReference[],
    initialContext?: VisualizeFieldContext | VisualizeEditorContext,
    indexPatternRefs?: IndexPatternRef[],
    indexPatterns?: IndexPatternMap
  ) => T;

  getLayers: (state: T) => string[];

  toExpression: (
    state: T,
    layerId: string,
    indexPatterns: IndexPatternMap,
    dateRange: DateRange,
    nowInstant: Date,
    searchSessionId?: string
  ) => ExpressionAstExpression | string | null;
}

Then, the existing client-side interfaces can inherit from the common versions (on the implementation side, a spread operator does the trick).

This pattern could be repeated for the form-based datasource's operators.

Other panel types

Each panel type has a distinct architecture and will require a distinct amount of effort (e.g. extracting the query from Vega visualizations is trivial since it is stored in the configuration itself).

In the context of integrations, Lens is by-far the most important panel to support. However, it would be nice to expand. By usage, saved searches are the number-2 panel out of those that initiate a request.

Current PoC Limitations

None of these limitations are fundamental. They simply exist because I ran out of time. I expect most of them to be quite simple to resolve.

• Only Lens is currently supported
  • Only terms and metric-type functions are supported in form-based visualizations
  • Requests from query-based annotations aren't yet recorded
• Only by-value panels are currently supported
• Inherited dashboard context (filters, queries) is not currently taken into account. This is needed for any serious evaluation of dashboard performance.
• The client is borked :D — I didn't pay much attention to broken imports when I moved things around
• The search service currently runs the requests. It should be prevented from doing so via a flag, etc.
• Hosting the route in the dashboard app creates a circular dependency with Lens (probably other panel types, too). Probably makes sense to host the route it its own plugin. no longer true as of [Lens] Cleanup, removes dashboard plugin dependency #179245