Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Lens] Time offset using filters #95635

Closed
wants to merge 6 commits into from

Conversation

flash1293
Copy link
Contributor

Rough POC

Create a chart in Lens with a count metric with label "shifted" - it will get shifted by one day

Mechanism

This works by doing the "base request" for the actually selected time range first, then serializing the bucket hierarchy into a flat list of filters (which time shifts applied) and doing a second request to fetch the shifted time:

Configuration:
Screenshot 2021-03-29 at 11 27 45

First request:

{
    "index": "kibana_sample_data_logs",
    "body": {
        "aggs": {
            "2bc261dc-92c1-48f9-b117-af885fb6539c": {
                "date_histogram": {
                    "field": "@timestamp",
                    "calendar_interval": "1d",
                    "time_zone": "Europe/Berlin"
                },
                "aggs": {
                    "fa2b4f03-c718-4131-8d4b-8fe69e779abe": {
                        "terms": {
                            "field": "extension.keyword",
                            "order": {
                                "_count": "desc"
                            },
                            "size": 2
                        }
                    }
                }
            }
        },
        "query": {
            "bool": {
                "must": [],
                "filter": [
                    {
                        "match_all": {}
                    },
                    {
                        "range": {
                            "@timestamp": {
                                "gte": "2021-03-25T23:00:00.000Z",
                                "lte": "2021-03-27T23:00:00.000Z",
                                "format": "strict_date_optional_time"
                            }
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    },
}

First response:

{
    "aggregations": {
        "2bc261dc-92c1-48f9-b117-af885fb6539c": {
            "buckets": [
                {
                    "key_as_string": "2021-03-26T00:00:00.000+01:00",
                    "key": 1616713200000,
                    "doc_count": 230,
                    "fa2b4f03-c718-4131-8d4b-8fe69e779abe": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 113,
                        "buckets": [
                            {
                                "key": "",
                                "doc_count": 79
                            },
                            {
                                "key": "css",
                                "doc_count": 38
                            }
                        ]
                    }
                },
                {
                    "key_as_string": "2021-03-27T00:00:00.000+01:00",
                    "key": 1616799600000,
                    "doc_count": 230,
                    "fa2b4f03-c718-4131-8d4b-8fe69e779abe": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 102,
                        "buckets": [
                            {
                                "key": "",
                                "doc_count": 88
                            },
                            {
                                "key": "gz",
                                "doc_count": 40
                            }
                        ]
                    }
                }
            ]
        }
    }
}

Time-shift request (additional shifts would happen in parallel)

{
    "index": "kibana_sample_data_logs",
    "body": {
        "aggs": {
            "rootSerialization": {
                "filters": {
                    "filters": {
                        "serialized-row-0": {
                            "bool": {
                                "must": [],
                                "filter": [
                                    {
                                        "bool": {
                                            "filter": [
                                                {
                                                    "range": {
                                                        "@timestamp": {
                                                            "gte": "2021-03-24T23:00:00.000Z",
                                                            "lte": "2021-03-25T23:00:00.000Z"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match_phrase": {
                                                        "extension.keyword": ""
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "should": [],
                                "must_not": []
                            }
                        },
                        "serialized-row-1": {
                            "bool": {
                                "must": [],
                                "filter": [
                                    {
                                        "bool": {
                                            "filter": [
                                                {
                                                    "range": {
                                                        "@timestamp": {
                                                            "gte": "2021-03-24T23:00:00.000Z",
                                                            "lte": "2021-03-25T23:00:00.000Z"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match_phrase": {
                                                        "extension.keyword": "css"
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "should": [],
                                "must_not": []
                            }
                        },
                        "serialized-row-2": {
                            "bool": {
                                "must": [],
                                "filter": [
                                    {
                                        "bool": {
                                            "filter": [
                                                {
                                                    "range": {
                                                        "@timestamp": {
                                                            "gte": "2021-03-25T23:00:00.000Z",
                                                            "lte": "2021-03-26T23:00:00.000Z"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match_phrase": {
                                                        "extension.keyword": ""
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "should": [],
                                "must_not": []
                            }
                        },
                        "serialized-row-3": {
                            "bool": {
                                "must": [],
                                "filter": [
                                    {
                                        "bool": {
                                            "filter": [
                                                {
                                                    "range": {
                                                        "@timestamp": {
                                                            "gte": "2021-03-25T23:00:00.000Z",
                                                            "lte": "2021-03-26T23:00:00.000Z"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match_phrase": {
                                                        "extension.keyword": "gz"
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "should": [],
                                "must_not": []
                            }
                        }
                    }
                }
            }
        },
        "query": {
            "bool": {
                "must": [],
                "filter": [
                    {
                        "match_all": {}
                    },
                    {
                        "range": {
                            "@timestamp": {
                                "gte": "2021-03-24T23:00:00.000Z",
                                "lte": "2021-03-26T23:00:00.000Z",
                                "format": "strict_date_optional_time"
                            }
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    },
}

Time shift response

{
    "aggregations": {
        "rootSerialization": {
            "buckets": {
                "serialized-row-0": {
                    "doc_count": 84
                },
                "serialized-row-1": {
                    "doc_count": 31
                },
                "serialized-row-2": {
                    "doc_count": 79
                },
                "serialized-row-3": {
                    "doc_count": 38
                }
            }
        }
    }
}

Limitations

  • Other bucket doesn't work because of agg config internals (easily fixable)
  • Only date histogram and terms implemented so far (easy to extend to histogram, filters, ...)
  • Parent pipeline aggs won't work at all (because the are only allowed on date histograms / histograms
    • No way around that
    • Wouldn't be a problem for Lens because we don't expose them anyway
  • It's not possible to sort by a shifted metric
    • this is partially fixable by making the shift we want to sort by the base request
    • It's not easily fixable if different sorts are operating on different shifts
    • We could also leave it up to the user to change the time range and use a different shift, so the metric they want to sort by becomes the one of the actual time range
  • This requires two requests in series (which means it doesn't work well with send-to-background, something we might be able to fix Kibana side)
  • If there's an "other" bucket, it requires three requests in series

[skip ci]

@flash1293
Copy link
Contributor Author

Discussed with @ppisljar how this could get implemented:

  • Add time shift property to all metrics
  • Add time shift property to createFilter (not used for everything except for date histogram and probably date range)
  • Introduce new field timeFilter - it's merged into the actual request like filter, but is shadowing existing timeFilter fields of parent search sources
    • This is used to set the time range, so we can override it
    • Just takes fields array and time range, builds the filter internally
  • Move other bucket stuff from search source into agg configs (global onPostFlightRequest hook)
  • Add time shift request building and response parsing into this hook as well
    • Will walk the response of the base request like other bucket logic, building large filters array
    • Sends out the filters request
    • Will walk the response same way as other bucket, putting the shifted metrics into the first response so tabify will catch them with the default mapping logic and constructure the final table as a last step

@flash1293
Copy link
Contributor Author

See #98781

@flash1293 flash1293 closed this May 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant