Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build EQL Query Parser and planner on top of reusable xQL components #49997

Closed
13 of 15 tasks
costin opened this issue Dec 9, 2019 · 4 comments
Closed
13 of 15 tasks

Build EQL Query Parser and planner on top of reusable xQL components #49997

costin opened this issue Dec 9, 2019 · 4 comments
Assignees
Labels

Comments

@costin
Copy link
Member

costin commented Dec 9, 2019

Meta issue describing the rough steps for reusing the SQL
common core inside EQL.

Parsing & AST

  • Make EQL parser take advantage of the new Node and Source packages
  • Create EQL specific exceptions based on the common ones to properly catch
  • Hook parser to the Expression tree and map WHERE query to a Filter

Session & Analysis

  • Create dedicated Analyzer for EQL using a subset of rules from SQL. In particular those around CTE and missing references should not be needed. It's worth starting with a small set and expand later if needed.
  • Pay special attention to the index resolution. As oppose to SQL, the index is resolved before the query so the index information EsIndex is already available.
    In practice this translates to a simpler PlanExecutor, that is the entry point in converting the query into an actual plan.
  • Start defining a Session and Configuration class - while it shares a lot of things, for the most part this can be EQL specific since the risk of duplication is low (quantity wise as well).
  • Add dedicated Verifier - just like with the Analyzer this will likely have a smaller set of rule. As a tangent, it is a good opportunity to potentially add some generic ones such as verifying that the input of a node is at most a subset of its child.
    This can act as a good deterrent in catching potential bugs that work across nodes.

Optimizer

  • Add EQL Optimizer - for the most part it will contain the common rules around Expression as most LogicalPlan ones are not useful at this stage.
  • It might be worth adding a basic second Verifier to check that the Optimizer hasn't introduced any bugs

Query translator

  • Move querying AST in QL
  • Extract basic rules for translating search queries (not aggregations) to QL

Query Planner

  • As the first release handles only basic queries, a trivial/basic query planner will be used (there's no planning required).

Query Executor

  • Create result representation across different response types
  • Reuse SearchHitExtractor from QL for parsing result set
  • Introduce search_after querying. This will be useful in the future for sequence/joins. Investigate whether serialization is required for the first stage.

Relates to #49773

@costin costin added the :Analytics/EQL EQL querying label Dec 9, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (:Search/EQL)

@costin costin added the :Analytics/SQL SQL querying label Dec 9, 2019
@costin costin added the Meta label Dec 9, 2019
@colings86 colings86 changed the title EQL: Meta-issue around xQL common components Build EQL Query Parser and planner on top of reusable xQL components Dec 10, 2019
@costin
Copy link
Member Author

costin commented Jan 13, 2020

See #50904

@costin
Copy link
Member Author

costin commented Jan 23, 2020

#51328 handles the extensibility of DataType, type conversion and pluggable operators.

@costin
Copy link
Member Author

costin commented Feb 27, 2020

Closing this as the transpiler is complete and EQL now executes queries against Elasticsearch. Any further issues will be tracked separately.

@costin costin closed this as completed Feb 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants