From 9c30cc4d5503eb7543953e588b34728119ecd897 Mon Sep 17 00:00:00 2001 From: Nik Everett Date: Sun, 20 Aug 2023 13:22:12 -0400 Subject: [PATCH] ESQL: Instructions for functions Adds package javadoc for ESLQ's `function.scalar` package with instructions on how to write a function. --- .../function/scalar/package-info.java | 154 ++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/package-info.java diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/package-info.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/package-info.java new file mode 100644 index 0000000000000..8999726b25510 --- /dev/null +++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/package-info.java @@ -0,0 +1,154 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0; you may not use this file except in compliance with the Elastic License + * 2.0. + */ + +/** + * Functions that take a row of data and produce a row of data without holding + * any state between rows. This includes both the {@link org.elasticsearch.xpack.ql.expression.function.scalar.ScalarFunction} + * subclass to link into the QL infrastucture and the {@link org.elasticsearch.compute.operator.EvalOperator.ExpressionEvaluator} + * implementation to run the actual function. + * + *

Guide to adding new function

+ *

+ * Adding functions is fairly easy and should be fun! + * This is a step by step list of how to do it. + *

+ *
    + *
  1. Fork the Elasticsearch repo.
    2. + *
  2. Clone your fork locally.
    3. + *
  3. Add Elastic's remote, it should look a little like: + * {@code
    + * [remote "elastic"]
+ * url = git@github.com:elastic/elasticsearch.git
+ * fetch = +refs/heads/*:refs/remotes/elastic/*
+ * [remote "nik9000"]
+ * url = git@github.com:nik9000/elasticsearch.git
+ * fetch = +refs/heads/*:refs/remotes/nik9000/*
+ *
    } + *
    4. + *
  4. + * Feel free to use {@code git} as a scratch pad. We're going to squash all commits + * before merging and will only keep the PR subject line and description in the + * commit message. + *
    5. + *
  5. + * Open Elasticsearch in IntelliJ. + *
    6. + *
  6. + * Open {@code x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/CsvTests.java} + * and run it. IntelliJ will take a few minutes to compile everything but the test itself + * should take only a few seconds. This is a fast path to running ESQL's integration tests. + *
    7. + *
  7. + * Pick one of the csv-spec files in {@code x-pack/plugin/esql/qa/testFixtures/src/main/resources/} + * and add a test for the function you want to write. These files are roughly themed but there + * isn't a strong guiding principle in the theme. + *
    8. + *
  8. + * Rerun the {@code CsvTests} and watch your new test fail. Yay, TDD doing it's job. + *
    9. + *
  9. + * Find a function in this package similar to the one you are working on and copy it to build + * yours. There's come ceremony required in each function class to make it constant foldable + * and return the right types. Take a stab at these, but don't worry too much about getting + * it right. + *
    10. + *
  10. + * There are also methods annotated with {@link org.elasticsearch.compute.ann.Evaluator} + * that contain the actual inner implementation of the function. Modify those to look right + * and click {@code Build->Recompile 'FunctionName.java'} in IntelliJ. This should generate + * an {@link org.elasticsearch.compute.operator.EvalOperator.ExpressionEvaluator} implementation + * calling the method annotated with {@link org.elasticsearch.compute.ann.Evaluator}. + *
  11. + * Once your evaluator is generated you can implement {@link org.elasticsearch.xpack.esql.planner.Mappable#toEvaluator}, + * having it return the generated evaluator. + *
    12. + *
  12. + * Add your function to {@link org.elasticsearch.xpack.esql.expression.function.EsqlFunctionRegistry}. + * This links it into the language and {@code SHOW FUNCTIONS}. Also add your function to + * {@link org.elasticsearch.xpack.esql.io.stream.PlanNamedTypes}. This makes your function + * serializable over the wire. Mostly you can copy existing implementations for both. + *
    13. + *
  13. + * Rerun the {@code CsvTests}. They should find your function and maybe even pass. Add a + * few more tests in the csv-spec tests. They run quickly so it isn't a big deal having + * half a dozen of them per function. + *
    14. + *
  14. + * Now it's time to make a unit test! The infrastructure for these is under some flux at + * the moment, but it's good to extend from {@code AbstractScalarFunctionTestCase}. All of + * these tests are parameterized and expect to spend some time finding good parameters. + *
    15. + *
  15. + * Once you are happy with the tests run the auto formatter: + * {@code
    ./gradlew -p x-pack/plugin/esql/ spotlessApply
    } + *
    16. + *
  16. + * Now you can run all of the ESQL tests liks CI: + * {@code
    ./gradlew -p x-pack/plugin/esql/ check
    } + *
    17. + *
  17. + * Now it's time to write some docs! Open {@code docs/reference/esql/esql-functions.asciidoc} + * and add your function in alphabetical order to the list at the top and then add it to + * the includes below. + *
    18. + *
  18. + * Now go make a file to include. You can start by copying one of it's neighbors. + *
    19. + *
  19. + * It's important that any examples you add to the docs be included from the csv-spec file. + * That looks like: + * {@code
    + * [source.merge.styled,esql]
+ * ----
+ * include::{esql-specs}/math.csv-spec[tag=mv_min]
+ * ----
+ * [%header.monospaced.styled,format=dsv,separator=|]
+ * |===
+ * include::{esql-specs}/math.csv-spec[tag=mv_min-result]
+ * |===
+ *
    } + * This includes the bit of the csv-spec file fenced by {@code // tag::mv_min[]}. You'll + * want a fence descriptive for your function. Consider the non-includes lines to be + * asciidoc ceremony to make the result look right in the rendered docs. + *
    20. + *
  20. + * Build the docs by cloning the docs repo + * and running: + * {@code
    + * ../docs/build_docs --doc docs/reference/index.asciidoc --resource x-pack/docs/ --open --chunk 1
+ *
    } + * from the elasticsearch directory. The first time you run the docs build it does a bunch + * of things with docker to get itself ready. Hopefully you can sit back and watch the show. + * It won't need to do it a second time unless some poor soul updates the Dockerfile in the + * docs repo. + *
    21. + *
  21. + * When it finishes building it'll open a browser window. Go to the + * functions page to see your + * function in the list and follow it's link to get to the page you built. Make sure it + * looks ok. + *
    22. + *
  22. + * Open the PR. The subject and description of the PR are important because those'll turn + * into the commit message we see in the commit history. Good PR descriptions make me very + * happy. But functions don't need an essay. + *
    23. + *
  23. + * Add the {@code >enhancement} and {@code :Query Languages/ES|QL} tags if you are able. + * Request a review if you can, probably from one of the folks that github proposes to you. + *
    24. + *
  24. + * CI might fail for random looking reasons. The first thing you should do is merge {@code main} + * into your PR branch. That's usually just: + * {@code
    + * git checkout main && git pull elastic main && git checkout mybranch && git merge main
+ *
    } + * Don't worry about the commit message. It'll get squashed away in the merge. + *
    25. + *
+ */ +package org.elasticsearch.xpack.esql.expression.function.scalar;