Jmespath extensions

Apart from the jmespath functionality documented on their site some extensions had to be added:

1. Support for json.Number values as the current implementation does not support it.

codon (and swagger) parses json from incoming HTTP responses using the UseNumber() directive of the decoder. It does so to preserve the number format of an upstream response, otherwise all numbers will converted to float and be returned downstream as float. jmespath does not support operations on json.Number types natively.

2. Support for json meta tags in for struct objects.

If Json has been unmarshaled into a struct instead of a map, you can use json meta tag of a child to refer to it in jmespath. This functionality was initially added because swagger decodes into structs for validation. However swagger templates were changed to use mapstructure to get map from struct before jmespath operations. Implementation is O(n) for getting every tag. Needs improvement.

3. Added $ to refer to root node.

Sometimes you need to break out of local context (@) to refer to elements relative to the root node. Consider this json:

{
  "main": {
    "item_list": [
      {
        "item_id": 1,
        "item_val": "1"
      },
      {
        "item_id": 5,
        "item_val": "2"
      }
    ]
  },
  "item": 5
}

If you need to get an item with id 5 from the list main.item_list you can use jmespath expression main.item_list[?item_id==`5`]. But if you need to fetch this dynamically depending on value of item then you need to break out of local context inside the select: main.item_list[?item_id==$.item] (item_id can be directly referenced because it is relative to the current node). Support for $ was added because codon workflows store all the data in variable buckets which need to be accessed via jmespath.

4. dedup and dedup_by function to "de-duplicate" elements of an array.

This functions removes duplicate elements and returns an array with unique elements. A function countmap() might be better as dedup() would be equivalent of countmap().keys(). But that isnt implemented yet. Example json:

{
  "main": {
    "my_list": [1,2,6,2,7]
  }
}

The jmespath expression dedup(main.my_list) would return [1,2,6,7]. dedup_by does something similar but it operates on a list of objects (instead of strings or numbers) and dedups by an expression on the objects:

{
  "main": {
    "my_list": [
      {
        "id": 1
      },
      {
        "id": 1
      }
    ]
  }
}

dedup_by(main.my_list, &id) would evaluate to [{"id": 1}]

5. Support for & expressions as key when performing multi select hash

If you have a json as:

{"foo": "a", "bar": "b", "key": "my_key_name"}

you can construct a json object by using the jmespath query {newfoo: foo, newbar: bar}, where newfoo is the key name. But if you want a dynamic response based on the value of key you can not used the expression {&key: foo}. The expression &key is evaluated to "my_key_name" and the final result would be {"my_key_name": "a"}.

6. Functions zip, from_items and to_object which are still being discussed upstream here.

7. Function get to dynamically get value from a key expressions.

Useful for getting a value from an object by using a key from another expression. Consider this use case:

{"foo": "a", "bar": "b", "key": "bar"}

Lets say you need to construct a new object with only one key from the value of key and the corresponding value. That is, the output should be {"bar": "b"}. But if the json input was {"foo": "a", "bar": "b", "key": "foo"} the output should be {"foo": "a"}. The jmespath expression for this would be {&key: get(@, key)}. get(@, key) would get from current object the value of the key evaluated from key.

8. shuffle function which takes in an array and randomly shuffles the elements of the array and returns it.

Example json:

{
  "list": [1,2,3]
}

The jmespath expression shuffle(list) would return [1,3,2] or [2,1,3] and so on.

9. Custom functions: A layer on top of internal function caller of go-jmespath which allows the user of the library to add their own functions.

To use custom jmespath function inside a workflow the function must be registered with the library before the server is started itself. In any file in package restapi of your project, write your function:

import (
	jmespath "github.com/jmespath/go-jmespath"
	"errors"
)

func multiply(input []jmespath.Value, executor *jmespath.Executor) (interface{}, error) {
	if len(input) != 2 {
		return nil, errors.New("Incorrect number of arguments in multiply")
	}
	for i := 0; i < 2; i++ {
		if !input[i].IsNumber() {
			return nil, errors.New("Only numbers should be sent to multiply")
		}
	}

	return input[0].Float() * input[1].Float(), nil
}

input contains the list of arguments passed to the function. jmespath.Value is a struct type which is a wrapper on top of interface{} and reflect.Value and has some helper functions to validate and convert argument types. Since the addition of this function is dynamic, all the arguments will need to be validated and converted inside the function. len(input) != 2 asserts that the number of arguments is always 2. input[i].IsNumber() asserts that the argument is type number. input[i].Float() converts the input argument to float64.

The function should be registered before creating a compiler for an expression which uses the function. In go-codon we handle it by registering the function like:

var _ = jmespath.RegisterFunction("multiply", multiply)

during variable declaration phase.

go-codon makes sure that all the jmespath expressions are compiled after this call. It does so by creating the compiler object in init(). So do not register your function in an init() function. After registering this function, multiply should be available for use in any jmespath expression in the project.