Filter through command

[benatkin]

It would be nice to be able to filter each value in an array or object through a shell command

There is an issue in jq, which is popular but hasn't yet made it into the library. Perhaps it would be a lot easier to implement with golang :)

jqlang/jq#147

[itchyny]

You can use gojq.WithFunction to add a custom internal function. You can invoke shell command if you like to within your function implementation.

[benatkin]

Thanks. Also appreciate you closed it, I would rather a repo I use have few open issues than a bunch of out of scope feature requests. I'll blog if I do anything interesting with my newfound knowledge of WithFunction!

[noperator]

Hey @benatkin, did you end up using WithFunction to filter values through a shell command? Would love an example if you have one.

[wader]

@noperator here is an example of a function that runs a shell command #101 (comment) but as usual with shell and exec, be careful

[noperator]

That's perfect, thanks @wader. @lgfausak suggested in that same issue that one could:

compile in a new module which I have access to as a function from jq

If I understand that comment correctly, then it seems like I could:

1. Use gojq to compile a module providing a custom function (e.g., cmd())
2. Write that module to disk at ~/cmd.jq
3. Import that module into jq to invoke the custom function, like jq -L ~/cmd.jq '"hello" | cmd(["md5"])'

I'm going to play around with that to test my understanding of this whole process. If you happen to see any obvious gaps in that approach, please let me know.

[wader]

@noperator Your welcome. There is no support for loading "native" modules as such at runtime, instead you can for example build your own version of the gojq cli tool with that function added.

[webwurst]

Loading extensions at runtime is general not easy to achieve with golang? I think if possible it would be very nice to implement new or special features to gojq as an extension in a separate file. To provide a proof-of-concept or try out something and gather feedback. gojq could stay close to upstream. Maybe extensions could be namespaced by default, so no collisions with future jq changes possible.

[wader]

That is my understanding also but i have no experience doing it. Read a bit about the plugin package out of curiosity and it seems like it requires the plugin to be compiled with exact same golang version and GOPATH etc, not very convenient. Other option i guess would be wasm somehow? but in either case gojq package would require to have some kind of stable plugin API i guess and at what level? WithFunction/WithIterator or lower level?

Maybe if you want to experiment you could import gojq and build some bridge to WithFunction/WithIterator somehow?

Depending on what performance your looking for an alternative could be to mimic something like fastcgi? let an external process get calls as serialized JSON and respond with JSON? using the code i linked to above you can prototype such things like this:

def _process_rpc($cmd): tojson | cmd($cmd) | fromjson;
def json_http($method; $url):
    _process_rpc(["curl", "-X", $method, "-H", "Content-Type: application/json", "-d", "@-", "-o", "-", $url]);

{bla: 123} | json_http("GET"; "https://httpbin.org/anything")

{
    "args": {},
    "data": "{\"bla\":123}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "*/*",
        "Content-Length": "11",
        "Content-Type": "application/json",
        "Host": "httpbin.org",
        "User-Agent": "curl/7.84.0",
        "X-Amzn-Trace-Id": "Root=1-63669f6d-17a435df1c8948ba322b588c"
    },
    "json": {
        "bla": 123
    },
    "method": "GET",
    "origin": "...",
    "url": "https://httpbin.org/anything"
}