Enhance parsing of JSON for firewall IP aliases

[gglockner]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Is your feature request related to a problem? Please describe.

Amazon Web Services (AWS) publishes their IP address ranges in JSON format. Here is the documentation and here is the direct link to the JSON data. Using OPNsense 25.1 (amd64), I would like to use Firewall > Aliases to import this data as Type "URL Table in JSON format (IPs)". I have looked at the source in src/opnsense/scripts/filter/lib/alias/uri.py; as best as I can determine, this JSON data cannot be parsed by this OPNsense feature.

Describe the solution you like

The "Path expression" field should support more query options, so that this data can be used for firewall rules. Consider jq, a popular tool to process JSON data. The following Linux shell command gives all IPv4 address ranges from AWS:

curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[].ip_prefix'

jq is quite powerful, so you could do a more sophisticated filter such as: all IPv4 ranges for AWS EC2 in the us-east-1 region:

curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select(.region=="us-east-1") | select(.service=="EC2") | .ip_prefix'

This type of logic would be difficult to replicate using the limited single-line parsing implementation in src/opnsense/scripts/filter/lib/alias/uri.py.

Describe alternatives you considered

It would be straightforward to implement this in src/opnsense/scripts/filter/lib/alias/uri.py if the jq shell package were installed by default in OPNsense. Alternately, there are a few pure-Python data transformation libraries, but they either have a nonstandard syntax or implement only a subset of jq syntax.

Additional context

I started discussing this topic in #8107 but split this into a separate Github issue.

[AdSchellevis]

I might need to revise my original answer, for some reason I missed the native python binding available via https://github.com/opnsense/ports/blob/master/textproc/py-jq, which is more or less a stub around the library itself.
Usually these are not too complicated in maintenance.

I might do a small proof of concept with this, there are pro's and cons here. jqlang is used by quite some people, has an online tester available (https://jqplay.org/), but is not very easy to learn for less experienced users.

Offering only simple patterns a.b, a.*.b might be easier to explain, but has the downside of not being extendable in the future. As we already released the a.b pattern in 25.1, there's also the small issue of different formats in its current state.

[gglockner]

If we agree that this would be implemented via a Python front-end to jq, I would be happy to contribute the code for this. The key thing that prevents me from implementing it myself and submitting a pull request is that I cannot figure out how to ensure that the packages for jq and textproc are installed by default.

I agree there is a very small chance this could break some installations. However, the chance should be tiny considering this feature was introduced merely a few days ago in 25.1.

[AdSchellevis]

let me fiddle a bit and get back to you, compiling the port in this case is rather quick

[Zerophase]

Are the Spamhaus block lists valid for testing this?

I believe it's valid JSON as well, but can't be interpreted line by line.

[AdSchellevis]

Are the Spamhaus block lists valid for testing this?

I believe it's valid JSON as well, but can't be interpreted line by line.

These should already work in 25.1, see #8107

[Zerophase]

Figured out what I was doing wrong. I was looking up JSONPath expression format and putting it in in the form of $.*.cidr, rather than just cidr. Most of the online path expression validators can't handle either form.

[AdSchellevis]

There's no documentation yet, other than what's in the help text, This ticket is about extending the support, possibly with jq expressions, JSONPath is another option, neither unfortunately exist by default in python.