feat(jmespath): add lexer component

[dreamorosi]

Description of your changes

This PR includes the implementation of the Lexer component, part of the JMESPath utility.

As discussed in the linked issue, the purpose of a lexer is to break down the input JMESPath expression into smaller meaningful units (tokens). These tokens represent the building blocks of the JMESPath language and are defined in the language grammar (#2192).

The Lexer's main method (public *tokenize() is implemented as a generator. This pattern allows the lexer to walk the expression iteratively and yield (aka return) tokens as it goes. While not a direct equivalent, the closest patten to describe this implementation would be a recursive function that maintains an external state (aka a reducer).

At each step, the lexer interprets a certain character and based on its type it performs certain actions.

To describe how the lexer works, let's take this expression as an example: foo.bar (leading white space is intentional).

With the expression above, the lexer will start looking at each character in the order they appear:

• position: 0 - Since the first character is a white space, the lexer advances with no further action (source)
• position: 1 - Next, the lexer encounters a valid character (f). At this point the lexer needs to understand how long this identifier is and so it will advance the position until a non-identifier character (aka anything that is not a number or letter) is found (source). In this example it will advance to position 4 and interpret foo as a single token.
• position: 4 - The next character is a dot (aka .) which in the context of a lexer is considered a simple token.
• position: 5 - Next, the lexer encounters another character (b), so just like one of the previous steps, it advances until a non-identifier character is found. In this case the lexer reaches the end of the expression.

This is a relatively simple example, but hopefully it helps clarifying the flow of the processing. For simpler tokens the implementation is inlined in the public *tokenize() method, while in other cases where the processing of a token required a more involved logic a dedicated method was created.

Related issues, RFCs

Issue number: #2205

Checklist

• My changes meet the tenets criteria
• I have performed a self-review of my own code
• I have commented my code where necessary, particularly in areas that should be flagged with a TODO, or hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my change is effective and works
• The PR title follows the conventional commit semantics

Breaking change checklist

Is it a breaking change?: NO

• I have documented the migration process
• I have added, implemented necessary warnings (if it can live side by side)

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud