feat(ai): Make response parsing extensible

[planger]

What it does

Turns the response parsing method of the chat agents into a more flexible algorithm that can work with multiple response content matchers. Each response content matcher has a start and an end regexp to define a match, as well as a contentFactory function that is invoked to turn the matched content into a specific ChatResponseContent object.

This way, clients can add and combine multiple response content matchers to process LLM responses and turn them into custom response parts, while still keeping the generically useful ones, like the matcher for code blocks, etc.

Additionally, the parsing method has a fallback content factory that will be applied to all unmatched parts. By default, this is the existing markdown content part, but this can be customized if needed.

Both, the response content matchers and the fallback content factory are configurable via DI.

Contributed on behalf of STMicroelectronics.

How to test

Ensure normal behavior is still working as expected. That is, we should test the processing and rendering of plain chat responses that contain an arbitrary mixture of markdown parts, code parts, and tool call parts.

In addition, one could test to customize the DefaultResponseContentFactory or register additional ResponseContentMatcherProvider.

    // Change the default response part
    rebind(DefaultResponseContentMatcherProvider).to(YourDefaultPartCustomization);
    // Add additional matchers
    bind(ResponseContentMatcherProvider).to(YourAdditionalResponseContentMatcherProvider);
    // Change default matchers (code)
    rebind(DefaultResponseContentFactory).to(YourCustomization);

Since the parsing is a bit of a dense code, I added unit tests for it.

Follow-ups

I tried to simplify the parsing. However, there is one aspect, which I found a bit confusing but is not changed by this PR:
The fall back to markdown, handling of tool calls, or parsing into specific parts is a bit spread across the code

• 2x in parse()
• 3x in addContentsToResponse, where we currently need to ensure, we prevent parsing, if the previous content was a tool call, which is also somewhat hidden.

Maybe there is an opportunity to further simplify this and parse this all in one place and one go.

Also it was unclear to me why we don't parse for specific content parts in case we have a isLanguageModelTextResponse === true case. This is actually changed so that it now uses the parseContents function to obtain the content parts instead of creating a plain markdown part either way, which feels more consistent.

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[eneufeld]

I like the tests.
Maybe we could improve the readability a bit as the findEarliestMatch and parseContents methods are hard to read.
I will take a second look maybe I find something to suggest

[planger]

@eneufeld Thank you for the review. I've addressed your comments. Can you please re-check and re-test? Thank you!

[eneufeld]

Works for me, thank you

[JonasHelming]

LGTM!