feat(js/plugins): added experimental_debugTraces option to googleai and vertexai plugins

[pavelgj]

const ai = genkit({
  plugins: [googleAI({experimental_debugTraces: true})],
});

Checklist (if applicable):

• PR title is following https://www.conventionalcommits.org/en/v1.0.0/
• Tested (manually, unit tested, etc.)
• [n/a] Docs updated (updated docs or a docs bug required)

[chrisraygill]

Questions: Does this need to be plugin-specific? Or is there something we can do at the framework level to enable this for all plugins?

[pavelgj]

Questions: Does this need to be plugin-specific? Or is there something we can do at the framework level to enable this for all plugins?

I think there's an opportunity to do something at the framework level, but tracing logic must be implemented per-plugin anyway...

I think it's fine like this. We can revisit this later. For now the dev can always do something like:

const ai = genkit({
  plugins: [googleAI({debugTraces: process.env.DEBUG_TRACES === 'true'})],
});

[MichaelDoyle]

This is super interesting, thanks! It's potentially relevant to some things we are looking at in AIM.

I have a few questions.

1. What was the motivation for this?
2. Is output also relevant?
3. Depending on the motivation, is this something that ideally the Vertex/Gemini SDKs should be tracing? If yes, I wonder if we can/should try and contribute it there (and eventually migrate over to that). Or if we should engage with that team to prioritize it (we might know some folks involved in trying to get OTel adopted there for other reasons).

Should we consider (in addition to input/output, which is Genkit specific) layering in the semantic conventions? If we write this trace according to the semantic conventions, in production, Cloud Trace will show it really nicely in a gen AI specific view. and eventually AIM will too.

Currently, we are planning to do that on the model action span: #1601

See also: https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/

[MichaelDoyle]

cc @bryanatkinson

[MichaelDoyle]

Couple more questions. Is this mostly geared towards our team (i.e. people developing Genkit)? Or do we expect app developers to use it? Troubleshooting with us? In prod?

I am wondering if "debug" is descriptive enough for what the use case is (all tracing is debug related in a way 😄) versus say "internal" traces. Inspiration: https://opentelemetry.io/docs/collector/internal-telemetry/

[MichaelDoyle]

This looks cool. I want to use it already! It also made me think of a bunch of questions.

[pavelgj]

This is super interesting, thanks! It's potentially relevant to some things we are looking at in AIM.

I have a few questions.

1. What was the motivation for this?
2. Is output also relevant?
3. Depending on the motivation, is this something that ideally the Vertex/Gemini SDKs should be tracing? If yes, I wonder if we can/should try and contribute it there (and eventually migrate over to that). Or if we should engage with that team to prioritize it (we might know some folks involved in trying to get OTel adopted there for other reasons).

Should we consider (in addition to input/output, which is Genkit specific) layering in the semantic conventions? If we write this trace according to the semantic conventions, in production, Cloud Trace will show it really nicely in a gen AI specific view. and eventually AIM will too.

Currently, we are planning to do that on the model action span: #1601

See also: https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/

1. The motivation is that Genkit plugin does extremely non-trivial translation of Genkit data model into model SDK specific model. A more specific motivation came from one user who was troubleshooting tool calling ([JS] Tool calling doesn't respect input schema #2022) and the suspicion was a bug in the plugin logic... which led to fix(js/plugins): fixed gemini tool schema converter to correctly handle descriptions, enums and nullable fields #2027 (a bug). This feature was instrumental in helping me debug the issue.

Beyond that, I believe it will be super helpful for developers who are already familiar with provider SDK to understand what Genkit actually does to generate the request.

2. we already include raw response in the GenerateResponse... so technically it's not super useful... but we can add more stuff in there if we want.

3. that's interesting... we should explore this.

[pavelgj]

Couple more questions. Is this mostly geared towards our team (i.e. people developing Genkit)? Or do we expect app developers to use it? Troubleshooting with us? In prod?

I am wondering if "debug" is descriptive enough for what the use case is (all tracing is debug related in a way 😄) versus say "internal" traces. Inspiration: https://opentelemetry.io/docs/collector/internal-telemetry/

I think devs might be interested in looking at those traces... those traces should look familiar to devs who used the SDK directly in the past... but internal-telemetry approach looks interesting as well.

[pavelgj]

decided to prefix the flag with experimental_ -- we will probably want to tweak this.