Properly construct instance of LateBoundBatchSpanProcessor

[geoand]

This is now done entirely in a CDI producer method which is then fully initialized when OpenTelemetry autoconfigures itself

Fixes: #29987 (which with newer Quarkus versions is even more problematic)

Detailed explanation:

• The first commit fixes the case where the DataSource cannot properly be constructed because the OpenTelemetry has not yet been created by CDI. This could happen because there was not declaration of the need for that bean when constructing the DataSource bean, only programmatic lookup.
• The second commits actually fixes the issue with the missing delegate in a pretty straightforward way: We move the creation of LateBoundBatchSpanProcessor to a synthetic bean - prior to this commit the LateBoundBatchSpanProcessor bean was created from a CDI producer but was then "updated" by a recorder. These two operations are obviously not "atomic", meaning that the OpenTelemetry bean that consumed LateBoundBatchSpanProcessor would see the bean in the intermediate state - where it was created but not updated.
  • This commit also includes the removal of various intermediate build items whose only purpose was to pass data between the various steps. These are no longer necessary, since everything can be accomplished with the combination of CDI and the BeanContainer in recorder methds.
  • Furthermore, this commit makes sure that we no longer use Arc.container()... in recorder methods as these are extremely sensitive to timing issues - by utilizing synthetic beans and the BeanContainer in recorder methods we overcome this problem.
• The third commit builds on the second one to make the configuration of the SpanProcessors of OpenTelemetry even more reasonable. We essentially only include LateBoundBatchSpanProcessor in the OpenTelemetry bean if a delegate has been set.

Future work: The creation of the OpenTelemetry can be done via a synthetic bean as this allows us complete control over the injection points and does not suffer from any bean creating timing issues (assuming of course the creation is done properly).
This would essentially take what is not done in OpenTelemetryProducer and move it into a recorder method that includes all the creation logic (including the logic that currently resides in the synthetic bean for LateBoundBatchSpanProcessor)

[quarkus-bot]

/cc @brunobat (opentelemetry), @radcortez (opentelemetry)

[radcortez]

So what we are doing now is initializing everything at runtime, correct? (instead of the split we had before, where we created a few things at static init).

[geoand]

Correct, the bean initialization is not forced at build time anymore

[brunobat]

This makes everything much simpler. Not sure about the increased startup time... Most likely not noticeable.
I need to check possible implications on the OpenTelemetryProducer, but I won't have time today.

[geoand]

No rush!

[gsmet]

If it gets merged, should it be backported?

[geoand]

I dunno, I'm on the fence...

[radcortez]

I think we can hold this until 3.3.

[brunobat]

I need to create a new test before approving this PR.

[brunobat]

@radcortez @gsmet I'd like this back-ported to 3.2.* because these changes affect core parts of the extension and 3.2.* will run for a long time, potentially causing complex merges in the future.

[geoand]

If we want this backported, we probably should do it as soon as possible instead of waiting for late 3.2.z releases

[brunobat]

Let's be pragmatic then.
I'll not be able to do this while at the RivieraDev conf.
Any iD generator issue can be dealt in this new issue: #34668

[geoand]

I'm going to have to redo this one unfortunately since there are plenty of conflicts...

Furthermore, this will conflict with #34647

[geoand]

Here is my proposal for this one:

I redo it after #34647 is in, in order to avoid additional conflicts.
For 3.2, I open a new PR specifically targeting the 3.2 branch (this exact change should apply cleanly there).

WDYT?

[radcortez]

Sure.

[geoand]

This now turns out to be harder. I'm looking into how it can be done.

[geoand]

Should be working now

[geoand]

But there is another thing I want to improve if possible

[geoand]

Should be good to go now

[geoand]

In future, I would like to move the creation of the OpenTelemetry bean out of OpenTelemetryProducer and into a synthetic bean, but I'll leave that for some other time because it's a lot of mechanical work that is not absolutely necessary at the moment

[geoand]

Oh, I didn't know about those tests...

[geoand]

The JDBC telemetry stuff is pretty broken (it only worked by accident before) so I need to see what can be done...

[brunobat]

Those tests are a bit brittle but beware that they end up finding a lot of problems.

[geoand]

The tests themselves are great and they indeed show a real problem here.

[geoand]

The JDBC failures are now fixed.

In general, there were (and probably still are) a lot of timing issues that relate to when the various CDI beans are constructed and obtained.
That's why I want to move the OpenTelemetry bean creation to a synthetic bean in the future.

[brunobat]

Ok @geoand. The interdependencies with vert.x in InstrumentationRecorder.createTraces(..)
Will an OTel Producer as a synthetic bean work with dev mode reload?

[geoand]

Yes it would.

But it's not something we need to do right now - it can wait for the other stuff I want to do around telemetry

[quarkus-bot]

✔️ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

[geoand]

All tests pass so this is good from my end.

[brunobat]

Thanks @geoand !