When using observability annotations, recommend that care is taken to avoid double instrumentation

[helpermethod]

Today, I've stumbled over this issue:

micrometer-metrics/micrometer#780

It seems that this happens not only when annotating controller methods with @Timed but also when annotating Spring Data repositories (and possibly other beans).

Is there some way for Spring Boot to prevent this?

[wilkinsona]

Unfortunately, I don't think there's anything that Boot can do here as it doesn't have the necessary information to detect a double registration. To do so, it would have to:

1. Detect @Timed on a method
2. Detect that the TimedAspect is in play
3. Know that the annotated method is also instrumented by the underlying framework

1 is possible. 2 is harder as it relies both on a TimedAspect bean and the necessary AOP machinery being in place. 3 is the real stumbling block as it isn't possible without coupling Boot more tightly than we would like to behavior in Spring MVC, Spring Data, etc.

If you've opted into using @Timed (or @Counted), you may also want to consider defining TimedAspect yourself with a predicate that performs your desired filtering, as described in this comment from @jonatan-ivanov.

[helpermethod]

Hi @wilkinsona,

thanks for the detailed answer, that makes sense.

I've used the predicate workaround to solve this issue, but it took me quite a while to find out what the actual problem was. Maybe the metrics chapter @Timed usage should link to the workaround / give some hints. Wdyt?

[wilkinsona]

Yes, I think we could add something to https://docs.spring.io/spring-boot/3.3/reference/actuator/metrics.html#actuator.metrics.supported.timed-annotation. Thanks for the suggestion.

[jonatan-ivanov]

FWIW, I think by adding @Timed/@Observed/etc. to something that is already instrumented out of the box, (without disabling that instrumentation) you are explicitly asking for double instrumentation (built-in + annotation).
Also, since annotating a controller method with @Timed/@Observed/etc. is technically not exactly the same (Spring Framework instruments these on the ServletFilter level), this "double instrumentation" could be a ~valid use-case if you want to distinguish between just the method call and the method call + Spring overhead.

Because of these, I think Boot should not do anything but making it easy to turn auto-instrumentation on/off (this is already happening).

@wilkinsona For the doc change, could you please also consider @Counted, @Observed, and @NewSpan next to @Timed?

[wilkinsona]

Yes, I think so. We could mention something in https://docs.spring.io/spring-boot/reference/actuator/observability.html#actuator.observability.annotations as well, or perhaps tidy things up a bit into a single common section.