Instrument RetryingTransactionHelper

[todorinskiz]

RetryingTransactionHelper can be used to hide many underlying concurrency issues.
We would like to instrument it to get more insights on:

• How long does a transaction take
• How many transactions failed
• How many transactions were retried (would be hard to sniff out)
• How many exceptions of each type were thrown (only the last exception is available, would be hard to sniff out all of the thrown exceptions)

https://github.com/Alfresco/alfresco-repository/blob/master/src/main/java/org/alfresco/repo/transaction/RetryingTransactionHelper.java

[todorinskiz]

Retry count mechanism: https://github.com/Alfresco/alfresco-repository/blob/master/src/main/java/org/alfresco/repo/transaction/RetryingTransactionHelper.java#L430

Only the last exception is recorded: https://github.com/Alfresco/alfresco-repository/blob/master/src/main/java/org/alfresco/repo/transaction/RetryingTransactionHelper.java#L429

[todorinskiz]

Seems like our only option is AOP to intercept the method calls, then override the bean globally.
This would allow us to cover all of the above points.

[kerkhofsd]

Please notice that in a lot of cases, a (new) instance of a RetryingTransactionHelper is acquired via the TransactionService and not by injecting the retryingTransactionHelper bean.

[todorinskiz]

Good remark @kerkhofsd
https://github.com/Alfresco/alfresco-repository/blob/master/src/main/java/org/alfresco/repo/transaction/TransactionServiceImpl.java#L312

[todorinskiz]

@AFaust pointed out that since the RetryingTransactionHelper is one of the most used components, instrumentation will have to be as efficient as possible to avoid adding more load to the system.
A solution for that issue can be a short circuit in each public method, that would check if the instrumentation must be enabled or not - and if not, forward the call to the base implementation immediately without recording metrics.

[todorinskiz]

One of the most annoying edge cases @AFaust comes across is the transaction rollback without logging an exception.
This mean that the webscript would respond with Http Status 200 since the exception was not thrown and the state will not be changed because of the transaction rollback.

This information could also be instrumented, to know for which exceptions the transaction has been rolled back (and e.g. how many transaction rollbacks were there).
https://github.com/Alfresco/alfresco-repository/blob/master/src/main/java/org/alfresco/repo/transaction/RetryingTransactionHelper.java#L454

[AFaust]

Minor note on the "most annoying edge cases": In that case no exception actually reaches the RetryingTransactionHelper, because another component (Alfresco AOP interceptor) already caught and handled the exception, and set the transaction status to Status.STATUS_MARKED_ROLLBACK. In that case the main benefit of the instrumentation would be to know that a "silent" rollback is happening (and how often) without having to enable excessive debug logging or attachign a remote debugger.