Message loss: message fragments written across terms boundary are discarded in re-assembly

[fredfp]

Description

Artio version 0.154

When working as initiator, some incoming FIX messages are lost/discarded. Symptoms are:

• in the context of a connected and working session
• some incoming messages are lost:
• Logged as Enqueued in uk.co.real_logic.artio.protocol.GatewayPublication#saveMessage(org.agrona.DirectBuffer, int, int, int, long, long, int, long, uk.co.real_logic.artio.messages.MessageStatus, int, long, org.agrona.DirectBuffer, int)
• never received by the library i.e., not logged as Received in uk.co.real_logic.artio.library.LibraryPoller#onMessage
• however the next message (with the following sequence number) is correctly logged as Enqueued and then Received, which triggers a resend request.

Analysis

• The same happens when replaying from the archive
• when digging deeper:
  • the "lost" message is present in the archive
  • however it is discarded in io.aeron.FragmentAssembler#handleFragment
  • the second fragment is not considered because the header.termOffset() == builder.nextTermOffset() test fails, see more debug details below
• artio is handling the fragmentation, but assembling the fragments is handled by aeron. I suspect something doesn't match anymore in there. For instance, the failing check was added recently in aeron: real-logic/aeron@54e8c73

More debug details

Data below seem to indicate that the problem indeed happens at term boundary, for the discarded fragment, in io.aeron.FragmentAssembler#handleFragment, I get the following values:

• termLength = 4194304
• builder.nextTermOffset() = 4193728
• header.termOffset() = 0 indicates we just started a new term
• length = 555 might be a clue:
  • builder.nextTermOffset() + length < termLength doesn't suggest reaching a new term (and requiring to fill the current one with padding).
  • could it be that artio assumes the current message fragments will fit in the rest of the term when it's not the case?

[wojciech-adaptive]

Thanks for the detailed bug report, fixed. Will do a release soon.

[fredfp]

Nice one, thank you!