Test failure: nodeos_startup_catchup_if_lr_test

[heifner]

https://github.com/AntelopeIO/spring/actions/runs/11707186905/job/32607464282

2024-11-06T16:28:25.027786 Node10 has 509 unlinkable_block in /__w/spring/spring/build/TestLogs/nodeos_startup_catchup8/node_10
2024-11-06T16:28:25.027940  ERROR: Node10 has 509 which is more than the configured allowed 500 unlinkable blocks: /__w/spring/spring/build/TestLogs/nodeos_startup_catchup8/node_10.

This is different than #998

[bhazzard]

When looking into this test, consider whether any changes would need to be backported to 1.0.x.

[linh2931]

Two questions, @heifner:

1. numUnlinkable is the total number of unlinkable blocks in all the logging files. Should it be the number of unlinkable blocks in only the latest log file?
2. For the test to be less flaky, should we check whether or not the unlinkable block numbers are consecutively incremented (no gaps), instead of checking against a fixed number of unlinkable like 500?

[heifner]

If you have verified that the unlinkable blocks are expected, then I think counting consecutive blocks less than fetch-span as one is a good choice. If you make that change, seems like option 1 doesn't matter.

[linh2931]

If you have verified that the unlinkable blocks are expected, then I think counting consecutive blocks less than fetch-span as one is a good choice. If you make that change, seems like option 1 doesn't matter.

Thanks Kevin. Will do that as discussed in the meeting and fix it in 1.0.x.

[linh2931]

The test is broken in Rel 1.0.3 and flaky in main. It counts the number of unlinkable_block occurrences in logging files and make sure it is not greater than 500.

But in Rel 1.0.3, unlinkable_block (part of unlinkable_block_exception) shows up 3 times per unlinkable block, for example:

warn  2024-11-14T13:48:06.038 nodeos    controller.cpp:4334           push_block           ] 3030001 unlinkable_block_exception: Unlinkable block
info  2024-11-14T13:48:06.038 nodeos    producer_plugin.cpp:938       operator()           ] Exception on block 130: 3030001 unlinkable_block_exception: Unlinkable block
info  2024-11-14T13:48:06.038 nodeos    net_plugin.cpp:3870           process_signed_block ] unlinkable_block_exception connection - 1: #130 1d74c43582d10251...: Unlinkable block (3030001)

While in main, only one occurrence of unlinkable_block per unlinkable block:
debug 2024-11-06T16:28:13.927 net-3 net_plugin.cpp:3744 operator() ] unlinkable_block 193 : 000000c191e645305efb7e0f7d404e2ca8abf78ae59e941fa0bd1a1c3c153042, previous 192 : 000000c096685ceb09297cbfef42fcf458a572bdeeb9243a8d96ca51c798d454

[linh2931]

The fixes for 1.0.3 and main will be different.