Zebra lags behind the best tip on testnet

[teor2345]

Version

zebrad 1.0.0-alpha.5 with PR #1940

Platform

Linux ... 5.4.96 #1-NixOS SMP Sun Feb 7 14:35:50 UTC 2021 x86_64 GNU/Linux

Description

Testnet instances lag a few hundred blocks behind the tip synced by zcashd.

The zcashd tip is 1338659, and the Zebra tip is 1338519, a difference of 140 blocks.

We've had this bug before, it's possibly due to either network code hangs or peer issues (#1905), or the small number of peers on testnet (#704). I haven't seen it happen on mainnet since the late 2020 / early 2021 network fixes.

Commands

zebrad start with an epemeral testnet config with some local peers

Logs

Mar 25 17:17:05.551  INFO {zebrad="98726b0c" net="Test"}:peer{addr=127.0.0.1:38233}:msg_as_req{msg=getblocks}:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1338519) response_len=0 chain_tip_height=Height(1338519) stop_height=None intersection_height=Some(Height(1338519))
Mar 25 17:17:07.593  INFO {zebrad="98726b0c" net="Test"}:sync: zebrad::components::sync: exhausted prospective tip set
Mar 25 17:17:07.593  INFO {zebrad="98726b0c" net="Test"}:sync: zebrad::components::sync: waiting to restart sync timeout=61s
Mar 25 17:18:08.595  INFO {zebrad="98726b0c" net="Test"}:sync: zebrad::components::sync: starting sync, obtaining new tips
Mar 25 17:18:08.595  INFO {zebrad="98726b0c" net="Test"}:sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(1338519) min_locator_height=1338420 locators=[Height(1338519), Height(1338518), Height(1338517), Height(1338515), Height(1338511), Height(1338503), Height(1338487), Height(1338455), Height(1338420)]
Mar 25 17:18:08.595  INFO {zebrad="98726b0c" net="Test"}:sync:obtain_tips: zebrad::components::sync: trying to obtain new chain tips tip=block::Hash("003e1b484c2c14ef29691c3640a7d5d66c4454029d7fb731a5b7c5111bad6fc4")
Mar 25 17:18:12.554  INFO {zebrad="98726b0c" net="Test"}:peer{addr=127.0.0.1:38233}:msg_as_req{msg=getblocks}:state: zebra_state::service: responding to peer GetBlocks or GetHeaders final_height=Height(1338519) response_len=0 chain_tip_height=Height(1338519) stop_height=None intersection_height=Some(Height(1338519))
Mar 25 17:18:14.597  INFO {zebrad="98726b0c" net="Test"}:sync: zebrad::components::sync: exhausted prospective tip set
Mar 25 17:18:14.597  INFO {zebrad="98726b0c" net="Test"}:sync: zebrad::components::sync: waiting to restart sync timeout=61s