Regularly yield to other zebrad async tasks

[teor2345]

Motivation

tokio tasks can run for a long time before yielding to other tasks on the same thread. The executor attempts to compensate by adding extra yield points. But the defaults don't always work for Zebra.

This can cause some Zebra tasks to hang for long periods of time. So we might want to add some extra yield points to Zebra.

Tokio Documentation

Summary:
https://docs.rs/tokio/1.14.0/tokio/task/index.html#yield_now

Detailed yield_now documentation:
https://docs.rs/tokio/1.14.0/tokio/task/fn.yield_now.html#non-guarantees

Tasks

Decide where we should add yield points to Zebra.

Some alternatives are:

Network:

• after sending every network request to the peer - is this too frequent?
• after processing every network response in Zebra - is this too frequent?
• after processing every syncer ObtainTips and ExtendTips

Verifiers:

• after processing every verifier response in the sync downloader, inbound downloader, or mempool downloader
• after verifying each mined transaction in a block - is this too frequent?

State:

• after each expensive state operation

Completed Tasks

#3191:

• after sending every request in a fanout
  • might be obsoleted by Limit fanouts to the number of active peers, Credit: Equilibrium #2214
• after every crawler DemandDrop

Related Work

• Move the metrics recorder to a blocking thread to avoid hangs #2201
• Move batch cryptography flushes to a blocking thread to avoid hangs #2202
• Move finalized state database and filesystem operations to blocking threads to avoid hangs #2188
• Security: Spawn each initial peer handshake in a separate task, Credit: Equilibrium #2163

[mpguerra]

Hey team! Please add your planning poker estimate with ZenHub @conradoplg @dconnolly @jvff @oxarbitrage @teor2345 @upbqdn

[teor2345]

We should identify tasks that run without yielding using tokio-console, then fix those specific bugs.