eliminate lock on record

[jeffwashington]

Problem

PohRecorder.Record requires a PohRecorder mutex lock. This results in contention with poh. Poh locking delays cause drift.

Summary of Changes

Create a new PohRecorder 'Recorder' which can use non-blocking channels to record transactions in poh.

Fixes #

[jeffwashington]

Here are some rough metrics:
With Stephen's test on the colo machine, throttled to ~3k tps requested, 107600 hashes/tick (should be 100% of machine capability), 5s sample time, I get:
master: 17% of the time spent not hashing (mutexes, sending record entry on channel, etc.). 672 ticks (800 is ideal)
my branch: 6% of the time not hashing. 751 ticks (800 is ideal)

[codecov]

Codecov Report

Merging #15929 (234b4a8) into master (482c027) will decrease coverage by 0.0%.
The diff coverage is 84.7%.

@@            Coverage Diff            @@
##           master   #15929     +/-   ##
=========================================
- Coverage    79.9%    79.9%   -0.1%     
=========================================
  Files         409      409             
  Lines      107723   107864    +141     
=========================================
+ Hits        86153    86247     +94     
- Misses      21570    21617     +47     

[sakridge]

Can we also add performance information from a gcp bench-tps run and see where we are in terms of slot time under load and just the TPS. Can run it on gcp manually or through the same configuration as #performance-results

[sakridge]

Want to mark ready-for-review if that's the case?

[jeffwashington]

Can we also add performance information from a gcp bench-tps run and see where we are in terms of slot time under load and just the TPS. Can run it on gcp manually or through the same configuration as #performance-results

last perf run:
mean_tps: 15067
max_tps: 28837
mean_confirmation_ms: 1586
max_confirmation_ms: 3513
99th_percentile_confirmation_ms: 2461
max_tower_distance: 31
last_tower_distance: 31
slots_per_second: 1.273

this pr:
mean_tps: 24424
max_tps: 70562
mean_confirmation_ms: 1777
max_confirmation_ms: 12530
99th_percentile_confirmation_ms: 5267
max_tower_distance: 64
last_tower_distance: 31
slots_per_second: 1.523

[sakridge]

Seems to make poh_service.elapsed_ms a bit worse, but probably because it's going a lot faster.

[sakridge]

lgtm