perf: message pool: change locks to RWMutexes for performance

[snissn]

Related Issues

should effect #10538 because of message pool locks

Proposed Changes

message pool has two locks - change mp.lk and mp.curTsLk to RW locks for API efficiency

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

• Commits have a clear commit message.
• PR title is in the form of of <PR type>: <area>: <change being made>
  • example: fix: mempool: Introduce a cache for valid signatures
  • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
  • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
• New features have usage guidelines and / or documentation updates in
  • Lotus Documentation
  • Discussion Tutorials
• Tests exist for new functionality or change in behavior
• CI is green

[magik6k]

Generally looks good, we can switch to RLock in message selection as well.

[snissn]

closed #10545 for this --

The lock contention seems gone from the eth rpc path now! 100 requests per second compared with 8 seconds per request :) :)

$ cat bench.sh
ab -p post_data.json -T application/json  -c 10 -n 20000 http://localhost:1234/rpc/v1
ubuntu@ip-10-0-4-29:~$ bash bench.sh |tee bench-PR.out
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 2000 requests
Completed 4000 requests
Completed 6000 requests
Completed 8000 requests
Completed 10000 requests
Completed 12000 requests
Completed 14000 requests
Completed 16000 requests
Completed 18000 requests
Completed 20000 requests
Finished 20000 requests


Server Software:
Server Hostname:        localhost
Server Port:            1234

Document Path:          /rpc/v1
Document Length:        45 bytes

Concurrency Level:      10
Time taken for tests:   189.902 seconds
Complete requests:      20000
Failed requests:        0
Total transferred:      3240000 bytes
Total body sent:        6700000
HTML transferred:       900000 bytes
Requests per second:    105.32 [#/sec] (mean)
Time per request:       94.951 [ms] (mean)
Time per request:       9.495 [ms] (mean, across all concurrent requests)
Transfer rate:          16.66 [Kbytes/sec] received
                        34.45 kb/s sent
                        51.12 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    29   95  59.0     83     843
Waiting:       29   95  59.0     83     843
Total:         29   95  59.0     83     843

Percentage of the requests served within a certain time (ms)
  50%     83
  66%     93
  75%    112
  80%    124
  90%    163
  95%    210
  98%    268
  99%    316
 100%    843 (longest request)

[snissn]

I'm looking into the failing tests not sure if actual or flaky

[snissn]

hmm definitely still seeing problems with this - not ready just yet

[snissn]

Ok great! I think the last issue was repub needs Lock not RLock
831f8a4

tests are passing now, ready for review

[snissn]

latest bench from #41ea5a6f638ca5108a51f70a4cef85d9b98802b7

This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 2000 requests
Completed 4000 requests
Completed 6000 requests
Completed 8000 requests
Completed 10000 requests
Completed 12000 requests
Completed 14000 requests
Completed 16000 requests
Completed 18000 requests
Completed 20000 requests
Finished 20000 requests


Server Software:
Server Hostname:        localhost
Server Port:            1234

Document Path:          /rpc/v1
Document Length:        45 bytes

Concurrency Level:      10
Time taken for tests:   200.514 seconds
Complete requests:      20000
Failed requests:        0
Total transferred:      3240000 bytes
Total body sent:        6700000
HTML transferred:       900000 bytes
Requests per second:    99.74 [#/sec] (mean)
Time per request:       100.257 [ms] (mean)
Time per request:       10.026 [ms] (mean, across all concurrent requests)
Transfer rate:          15.78 [Kbytes/sec] received
                        32.63 kb/s sent
                        48.41 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    52  100  48.6     82     624
Waiting:       52  100  48.6     82     624
Total:         52  100  48.6     82     624

Percentage of the requests served within a certain time (ms)
  50%     82
  66%     90
  75%    103
  80%    120
  90%    156
  95%    203
  98%    262
  99%    297
 100%    624 (longest request)