Network refactoring tester #3841
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
modship
commented
Apr 19, 2023
•
modship
commented
- document all added functions
- try in sandbox /simulation/labnet
- unit tests on the added/changed features
- make tests compile
- make tests pass
- add logs allowing easy debugging in case the changes caused problems
- if the API has changed, update the API specification
Closed
4 tasks
modship
commented
Apr 20, 2023
AurelienFT
approved these changes
Apr 24, 2023
|
Let's merge it as a first version to unlock the rest |
AurelienFT
added a commit
that referenced
this pull request
May 3, 2023
* Add skeleton new massa protocol. * Add architecture handler * Add basic peer management to massa * Setup a simple test. * Get best peer from peer manager * Fix connections to not retry if unnecessary * Add operations handler * Add handler skeleton endorsement * Add block handler skeleton * Improve announcement serialization * Add endorsement default behavior * Add pool controller * Add channels to communicate from handlers to outside world * Fix tests. * Add test endorsements to complete * Add todo ip * Rework network deserialize messages (#3750) * Use new branch peernet * Change handlers messages deserialization architecture. * Remove un-necessary serialization of message ID in messages Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Also deletes for the other type of message Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Remove unused lines Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Refactor serializing of message (#3786) * Add new way of serializing messages. * Use peernet default branch * Format and add comment * Rework id management * Add comment messages * Clean basic test * add command channel in PeerManagementHandler (#3809) * improves the peer management and communication logic * Add structure of operation handler (#3817) * Add structure of operation handler * Update cache to use a simplified Lru and share it. Use the same channel for all communications to the propagator * Add send of op and remove peers from cache when needed * Update lock * Add management of messages received from retrieval. * Change timers management and add pruning * Update caches and add storage to keep ops to be propagated * Remove useless line and add a comment * Remove useless write locks. * ban peer if note_operations_from_peer is Err (#3826) --------- Co-authored-by: Modship <yeskinokay@gmail.com> * Add basic endorsement handler code (#3827) * Add structure of operation handler * Update cache to use a simplified Lru and share it. Use the same channel for all communications to the propagator * Add send of op and remove peers from cache when needed * Update lock * Add management of messages received from retrieval. * Change timers management and add pruning * Update caches and add storage to keep ops to be propagated * Remove useless line and add a comment * Remove useless write locks. * Add basic code for endorsement handler * Fix runtime error in tests (#3836) * Network refactoring tester (#3841) * add 'thread_tester_count' in config.toml * refactor Tester::new * move thread_tester_count from NetworkConfig to ProtocolConfig * create shared channel for thread tester // add stop command to PeerManagementHandler * Tester : test the listener on recv msg * remove unwrap and print error * Add basic skeleton for block handler and fix endorsement one (#3840) * tmp Signed-off-by: Litchi Pi <litchi.pi@proton.me> * drop manager Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Add two different thread for block handler * fixup Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Add cache of block * Comment * Fix handler handshake in massa protocol. * Add basic block handler * Add skeleton reception message of block infos * Add all the code of the block handler * Finish code of block retrieval thread * Add the commands of propagation handling and fix compilation error * Add debug print and fix the peer that didn't connected after handshake * Fix listeners created too late. * Fix review notes. * Change ban node to warn in case it fails --------- Signed-off-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Litchi Pi <litchi.pi@proton.me> * Add stop command to all thread network refactoring (#3858) * tmp Signed-off-by: Litchi Pi <litchi.pi@proton.me> * drop manager Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Add two different thread for block handler * fixup Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Add cache of block * Comment * Fix handler handshake in massa protocol. * Add basic block handler * Add skeleton reception message of block infos * Add all the code of the block handler * Finish code of block retrieval thread * Add the commands of propagation handling and fix compilation error * Add debug print and fix the peer that didn't connected after handshake * Fix listeners created too late. * Fix review notes. * Change ban node to warn in case it fails * Add channels for exterior world on all handlers thread and add stop command * Update ports test * Add stop command on handler so that even if there is clone of the sender they are killed --------- Signed-off-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Litchi Pi <litchi.pi@proton.me> * send 100 peers to connected peers (#3864) * get_peers_to_send * update fn * send 100 random peers to all peer connected each 10 sec * use tuple for sender/receiver in PeerManagementHandler for readability * Wrap interaction with peernet to mock them (#3865) * Add basic mocks for tests * Wrap all interactions to peernet with a controller to mock it * Avoid overriding cache value * Add mock in test and port the first test * Update massa-protocol-worker-2/src/tests/ask_block_scenarios.rs Co-authored-by: Modship <yeskinokay@gmail.com> * Update massa-protocol-worker-2/src/tests/ask_block_scenarios.rs Co-authored-by: Modship <yeskinokay@gmail.com> * Change a todo --------- Co-authored-by: Modship <yeskinokay@gmail.com> * Fix cargo lock * Fix compil error after merge * Port tests network refactoring. (#3869) * Port tests block workflow. * Add more test on block handler and fix some bugs found. * Port tests in block operations and add propagation of operations from block handler to operation one * Add scenarios endorsements * Port operation tests * Port test ban_node and add disconnect of a banned node. * Add cache scenario * Fix compilation tests. * Fix last endorsement test * Change comment ban scenarios * Fix tests protocol * Update config on new protocol network and clean-up (#3875) * Add name to threads and more config values. * Bound all channels remove unwraps * Fix clippy warnings * Update config tests * network_refactoring - send peers in handshake (#3870) * send peers in handshake * remove peer_db.read() and use previous lock * remove useless scope * restore scope and send peers when perform_handshake start * fix : use saturating_sub * update massa network with new peernet trait * add serializer for peer management message in MassaHandshake to avoid many new serializer call * Add Clone derive to struct for serializer * send peers on failback_function + remove unwrap * add panic hook in test * remove revision tag for peernet in Cargo.toml and update it. * update OutConnectionConfig::Tcp with default value (can set rate limit) * remove dbg * Use new protocol (#3879) * Delete old network and protocol * Use the new massa-protocol * Change import of ProtocolCommandSender to ProtocolController * Fix all compile errors. * Fix all error in tests and use the keypair file. * Use limiter from settings * Use routable ip * Update peernet * Add all variables to config to launch the node * Fix compilation of API with new protocol * Fix compilation of the node * Remove network exports from bootstrap and replace interaction with protocol ones * Fix(tests) : test_peer_connected && test_list_peers (#3882) * Remove network and fix tests compilation * Fix all tests * Use initial peers from bootstrap and change initial peers file * Remove old network category in config. * Change print stop to info * Don't test peers if we are already connected to * Don't test our local ips. * Increase maximum limitation socket. * Update peernet * Add debug prints * Update peernet * Update peernet * Fix endorsement merge deletion * Try to disable endorsement pool to test * Fix peer not registered when received from remote. --------- Co-authored-by: Modship <yeskinokay@gmail.com> * Use new protocol #2 (#3886) * Delete old network and protocol * Use the new massa-protocol * Change import of ProtocolCommandSender to ProtocolController * Fix all compile errors. * Fix all error in tests and use the keypair file. * Use limiter from settings * Use routable ip * Update peernet * Add all variables to config to launch the node * Fix compilation of API with new protocol * Fix compilation of the node * Remove network exports from bootstrap and replace interaction with protocol ones * Fix(tests) : test_peer_connected && test_list_peers (#3882) * Remove network and fix tests compilation * Fix all tests * Use initial peers from bootstrap and change initial peers file * Remove old network category in config. * Change print stop to info * Don't test peers if we are already connected to * Don't test our local ips. * Increase maximum limitation socket. * Update peernet * Add debug prints * Update peernet * Update peernet * Fix endorsement merge deletion * Try to disable endorsement pool to test * Fix peer not registered when received from remote. * Add debug print * Add debug prints * Add debug print * Fix endorsement note endorsements * Add debug print * Add more debug print * Update peernet * Update peernet * Update peernet * Remove all debug prints * Display all len of all vectors to spot memory leak * Remove debug memory print * Add print when new peer connected * Add setup nasm in CI --------- Co-authored-by: Modship <yeskinokay@gmail.com> * Update port tests * Add todo on api * Add new hack comment * Remove useless scope * Use type PeerMessageTuple in messages handler * Added todo * Fix get rand peers and debug print. * Send myself in bootstrap peers & don't send local addresses. * Fix add of myself in bootstrap peers. * Change add myself * Add debug print and reduce lock length * Fix received fallback data in handshake. * Fix error offset in tester handshake * Update peernet * Fix tests and remove debug print * Update peernet. * Add debug print * Update peernet * Add more print deadlock * Update peernet * Remove debug print * Update announce in peerdb when handshake worked. * Readd some debug print. * Fix do not test our ip. --------- Signed-off-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Modship <yeskinokay@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.