Add basic endorsement handler code #3827

AurelienFT · 2023-04-14T12:47:47Z

document all added functions
try in sandbox /simulation/labnet
unit tests on the added/changed features
- make tests compile
- make tests pass
add logs allowing easy debugging in case the changes caused problems
if the API has changed, update the API specification

`🤖 Generated by Copilot at 31252a7`

Summary

🔄🗃️📡

This pull request improves the endorsement handling and propagation in the protocol worker by introducing a cache for endorsements, a command-based communication system, and a LruCache for operations. It also removes unused code and renames some modules for clarity. The main files affected are connectivity.rs, endorsement_handler/mod.rs, endorsement_handler/propagation.rs, endorsement_handler/retrieval.rs, operation_handler/propagation.rs, and endorsement_handler/cache.rs. The file endorsement_handler/internal_messages.rs was deleted and the file endorsement_handler/commands.rs was renamed to endorsement_handler/commands_propagation.rs.

We're haulin' on the EndorsementCache to store the votes we've seen
We're sendin' out the PropagationCommand to spread them 'cross the screen
We're workin' hard to refactor and improve the handler's code
We're sailin' on the blockchain with a lighter, faster load

Walkthrough

Rename the endorsement handler commands module from commands to commands_propagation and update the import statements accordingly (link, link, link, link, link)
Add the cache module for endorsements to the endorsement handler module and the connectivity module, and define the EndorsementCache struct and the SharedEndorsementCache type alias (link, link, link, link, link)
Modify the parameters of the endorsement handler constructor, adding the cache, the config, the sender_endorsements_ext channel, and removing the receiver_endorsements_ext channel (link, link)
Modify the logic of the propagation thread of the endorsement handler, replacing the use of the internal message channel with the use of the local receiver channel, and replacing the use of the cached endorsement ids with the use of the shared endorsement cache (link)
Modify the logic of the retrieval thread of the endorsement handler, replacing the use of the receiver_ext channel and the internal message channel with the use of the internal sender channel, and replacing the use of the cached endorsement ids with the use of the shared endorsement cache (link)
Modify the parameters of the retrieval thread constructor, adding the cache, the config, and the internal sender channel, and removing the receiver_ext channel and the internal sender channel (link)
Delete the internal_messages module of the endorsement handler, as it is no longer needed (link)
Add the import of the PeerMessageTuple type from the peer handler module to the endorsement handler module and the connectivity module, and use it to represent messages received from peers ( link)
Add the import of the LruCache type from the lru crate to the propagation module of the operation handler, and use it to implement a cache of operations known by peers (link, link)
Create a clone of the sender_endorsements_ext channel in the connectivity module, as it is moved into the endorsement handler constructor later (link)
Create a new instance of the EndorsementCache struct in the connectivity module, with the maximum size parameters set to the maximum possible values (link)

…el for all communications to the propagator

modship · 2023-04-14T13:47:49Z

massa-protocol-worker-2/src/handlers/endorsement_handler/propagation.rs

                            }
                        }
                    }
                }
                Err(err) => {
-                    println!("Error: {:?}", err);
+                    println!("Err: {:#?}", err);


maybe remove println! and use info!, warn!, error!

modship · 2023-04-14T13:50:04Z

massa-protocol-worker-2/src/handlers/endorsement_handler/retrieval.rs

+                                .send(EndorsementHandlerCommand::PropagateEndorsements(
+                                    endorsements_storage,
+                                ))
+                                .unwrap();


No, good catch !

modship

maybe replace println! by log (info, warn, error) macro

* Add skeleton new massa protocol. * Add architecture handler * Add basic peer management to massa * Setup a simple test. * Get best peer from peer manager * Fix connections to not retry if unnecessary * Add operations handler * Add handler skeleton endorsement * Add block handler skeleton * Improve announcement serialization * Add endorsement default behavior * Add pool controller * Add channels to communicate from handlers to outside world * Fix tests. * Add test endorsements to complete * Add todo ip * Rework network deserialize messages (#3750) * Use new branch peernet * Change handlers messages deserialization architecture. * Remove un-necessary serialization of message ID in messages Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Also deletes for the other type of message Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Remove unused lines Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Refactor serializing of message (#3786) * Add new way of serializing messages. * Use peernet default branch * Format and add comment * Rework id management * Add comment messages * Clean basic test * add command channel in PeerManagementHandler (#3809) * improves the peer management and communication logic * Add structure of operation handler (#3817) * Add structure of operation handler * Update cache to use a simplified Lru and share it. Use the same channel for all communications to the propagator * Add send of op and remove peers from cache when needed * Update lock * Add management of messages received from retrieval. * Change timers management and add pruning * Update caches and add storage to keep ops to be propagated * Remove useless line and add a comment * Remove useless write locks. * ban peer if note_operations_from_peer is Err (#3826) --------- Co-authored-by: Modship <yeskinokay@gmail.com> * Add basic endorsement handler code (#3827) * Add structure of operation handler * Update cache to use a simplified Lru and share it. Use the same channel for all communications to the propagator * Add send of op and remove peers from cache when needed * Update lock * Add management of messages received from retrieval. * Change timers management and add pruning * Update caches and add storage to keep ops to be propagated * Remove useless line and add a comment * Remove useless write locks. * Add basic code for endorsement handler * Fix runtime error in tests (#3836) * Network refactoring tester (#3841) * add 'thread_tester_count' in config.toml * refactor Tester::new * move thread_tester_count from NetworkConfig to ProtocolConfig * create shared channel for thread tester // add stop command to PeerManagementHandler * Tester : test the listener on recv msg * remove unwrap and print error * Add basic skeleton for block handler and fix endorsement one (#3840) * tmp Signed-off-by: Litchi Pi <litchi.pi@proton.me> * drop manager Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Add two different thread for block handler * fixup Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Add cache of block * Comment * Fix handler handshake in massa protocol. * Add basic block handler * Add skeleton reception message of block infos * Add all the code of the block handler * Finish code of block retrieval thread * Add the commands of propagation handling and fix compilation error * Add debug print and fix the peer that didn't connected after handshake * Fix listeners created too late. * Fix review notes. * Change ban node to warn in case it fails --------- Signed-off-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Litchi Pi <litchi.pi@proton.me> * Add stop command to all thread network refactoring (#3858) * tmp Signed-off-by: Litchi Pi <litchi.pi@proton.me> * drop manager Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Add two different thread for block handler * fixup Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Add cache of block * Comment * Fix handler handshake in massa protocol. * Add basic block handler * Add skeleton reception message of block infos * Add all the code of the block handler * Finish code of block retrieval thread * Add the commands of propagation handling and fix compilation error * Add debug print and fix the peer that didn't connected after handshake * Fix listeners created too late. * Fix review notes. * Change ban node to warn in case it fails * Add channels for exterior world on all handlers thread and add stop command * Update ports test * Add stop command on handler so that even if there is clone of the sender they are killed --------- Signed-off-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Litchi Pi <litchi.pi@proton.me> * send 100 peers to connected peers (#3864) * get_peers_to_send * update fn * send 100 random peers to all peer connected each 10 sec * use tuple for sender/receiver in PeerManagementHandler for readability * Wrap interaction with peernet to mock them (#3865) * Add basic mocks for tests * Wrap all interactions to peernet with a controller to mock it * Avoid overriding cache value * Add mock in test and port the first test * Update massa-protocol-worker-2/src/tests/ask_block_scenarios.rs Co-authored-by: Modship <yeskinokay@gmail.com> * Update massa-protocol-worker-2/src/tests/ask_block_scenarios.rs Co-authored-by: Modship <yeskinokay@gmail.com> * Change a todo --------- Co-authored-by: Modship <yeskinokay@gmail.com> * Fix cargo lock * Fix compil error after merge * Port tests network refactoring. (#3869) * Port tests block workflow. * Add more test on block handler and fix some bugs found. * Port tests in block operations and add propagation of operations from block handler to operation one * Add scenarios endorsements * Port operation tests * Port test ban_node and add disconnect of a banned node. * Add cache scenario * Fix compilation tests. * Fix last endorsement test * Change comment ban scenarios * Fix tests protocol * Update config on new protocol network and clean-up (#3875) * Add name to threads and more config values. * Bound all channels remove unwraps * Fix clippy warnings * Update config tests * network_refactoring - send peers in handshake (#3870) * send peers in handshake * remove peer_db.read() and use previous lock * remove useless scope * restore scope and send peers when perform_handshake start * fix : use saturating_sub * update massa network with new peernet trait * add serializer for peer management message in MassaHandshake to avoid many new serializer call * Add Clone derive to struct for serializer * send peers on failback_function + remove unwrap * add panic hook in test * remove revision tag for peernet in Cargo.toml and update it. * update OutConnectionConfig::Tcp with default value (can set rate limit) * remove dbg * Use new protocol (#3879) * Delete old network and protocol * Use the new massa-protocol * Change import of ProtocolCommandSender to ProtocolController * Fix all compile errors. * Fix all error in tests and use the keypair file. * Use limiter from settings * Use routable ip * Update peernet * Add all variables to config to launch the node * Fix compilation of API with new protocol * Fix compilation of the node * Remove network exports from bootstrap and replace interaction with protocol ones * Fix(tests) : test_peer_connected && test_list_peers (#3882) * Remove network and fix tests compilation * Fix all tests * Use initial peers from bootstrap and change initial peers file * Remove old network category in config. * Change print stop to info * Don't test peers if we are already connected to * Don't test our local ips. * Increase maximum limitation socket. * Update peernet * Add debug prints * Update peernet * Update peernet * Fix endorsement merge deletion * Try to disable endorsement pool to test * Fix peer not registered when received from remote. --------- Co-authored-by: Modship <yeskinokay@gmail.com> * Use new protocol #2 (#3886) * Delete old network and protocol * Use the new massa-protocol * Change import of ProtocolCommandSender to ProtocolController * Fix all compile errors. * Fix all error in tests and use the keypair file. * Use limiter from settings * Use routable ip * Update peernet * Add all variables to config to launch the node * Fix compilation of API with new protocol * Fix compilation of the node * Remove network exports from bootstrap and replace interaction with protocol ones * Fix(tests) : test_peer_connected && test_list_peers (#3882) * Remove network and fix tests compilation * Fix all tests * Use initial peers from bootstrap and change initial peers file * Remove old network category in config. * Change print stop to info * Don't test peers if we are already connected to * Don't test our local ips. * Increase maximum limitation socket. * Update peernet * Add debug prints * Update peernet * Update peernet * Fix endorsement merge deletion * Try to disable endorsement pool to test * Fix peer not registered when received from remote. * Add debug print * Add debug prints * Add debug print * Fix endorsement note endorsements * Add debug print * Add more debug print * Update peernet * Update peernet * Update peernet * Remove all debug prints * Display all len of all vectors to spot memory leak * Remove debug memory print * Add print when new peer connected * Add setup nasm in CI --------- Co-authored-by: Modship <yeskinokay@gmail.com> * Update port tests * Add todo on api * Add new hack comment * Remove useless scope * Use type PeerMessageTuple in messages handler * Added todo * Fix get rand peers and debug print. * Send myself in bootstrap peers & don't send local addresses. * Fix add of myself in bootstrap peers. * Change add myself * Add debug print and reduce lock length * Fix received fallback data in handshake. * Fix error offset in tester handshake * Update peernet * Fix tests and remove debug print * Update peernet. * Add debug print * Update peernet * Add more print deadlock * Update peernet * Remove debug print * Update announce in peerdb when handshake worked. * Readd some debug print. * Fix do not test our ip. --------- Signed-off-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Modship <yeskinokay@gmail.com>

AurelienFT added 12 commits April 12, 2023 13:23

Add structure of operation handler

d1c2aff

Update cache to use a simplified Lru and share it. Use the same chann…

23735c6

…el for all communications to the propagator

Add send of op and remove peers from cache when needed

434ac4a

Update lock

84e580c

Add management of messages received from retrieval.

5a4e1a7

Change timers management and add pruning

e814860

Update caches and add storage to keep ops to be propagated

ea36640

Merge branch 'network_refactoring' into add_operation_handler

f04b40b

Remove useless line and add a comment

7c13e4d

Remove useless write locks.

88f5e25

Add basic code for endorsement handler

8c5ad2a

Merge branch 'network_refactoring' into add_endorsement_handler

31252a7

AurelienFT marked this pull request as ready for review April 14, 2023 13:07

AurelienFT requested review from modship and litchipi April 14, 2023 13:07

AurelienFT self-assigned this Apr 14, 2023

modship reviewed Apr 14, 2023

View reviewed changes

modship approved these changes Apr 14, 2023

View reviewed changes

litchipi approved these changes Apr 14, 2023

View reviewed changes

Remove unwrap and change print to warn.

d7c5fbc

AurelienFT merged commit 7ee5be5 into network_refactoring Apr 17, 2023

AurelienFT deleted the add_endorsement_handler branch May 29, 2023 09:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add basic endorsement handler code #3827

Add basic endorsement handler code #3827

AurelienFT commented Apr 14, 2023 •

edited by ghost

Loading

modship Apr 14, 2023

AurelienFT Apr 17, 2023

modship Apr 14, 2023

AurelienFT Apr 17, 2023

modship left a comment

Add basic endorsement handler code #3827

Add basic endorsement handler code #3827

Conversation

AurelienFT commented Apr 14, 2023 • edited by ghost Loading

🤖 Generated by Copilot at 31252a7

Summary

Walkthrough

modship Apr 14, 2023

Choose a reason for hiding this comment

AurelienFT Apr 17, 2023

Choose a reason for hiding this comment

modship Apr 14, 2023

Choose a reason for hiding this comment

AurelienFT Apr 17, 2023

Choose a reason for hiding this comment

modship left a comment

Choose a reason for hiding this comment

AurelienFT commented Apr 14, 2023 •

edited by ghost

Loading

`🤖 Generated by Copilot at 31252a7`