-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mark reserved peers as explicit for gossipsub to avoid reputation decreasing #1423
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bvrooman
approved these changes
Oct 16, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the production configuration, are the reserved nodes the sentry nodes? I.e., Is it the sentry nodes that are losing reputation when gossiping/publishing to the authority and eventually getting cut off?
Or is it your node that is getting cut off from the sentry nodes?
xgreenx
added a commit
that referenced
this pull request
Oct 20, 2023
…reasing (#1423) Closes #1384 After debugging the issue with transaction broadcasting in the Beta 4 network, I can confirm that it is related to the gossipsub reputation. At some point, no one sends transactions to the authority node. When I manually restart one sentry node, this node starts to send transactions to the authority node. It sends transactions via gossiping and publishing: - Publishing: I send a transaction to sentry 1, which sends it to authority. - Gossiping: I send the transaction to sentry 0, it sends the transaction to sentry 1, and it sends the transaction to authority. But at some point, gossiping doesn't work while publishing still works(Sentry 0 gossips transaction to Sentry 1, and Sentry 1 doesn't gossip it to authority). After some time, the publishing doesn't work either. The gossipsub has two thresholds, one for gossiping and one for publishing. We use these values: <img width="652" alt="image" src="https://github.com/FuelLabs/fuel-core/assets/18346821/bcb1585c-239b-4cf9-9fae-c9376b958201"> So, the described behavior aligns with authority reputation decreasing. I don't know the reason why it happens(maybe decay), but adding all reserved peers to explicit peers should solve the problem.
xgreenx
added a commit
that referenced
this pull request
Oct 20, 2023
…reasing (#1423) (#1434) Applying the bugfix #1423 to `master`. Closes #1384 After debugging the issue with transaction broadcasting in the Beta 4 network, I can confirm that it is related to the gossipsub reputation. At some point, no one sends transactions to the authority node. When I manually restart one sentry node, this node starts to send transactions to the authority node. It sends transactions via gossiping and publishing: - Publishing: I send a transaction to sentry 1, which sends it to authority. - Gossiping: I send the transaction to sentry 0, it sends the transaction to sentry 1, and it sends the transaction to authority. But at some point, gossiping doesn't work while publishing still works(Sentry 0 gossips transaction to Sentry 1, and Sentry 1 doesn't gossip it to authority). After some time, the publishing doesn't work either. The gossipsub has two thresholds, one for gossiping and one for publishing. We use these values: <img width="652" alt="image" src="https://github.com/FuelLabs/fuel-core/assets/18346821/bcb1585c-239b-4cf9-9fae-c9376b958201"> So, the described behavior aligns with authority reputation decreasing. I don't know the reason why it happens(maybe decay), but adding all reserved peers to explicit peers should solve the problem.
bvrooman
pushed a commit
that referenced
this pull request
Nov 24, 2023
## Version 0.21.0 This release focuses on preparing `fuel-core` for the mainnet environment. - Most of the changes improved the security and stability of the node. - The gas model was reworked to cover all aspects of execution. - The benchmarking system was significantly enhanced, covering worst scenarios. - A new set of benchmarks was added to track the accuracy of gas prices. - Optimized heavy operations and removed/replaced exploitable functionality. Besides that, there are more concrete changes: - Unified naming conventions for all CLI arguments. Added dependencies between related fields to avoid misconfiguration in case of missing arguments. Added `--debug` flag that enables additional functionality like a debugger. - Improved telemetry to cover the internal work of services and added support for the Pyroscope, allowing it to generate real-time flamegraphs to track performance. - Improved stability of the P2P layer and adjusted the updating of reputation. The speed of block synchronization was significantly increased. - The node is more stable and resilient. Improved DoS resistance and resource management. Fixed critical bugs during state transition. - Reworked the `Mint` transaction to accumulate the fee from block production inside the contract defined by the block producer. FuelVM received a lot of safety and stability improvements: - The audit helped identify some bugs and errors that have been successfully fixed. - Updated the gas price model to charge for resources used during the transaction lifecycle. - Added `no_std` and 32 bit system support. This opens doors for fraud proving in the future. - Removed the `ChainId` from the `PredicateId` calculation, allowing the use of predicates cross-chain. - Improvements in the performance of some storage-related opcodes. - Support the `ECAL` instruction that allows adding custom functionality to the VM. It can be used to create unique rollups or advanced indexers in the future. - Support of [transaction policies](https://github.com/FuelLabs/fuel-vm/blob/master/CHANGELOG.md#version-0420) provides additional safety for the user. It also allows the implementation of a multi-dimensional price model in the future, making the transaction execution cheaper and allowing more transactions that don't affect storage. - Refactored errors, returning more detailed errors to the user, simplifying debugging. ### Added - [#1503](#1503): Add `gtf` opcode sanity check. - [#1502](#1502): Added price benchmark for `vm_initialization`. - [#1501](#1501): Add a CLI command for generating a fee collection contract. - [#1492](#1492): Support backward iteration in the RocksDB. It allows backward queries that were not allowed before. - [#1490](#1490): Add push and pop benchmarks. - [#1485](#1485): Prepare rc release of fuel core v0.21 - [#1476](#1453): Add the majority of the "other" benchmarks for contract opcodes. - [#1473](#1473): Expose fuel-core version as a constant - [#1469](#1469): Added support of bloom filter for RocksDB tables and increased the block cache. - [#1465](#1465): Improvements for keygen cli and crates - [#1642](#1462): Added benchmark to measure the performance of contract state and contract ID calculation; use for gas costing. - [#1457](#1457): Fixing incorrect measurement for fast(µs) opcodes. - [#1456](#1456): Added flushing of the RocksDB during a graceful shutdown. - [#1456](#1456): Added more logs to track the service lifecycle. - [#1453](#1453): Add the majority of the "sanity" benchmarks for contract opcodes. - [#1452](#1452): Added benchmark to measure the performance of contract root calculation when utilizing the maximum contract size; used for gas costing of contract root during predicate owner validation. - [#1449](#1449): Fix coin pagination in e2e test client. - [#1447](#1447): Add timeout for continuous e2e tests - [#1444](#1444): Add "sanity" benchmarks for memory opcodes. - [#1437](#1437): Add some transaction throughput tests for basic transfers. - [#1436](#1436): Add a github action to continuously test beta-4. - [#1433](#1433): Add "sanity" benchmarks for flow opcodes. - [#1432](#1432): Add a new `--api-request-timeout` argument to control TTL for GraphQL requests. - [#1430](#1430): Add "sanity" benchmarks for crypto opcodes. - [#1426](#1426) Split keygen into a create and a binary. - [#1419](#1419): Add additional "sanity" benchmarks for arithmetic op code instructions. - [#1411](#1411): Added WASM and `no_std` compatibility. - [#1405](#1405): Use correct names for service metrics. - [#1400](#1400): Add releasy beta to fuel-core so that new commits to fuel-core master triggers fuels-rs. - [#1371](#1371): Add new client function for querying the `MessageStatus` for a specific message (by `Nonce`). - [#1356](#1356): Add peer reputation reporting to heartbeat code. - [#1355](#1355): Added new metrics related to block importing, such as tps, sync delays etc. - [#1339](#1339): Adds `baseAssetId` to `FeeParameters` in the GraphQL API. - [#1331](#1331): Add peer reputation reporting to block import code. - [#1324](#1324): Added pyroscope profiling to fuel-core, intended to be used by a secondary docker image that has debug symbols enabled. - [#1309](#1309): Add documentation for running debug builds with CLion and Visual Studio Code. - [#1308](#1308): Add support for loading .env files when compiling with the `env` feature. This allows users to conveniently supply CLI arguments in a secure and IDE-agnostic way. - [#1304](#1304): Implemented `submit_and_await_commit_with_receipts` method for `FuelClient`. - [#1286](#1286): Include readable names for test cases where missing. - [#1274](#1274): Added tests to benchmark block synchronization. - [#1263](#1263): Add gas benchmarks for `ED19` and `ECR1` instructions. ### Changed - [#1512](#1512): Internally simplify merkle_contract_state_range. - [#1507](#1507): Updated chain configuration to be ready for beta 5 network. It includes opcode prices from the latest benchmark and contract for the block producer. - [#1477](#1477): Upgraded the Rust version used in CI and containers to 1.73.0. Also includes associated Clippy changes. - [#1469](#1469): Replaced usage of `MemoryTransactionView` by `Checkpoint` database in the benchmarks. - [#1468](#1468): Bumped version of the `fuel-vm` to `v0.40.0`. It brings some breaking changes into consensus parameters API because of changes in the underlying types. - [#1466](#1466): Handling overflows during arithmetic operations. - [#1460](#1460): Change tracking branch from main to master for releasy tests. - [#1454](#1454): Update gas benchmarks for opcodes that append receipts. - [#1440](#1440): Don't report reserved nodes that send invalid transactions. - [#1439](#1439): Reduced memory BMT consumption during creation of the header. - [#1434](#1434): Continue gossiping transactions to reserved peers regardless of gossiping reputation score. - [#1408](#1408): Update gas benchmarks for storage opcodes to use a pre-populated database to get more accurate worst-case costs. - [#1399](#1399): The Relayer now queries Ethereum for its latest finalized block instead of using a configurable "finalization period" to presume finality. - [#1397](#1397): Improved keygen. Created a crate to be included from forc plugins and upgraded internal library to drop requirement of protoc to build - [#1395](#1395): Add DependentCost benchmarks for `k256`, `s256` and `mcpi` instructions. - [#1393](#1393): Increase heartbeat timeout from `2` to `60` seconds, as suggested in [this issue](#1330). - [#1392](#1392): Fixed an overflow in `message_proof`. - [#1390](#1390): Up the `ethers` version to `2` to fix an issue with `tungstenite`. - [#1383](#1383): Disallow usage of `log` crate internally in favor of `tracing` crate. - [#1380](#1380): Add preliminary, hard-coded config values for heartbeat peer reputation, removing `todo`. - [#1377](#1377): Remove `DiscoveryEvent` and use `KademliaEvent` directly in `DiscoveryBehavior`. - [#1366](#1366): Improve caching during docker builds in CI by replacing gha - [#1358](#1358): Upgraded the Rust version used in CI to 1.72.0. Also includes associated Clippy changes. - [#1349](#1349): Updated peer-to-peer transactions API to support multiple blocks in a single request, and updated block synchronization to request multiple blocks based on the configured range of headers. - [#1342](#1342): Add error handling for P2P requests to return `None` to requester and log error. - [#1318](#1318): Modified block synchronization to use asynchronous task execution when retrieving block headers. - [#1314](#1314): Removed `types::ConsensusParameters` in favour of `fuel_tx:ConsensusParameters`. - [#1302](#1302): Removed the usage of flake and building of the bridge contract ABI. It simplifies the maintenance and updating of the events, requiring only putting the event definition into the codebase of the relayer. - [#1293](#1293): Parallelized the `estimate_predicates` endpoint to utilize all available threads. - [#1270](#1270): Modify the way block headers are retrieved from peers to be done in batches. #### Breaking - [#1506](#1506): Added validation of the coin's fields during block production and validation. Before, it was possible to submit a transaction that didn't match the coin's values in the database, allowing printing/using unavailable assets. - [#1491](#1491): Removed unused request and response variants from the Gossipsub implementation, as well as related definitions and tests. Specifically, this removes gossiping of `ConsensusVote` and `NewBlock` events. - [#1472](#1472): Upgraded `fuel-vm` to `v0.42.0`. It introduces transaction policies that changes layout of the transaction. FOr more information check the [v0.42.0](FuelLabs/fuel-vm#635) release. - [#1470](#1470): Divide `DependentCost` into "light" and "heavy" operations. - [#1464](#1464): Avoid possible truncation of higher bits. It may invalidate the code that truncated higher bits causing different behavior on 32-bit vs. 64-bit systems. The change affects some endpoints that now require lesser integers. - [#1432](#1432): All subscriptions and requests have a TTL now. So each subscription lifecycle is limited in time. If the subscription is closed because of TTL, it means that you subscribed after your transaction had been dropped by the network. - [#1407](#1407): The recipient is a `ContractId` instead of `Address`. The block producer should deploy its contract to receive the transaction fee. The collected fee is zero until the recipient contract is set. - [#1407](#1407): The `Mint` transaction is reworked with new fields to support the account-base model. It affects serialization and deserialization of the transaction and also affects GraphQL schema. - [#1407](#1407): The `Mint` transaction is the last transaction in the block instead of the first. - [#1374](#1374): Renamed `base_chain_height` to `da_height` and return current relayer height instead of latest Fuel block height. - [#1367](#1367): Update to the latest version of fuel-vm. - [#1363](#1363): Change message_proof api to take `nonce` instead of `message_id` - [#1355](#1355): Removed the `metrics` feature flag from the fuel-core crate, and metrics are now included by default. - [#1339](#1339): Added a new required field called `base_asset_id` to the `FeeParameters` definition in `ConsensusParameters`, as well as default values for `base_asset_id` in the `beta` and `dev` chain specifications. - [#1322](#1322): The `debug` flag is added to the CLI. The flag should be used for local development only. Enabling debug mode: - Allows GraphQL Endpoints to arbitrarily advance blocks. - Enables debugger GraphQL Endpoints. - Allows setting `utxo_validation` to `false`. - [#1318](#1318): Removed the `--sync-max-header-batch-requests` CLI argument, and renamed `--sync-max-get-txns` to `--sync-block-stream-buffer-size` to better represent the current behavior in the import. - [#1290](#1290): Standardize CLI args to use `-` instead of `_`. - [#1279](#1279): Added a new CLI flag to enable the Relayer service `--enable-relayer`, and disabled the Relayer service by default. When supplying the `--enable-relayer` flag, the `--relayer` argument becomes mandatory, and omitting it is an error. Similarly, providing a `--relayer` argument without the `--enable-relayer` flag is an error. Lastly, providing the `--keypair` or `--network` arguments will also produce an error if the `--enable-p2p` flag is not set. - [#1262](#1262): The `ConsensusParameters` aggregates all configuration data related to the consensus. It contains many fields that are segregated by the usage. The API of some functions was affected to use lesser types instead the whole `ConsensusParameters`. It is a huge breaking change requiring repetitively monotonically updating all places that use the `ConsensusParameters`. But during updating, consider that maybe you can use lesser types. Usage of them may simplify signatures of methods and make them more user-friendly and transparent. ### Removed #### Breaking - [#1484](#1484): Removed `--network` CLI argument. Now the name of the network is fetched form chain configuration. - [#1399](#1399): Removed `relayer-da-finalization` parameter from the relayer CLI. - [#1338](#1338): Updated GraphQL client to use `DependentCost` for `k256`, `mcpi`, `s256`, `scwq`, `swwq` opcodes. - [#1322](#1322): The `manual_blocks_enabled` flag is removed from the CLI. The analog is a `debug` flag. ## What's Changed * Added changelog in the same way as we did for `fuel-vm` by @xgreenx in #1287 * Decompose consensus params struct by @MitchTurner in #1262 * Replace all instances of `_` in CLI `long` arguments with `-` by @MitchTurner in #1290 * [rpc] Parallelise estimate_predicates endpoint by @YusongWang in #1293 * Add readable test names to test cases by @MitchTurner in #1286 * ED19 and ECR1 benchmarks by @Dentosal in #1263 * feat: Add `enable-relayer` flag to `fuel-core` CLI by @bvrooman in #1279 * Small nits found during deploying and benchmarking by @xgreenx in #1303 * Implemented `submit_and_await_commit_with_receipts` method by @xgreenx in #1304 * Removed the usage of flake and building of the bridge contract ABI by @xgreenx in #1302 * Download range of headers during syncing by @MitchTurner in #1270 * feat: Enable .env file loading using `env` feature by @bvrooman in #1308 * chore: Remove types::ConsensusParameters by @bvrooman in #1314 * docs: Debugging instructions using IDEs by @bvrooman in #1309 * test: Block Import benchmarks and test helpers by @bvrooman in #1274 * Fix audit CI for `webpki` by @xgreenx in #1319 * chore: Single buffer block sync by @bvrooman in #1318 * Run unit tests without default features only for specific crates by @xgreenx in #1321 * Added support for `debug` CLI flag. by @xgreenx in #1322 * Report Peers that give bad Block Info by @MitchTurner in #1331 * Pyroscope profiling by @Voxelot in #1324 * Add error handling for P2P requests to return `None` to requester and log error by @MitchTurner in #1342 * Importer Metrics by @Voxelot in #1355 * chore: Upgrade to Rust 1.72.0 in CI by @bvrooman in #1358 * chore: Upgrade `fuel-vm` to 0.37.0 by @bvrooman in #1338 * docs: remove stale schema by @calldelegation in #1362 * Replace message_id with nonce in `MessageProof` query by @MitchTurner in #1363 * feat: Configurable base asset by @bvrooman in #1339 * avoid gha caching for docker builds by @Voxelot in #1366 * Add tracking for peer heartbeats by @MitchTurner in #1356 * docs: fix broken link in README by @PaulRBerg in #1357 * Use right level of error for `.env` file and duplicated service by @xgreenx in #1372 * Add query and handling for `MessageStatus` by @MitchTurner in #1371 * Renamed `base_chain_height` to `da_height` and return current relayer height instead of latest Fuel block height by @xgreenx in #1374 * Remove `DiscoveryEvent` and use `KademliaEvent` instead by @MitchTurner in #1377 * Remove todo from code and provide some preliminary peer review config values by @MitchTurner in #1380 * Clippy lint to disallow logging crate macros in favor of tracing by @Dentosal in #1383 * fuel-vm nostd update by @Dentosal in #1367 * Up the `ethers` version to `2` to fix an issue with `tungstenite` by @xgreenx in #1390 * Increase heartbeat timeout to one minute by @Dentosal in #1393 * feat: batch transactions import by @bvrooman in #1349 * Fix an interger overflow in message_proof when looking beyond genesis by @Dentosal in #1392 * Improve keygen by @cr-fuel in #1397 * Benchmark {k256, s256, mcpi} opcodes using DependentCost by @Dentosal in #1395 * Use correct names for service metrics by @xgreenx in #1405 * feat: Retrieve finalized blocks from Ethereum by @bvrooman in #1399 * Updated CI to run `cargo update` weekly by @xgreenx in #1415 * CI `cargo update` weekly by @xgreenx in #1416 * Weekly `cargo update` by @github-actions in #1417 * Fix typos by @GoodDaisy in #1424 * feat: Support no_std and WASM compilation for `fuel-core` crates by @bvrooman in #1411 * Create sanity benchmark checks for all the arithmetic op codes by @MitchTurner in #1419 * Update benchmarking and collecting for `scwq`, `swwq` and `srwq` opcodes by @xgreenx in #1427 * Account-base fee collection by @xgreenx in #1407 * Rollback modification of the chain specification by @xgreenx in #1429 * Tx subscription cleanup (#1422) by @xgreenx in #1432 * Mark reserved peers as explicit for gossipsub to avoid reputation decreasing (#1423) by @xgreenx in #1434 * Weekly `cargo update` by @github-actions in #1438 * ci: add beta releasy to fuel-core by @kayagokalp in #1400 * setup a cron job to test beta-4 continuously by @Voxelot in #1436 * chore: Use MerkleRootCalculator when only BMT root is needed by @bvrooman in #1439 * Crypto op code sanity checks by @MitchTurner in #1430 * Update gas benchmarks for some storage opcodes by @Dentosal in #1408 * Split keygen into a create and a binary by @cr-fuel in #1426 * Basic Transfer TPS Benchmarking by @Voxelot in #1437 * "flow" op code sanity benchmarks by @MitchTurner in #1433 * Applying `#1435` to `master` by @xgreenx in #1440 * "memory" opcode sanity checks by @MitchTurner in #1444 * Add timeout to E2E test by @Voxelot in #1447 * fix owns_coins to properly paginate by @Voxelot in #1449 * Fixing incorrect measurement for fast(µs) opcodes by @xgreenx in #1457 * Call `flush` during end of the `FuelService` by @xgreenx in #1456 * chore: fix releasy test tracking branch from `main` to `master` by @kayagokalp in #1460 * Weekly `cargo update` by @github-actions in #1461 * `DB::repair` breaks the table for checkpoints by @xgreenx in #1463 * Avoid possible truncation of higher bits by @xgreenx in #1464 * Bumping `fuel-vm` to `v0.40.0` by @xgreenx in #1468 * Improvements for keygen cli and crates by @cr-fuel in #1465 * Handling overflows during arithmetic operations by @xgreenx in #1466 * test: Add contract code root benchmark by @bvrooman in #1452 * test: Add state root and contract id benchmarks by @bvrooman in #1462 * Avoid storage caching in benchmarks by @xgreenx in #1469 * Weekly `cargo update` by @github-actions in #1471 * Expose fuel-core version as a constant by @Br1ght0ne in #1473 * Upgrade ed25519_dalek 1.0.1 -> 2.0.0 by @Dentosal in #1475 * chore: Upgrade Rust 1.73 by @bvrooman in #1477 * Decrease number of iterations for sequential benchmarks by @xgreenx in #1480 * Add mldv bench by @Dentosal in #1481 * feat: Separate light and heavy gas costs (core) by @bvrooman in #1483 * chore: Remove `network` parameter from CLI by @bvrooman in #1484 * Transaction policies support by @xgreenx in #1472 * stage release candidate for beta-5 by @Voxelot in #1485 * setup docker auth for binary publishing by @Voxelot in #1488 * Fix macos binary publishing by @xgreenx in #1489 * Add documentation for our calculation of a _reasonable_ gas fee for storage by @MitchTurner in #1478 * "contract" op code sanity checks by @MitchTurner in #1453 * "other" op code sanity check by @MitchTurner in #1476 * Add push and pop benches by @MitchTurner in #1490 * chore: Remove unused gossipsub items by @bvrooman in #1491 * Adjusting of the prices benchmarks after sanity check benchmarks by @xgreenx in #1494 * Weekly `cargo update` by @github-actions in #1498 * Update benches that are affected by receipts by @Dentosal in #1454 * Support backward iteration in the RocksDB by @xgreenx in #1492 * Added validation of the coin's fields during block production and validation by @xgreenx in #1506 * Added price benchmark for `vm_initialization` by @xgreenx in #1502 * Sanity-check `gtf` opcode by @MitchTurner in #1503 * Tool for generating fee collection contract by @Voxelot in #1501 * Chain config for beta 5 network by @xgreenx in #1507 * Remove usage of iter_all_filtered from merkle_contract_state_range by @Dentosal in #1512 ## New Contributors * @YusongWang made their first contribution in #1293 * @calldelegation made their first contribution in #1362 * @PaulRBerg made their first contribution in #1357 * @cr-fuel made their first contribution in #1397 * @github-actions made their first contribution in #1417 * @GoodDaisy made their first contribution in #1424 **Full Changelog**: v0.20.8...v0.21.0 --------- Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #1384
After debugging the issue with transaction broadcasting in the Beta 4 network, I can confirm that it is related to the gossipsub reputation.
At some point, no one sends transactions to the authority node. When I manually restart one sentry node, this node starts to send transactions to the authority node. It sends transactions via gossiping and publishing:
But at some point, gossiping doesn't work while publishing still works(Sentry 0 gossips transaction to Sentry 1, and Sentry 1 doesn't gossip it to authority).
After some time, the publishing doesn't work either.
The gossipsub has two thresholds, one for gossiping and one for publishing. We use these values:
So, the described behavior aligns with authority reputation decreasing. I don't know the reason why it happens(maybe decay), but adding all reserved peers to explicit peers should solve the problem.