Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WARN near_store::db::rocksdb: target="store::db::rocksdb" making a write batch took a very long time, make smaller transactions! #11926

Closed
MaxZubrytskyi opened this issue Aug 13, 2024 · 3 comments

Comments

@MaxZubrytskyi
Copy link

Describe the bug
I'm starting fresh node, v2.0.0, config from off url, downloaded latest snapshot, my machine is pretty good one, 64 cores, 256GB RAM and few NVME in RAID, first it does download 1.4TB snapshot, then start pulling latest blocks to sync but warning started and it is keep on "downloading latest blocks" for 12h alredy (1.4TB snapshot was pulled within 4h)
warning message like

WARN near_store::db::rocksdb: target="store::db::rocksdb" making a write batch took a very long time, make smaller transactions! elapsed=178.067339ms backtrace=   0: <near_store::db::rocksdb::RocksDB as near_store::db::Database>::write
   1: near_store::StoreUpdate::commit
   2: near_chain::store::ChainStoreUpdate::commit
   3: near_chain::chain::Chain::postprocess_block
   4: near_client::client_actor::ClientActorInner::try_process_unfinished_blocks
   5: near_client::client_actor::ClientActorInner::check_triggers
   6: <actix::address::envelope::SyncEnvelopeProxy<M> as actix::address::envelope::EnvelopeProxy<A>>::handle
   7: <actix::contextimpl::ContextFut<A,C> as core::future::future::Future>::poll
   8: tokio::runtime::task::raw::poll
   9: tokio::task::local::LocalSet::tick
  10: tokio::task::local::LocalSet::run_until::{{closure}}
  11: std::sys_common::backtrace::__rust_begin_short_backtrace
  12: core::ops::function::FnOnce::call_once{{vtable.shim}}
  13: std::sys::pal::unix::thread::Thread::new::thread_start
  14: start_thread
  15: clone

To Reproduce
just start node

Expected behavior
fully synced and start work

Version (please complete the following information):

  • neard (release 2.0.0) (build unknown) (rustc 1.78.0) (protocol 69) (db 39)
    features: [default, json_rpc, rosetta_rpc]
  • mainnet
@MaxZubrytskyi
Copy link
Author

update: it does finished sync to latest block within 20h and once sync it does randomly throw this warning and right after it it keeps downloading last 10-30 blocks, issue is intermittent and occurs every 10-15m

also noticed huge disk IO usage.
maybe there are some settings which could help to reduce disk load?

@MaxZubrytskyi
Copy link
Author

cpu: AMD EPYC 7502P 32-Core Processor
RAM: 256GB (8x32GB) Speed: 3200 MT/s Manufacturer: Micron Technology
NVME: Micron_7450_MTFDKCC7T6TFR

@MaxZubrytskyi
Copy link
Author

we did replace server with new one, which do have gen V4 NVME slots (which does increase IOPS for us, as we does ahve gen3 slots with gen4 nvme, so that nvme were unable to use max performance ) and issue is gone, so for all who faced same: it was a disk pressure issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant