0.9.8 tokio-runtime-worker polkadot_overseer: [CRASH]

[gregorst3]

Op: Ubuntu 20.04.2 LTS
Db: RocksDb
Polkadot release version: 0.9.8
Chain: Kusama
After having panic error with ParityDb and 0.9.8 release, I switched to RocksDb.

Node runs fine for minutes, and then:

Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.029  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x244396ffc4d577ad524d0be8bcc6e5bcae6e1c28edeec9f8df722b>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.210  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x8768b090c9ea3d3402ace6516516dc2bdaf4d037cb123f9b7a1ea8>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.210  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x08f4746db187618877050101f6a12c39f68704d74ed3c133fd6ab7>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.210  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x709c729872af586cbc694258d313e02d4157578ef9ebf85ade2813>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.425  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xd62a3d4259fbf19799e9b89e619b934dd815329d9d647fde812500>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.425  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x05a46703d0d25b87292bc7bd87fa0b599a82328379e7aee61a8c61>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.425  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x85fb2127bc6b23b3a610e9272c0fd1c77ac77b6442021e81f84f53>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.618  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xdc921a72c752cbf1f50f03b3e083ba95e31c503d4736d05fb48f01>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.618  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x0fec997d00c7fa95508fef6b70272934e0f3d1f934d193c6b0d037>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.757  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xd6921e4e5df867b51e2ef9f5d45709ba51f95a79d9792a707c084a>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.862  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xb028bab7ad0234dcfd91323fe468379097124e58078496e5fc4cd3>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.862  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x0731f49b5955d668566fabda9a1c8f7db2dada3dbe1c68c99f3f4f>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.919  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x388c7ee9a407c1d99fb01ae2ac5369ab1867768eea1d9124fb246c>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.988  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xd6be31f57f722e69e17655aab17863a4fa077bb8275fcba9098c74>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.988  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xbd4ccb42581c0be77fb01644dbfb622a9ae864c9b45cd76d0fe03b>
Jul 07 21:46:58 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:58.988  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xa1b2b0e6eb15ef40968cc679fcdc1b6a56d737bfd52f77dededd38>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.078  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xabf2185c5790e8a6961db89ee5cf2093e8340d1993628c7caf56a8>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.143  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x7d6ad53c28009b04772bb33ee83b9c973155debc76c7fef8d01192>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.143  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x3ece9326345c598078a1eb3b7f1f0c5745a184f14ef76bdd397ebc>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.220  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x73835df175a0f8aad221d9c08342b50f2e06bd4617402f22c372e8>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.280  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x35932078196eac7bfcbf946c29c1da59d057175847e7406f1a7494>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.280  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xd7bba4ec8eddfef4132fafd1dda7ab7958052c190519f86640b77c>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.373  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x5ee76ae09fb1666fa41534622deecc8f6cdd25376158351eb98f19>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.462  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x932c3bb49dc57a561016f2ccd6be8e0692538d8a753e67b22eb8ee>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.462  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x28a2b576ae62d2d583f338924af1d3624f9b8c5c511dd7139aba5f>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.545  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x20fc5f4616c447139ef862db918659658c742d6966824c223eb0f6>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.627  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x620f840cdb4172c672b926a59f8de821f7df7707c97b6252942d38>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.627  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x82dcc39382bae96995d4a1ab95adc618f5c8e56db6164cd7b9c8cc>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.711  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x464145dd3aa23315d7f137c92da99cb69f16816d88b5a4ff499242>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.801  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x1bb7ca12052c43d220c3f2aba165ce4c68ae5ac084ec8af819c3c3>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.801  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xd1ed4c9160ec040bf73dd364cc6be24c253daac3817cef0934c977>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.801  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x5b1f3ba5a41dfc838c232831910160c07b54e3996955e8deea0cb4>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.905  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x70341dd71c1625eb21d0c97d5c0882dfaed984a2cf3228afe53aba>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.985  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x3f864a6e201126274b3e757d4cb5ea97c5a8cd387bcc73bc57e94f>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.985  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xd87fadd81fa86970180aedc5519a3cb9c6bb6fad7be6d97c664947>
Jul 07 21:46:59 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:46:59.985  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x847786c73a71efe77cf4d0980dd34f1993006f1777d696d9269b09>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.056  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x591741c7f9d4c4b8d5163774f2b9e21697d4fc6d37587d08f963c7>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.149  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0xbbae2993cfc0b32f2197a19677ef828f66f2272c77275b9099fd29>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.149  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x63960a40422079de95dfe76b7254caf869c78f86ff8cec06c92cf2>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.149  WARN tokio-runtime-worker parachain::availability: Candidate included without being backed? candidate_hash=0x909d0dedf5a59962d2218a558f9758e0e927b784dd7d1b214d180e>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.188 ERROR tokio-runtime-worker parachain::overseer: Subsystem availability-store-subsystem appears unresponsive. signal=BlockFinalized(0x0d58ed4f38b05e999301682f3432cf9>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.189 ERROR tokio-runtime-worker polkadot_node_subsystem_util: error receiving message from subsystem context for job job="CandidateBackingJob" err=Context("Signal channe>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.194 ERROR tokio-runtime-worker polkadot_overseer: subsystem exited with error subsystem="statement-distribution-subsystem" err=FromOrigin { origin: "statement-distribut>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.195 ERROR tokio-runtime-worker polkadot_overseer: subsystem exited with error subsystem="candidate-validation-subsystem" err=FromOrigin { origin: "candidate-validation">
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.196 ERROR tokio-runtime-worker sc_service::task_manager: Essential task `overseer` failed. Shutting down service.
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.197 ERROR tokio-runtime-worker polkadot_overseer: subsystem exited with error subsystem="chain-api-subsystem" err=FromOrigin { origin: "chain-api", source: Context("Sig>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: 2021-07-07 21:47:00.197 ERROR tokio-runtime-worker polkadot_overseer: subsystem exited with error subsystem="availability-distribution-subsystem" err=FromOrigin { origin: "availability-dis>
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: Error:
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]:    0: Other: Essential task failed.
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: Backtrace omitted.
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: Run with RUST_BACKTRACE=1 environment variable to display it.
Jul 07 21:47:00 Kusama-Cosmoon polkadot[21267]: Run with RUST_BACKTRACE=full to include source snippets.
Jul 07 21:47:00 Kusama-Cosmoon systemd[1]: polkadot-validator.service: Main process exited, code=exited, status=1/FAILURE

[ordian]

@gregorst3 are you running the binary built in the debug mode (cargo build instead of cargo build --release)?

[gregorst3]

@ordian I downloaded the already compiled version from here: https://github.com/paritytech/polkadot/releases/download/v0.9.8/polkadot

[ordian]

Thanks for the report, closing as a duplicate of #3242.

[gregorst3]

@ordian I'm sorry to reply again, but the official binary release , should be compiled with "cargo build --release", maybe it has not been compiled with flag --release? After this error, the node crashes

[ordian]

the official binary should be compiled in release mode, otherwise it would have been much larger in size

I've asked because your previous report indicated an overflow check in unsigned integer arithmetic, which is normally disabled in release builds, but we explicitly enable debug assertions for release builds, which also enables them

polkadot/.gitlab-ci.yml

Line 121 in 3239775

RUSTFLAGS: "-Cdebug-assertions=y -Dwarnings"

now the problem with Subsystem availability-store-subsystem appears unresponsive. is likely due to a db transaction taking a long time (>10 secs) on startup, but that issue is tracked in #3242