headState does not exist

[wemeetagain]

Report from discord: https://discord.com/channels/593655374469660673/743858262864167062/1035585735983644702

version=v1.1.1/9d07393, commit=9d07393709f6a870f9eeda9bca7a9 5196ca67c68

Oct-28 15:55:18.000[]                error: Node notifier error  headState does not exist
Error: headState does not exist
    at BeaconChain.getHeadState (file:///usr/app/packages/beacon-node/src/chain/chain.ts:319:27)
    at runNodeNotifier (file:///usr/app/packages/beacon-node/src/node/notifier.ts:55:31)

We thought this was fixed in v1.1.0:

• Node notifier error headState does not exist #4523

[twoeths]

logs from when the issue was reported

Oct-28 15:55:06.001[]                 info: Searching peers - peers: 13 - slot: 4205375 (skipped 95) - head: 4205280 0x8b1e…8d4e - execution: valid(0xab62…7e3e) - finalized: 0x0e57…6e29:131413
Oct-28 15:55:18.000[]                error: Node notifier error  headState does not exist
Error: headState does not exist
    at BeaconChain.getHeadState (file:///usr/app/packages/beacon-node/src/chain/chain.ts:319:27)
    at runNodeNotifier (file:///usr/app/packages/beacon-node/src/node/notifier.ts:55:31)
Oct-28 15:55:40.250[NETWORK]          info: Subscribed gossip core topics
Oct-28 15:55:40.258[SYNC]             info: Subscribed gossip core topics

[SeaMonkey82]

Still seeing the same error on Version: v1.2.1/cdde39f

Slightly different error.

Dec-27 13:22:24.008[rest]            error: Req req-13q getStateValidators error  No state found for id 'head'
Error: No state found for id 'head'
    at resolveStateId (file:///home/seamonkey/Downloads/goerli/lodestar/packages/beacon-node/src/api/impl/beacon/state/utils.ts:43:11)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.getStateValidators (file:///home/seamonkey/Downloads/goerli/lodestar/packages/beacon-node/src/api/impl/beacon/state/index.ts:57:21)
    at Object.handler (file:///home/seamonkey/Downloads/goerli/lodestar/packages/api/src/utils/server/genericJsonServer.ts:41:23)
    ```

[philknows]

Is this on the current stable release v1.2.2 @SeaMonkey82 ? Re-opened for investigation or we can create a new issue if it's unrelated @tuyennhv .

[philknows]

#4944 was recently merged into unstable if you want to try the chainsafe/lodestar:next image instead @SeaMonkey82 ?

[SeaMonkey82]

1.2.1/d317aac (current unstable)

Dec-28 18:38:33.710[rest]            error: Req req-3 getStateValidators error  No state found for id 'head'
Error: No state found for id 'head'
    at resolveStateId (file:///home/seamonkey/Downloads/goerli/lodestar/packages/beacon-node/src/api/impl/beacon/state/utils.ts:43:11)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.getStateValidators (file:///home/seamonkey/Downloads/goerli/lodestar/packages/beacon-node/src/api/impl/beacon/state/index.ts:57:21)
    at Object.handler (file:///home/seamonkey/Downloads/goerli/lodestar/packages/api/src/utils/server/genericJsonServer.ts:41:23)

[twoeths]

issue was resolved as confirmed by @SeaMonkey82 https://discord.com/channels/593655374469660673/593655641445367808/1058019244090081351

root cause was #4955

[twoeths]

the issue happened randomly, was able to catch same issue after starting the node
beacon-2022-12-31.log.headState.issue.zip

when it happened the node was not able to recover and got stuck there

Dec-31 12:55:43.412[]                 ^[[32minfo^[[39m: Initializing beacon from a valid checkpoint state slot=       5472192, epoch=171006, stateRoot=0x141e95ab937e61c7322c5f57ccdf6bf3fdd6360ea9ca0a48e2abfdf5fa1a01bb, isWithinWe       akSubjectivityPeriod=true

Dec-31 12:57:32.736[chain]           debug: Persisted block to hot DB slot=5472285, root=0xd281e99a7db6b24218341204d7fba56d524fe844659568d0f48397b17c242948

Dec-31 12:57:32.736[chain]         verbose: Block processed slot=5472285, root=0xd281e99a7db6b24218341204d7fba56d524fe844659568d0f48397b17c242948, delaySec=9.736000061035156

Dec-31 12:57:41.000[]                 ^[[32minfo^[[39m: Searching peers - peers: 4 - slot: 5472286 (skipped 94)        - head: 5472192 0xb721…ad66 - execution: valid(0x823b…2793) - finalized: 0x26d0…1c84:171004

Dec-31 12:57:55.044[chain]           debug: Skipping PrepareNextSlotScheduler - head slot is too behind current slot nextEpoch=171009, headSlot=5472192, clockSlot=5472287

Dec-31 12:57:59.001[chain]         verbose: Fork choice finalized epoch=171007, root=0x5900917c9d5434c3c4e438f3f25656a742ebe6090dcca64ef574633eae5d9339

Dec-31 12:57:59.002[chain]         verbose: Fork choice justified epoch=171008, root=0x996a20a1070a9090345b48a48a9bb1ee8775b0a1339008987337f2e01f125b62

Dec-31 12:58:04.813[chain]           error: Block error slot=5472256 error=No node if forkChoice for blockRoot 0xb721a57c04faa24a03cdfee8a32b650324a2440beef6d379926fbeed63eaad66

Dec-31 12:58:04.813[sync]          ^[[36mverbose^[[39m: Batch process error id=Head, startEpoch=171008, status=Processing error=No node if forkChoice for blockRoot 0xb721a57c04faa24a03cdfee8a32b650324a2440beef6d379926fbeed       63eaad66

Dec-31 12:58:05.000[]                ^[[31merror^[[39m: Node notifier error  headState does not exist
  Error: headState does not exist
  at BeaconChain.getHeadState (file:///usr/src/lodestar/packages/beacon-node/src/chain/chain.ts:333:27)
  at runNodeNotifier (file:///usr/src/lodestar/packages/beacon-node/src/node/notifier.ts:55:31)
  at runNextTicks (node:internal/process/task_queues:61:5)
  at processTimers (node:internal/timers:497:9)

Dec-31 12:58:08.337[chain]         ^[[36mverbose^[[39m: Running prepareForNextSlot nextEpoch=171010, prepareSlot=5472289, headSlot=5472286, headRoot=0x27ebccaf3d2d34bf50698aec2d9d3e636236dfba0384eaaacbaa8deb12226e4f, isEpo       chTransition=false

block 5472286 was added to forkchoice, forkchoice.getCommonAncestorDistance() was called with error (since the old head was removed from forkchoice - head got stuck a bit after we start because we used fake justified checkpoint). The result is we didn't add postState to state cache, and did not persist the block.