Client: PoA Clique (Goerli) run "Error: could not find parent header"

[holgerd77]

When the client is run on PoA with the new clique changes in it gets the following error "Error: could not find parent header" thrown in the BlockHeader.validate() function when trying to store the blocks in the blockchain via BlockFetcher.store().

I did some debugging this morning: this is due the the hash of the blocks being wrong respectively the BlockHeader.hash() function not delivering the correct hash, here is some output:

1 e26ba58f7923693693f3b6279b53bb29e17d6c7d1779bf2c793c14c969abf660 p:bf7e331f7f7c1dd2e05159666b3bf8bc7a8a3a9eb1d518969eab529dd9b88c1a 18
1 e26ba58f7923693693f3b6279b53bb29e17d6c7d1779bf2c793c14c969abf660 p:bf7e331f7f7c1dd2e05159666b3bf8bc7a8a3a9eb1d518969eab529dd9b88c1a 19
2 14db95de34b269dbbdae0d6b68d57e737270e98ebc6455716858cecf524fdd1f p:8f5bab218b6bb34476f51ca588e9f4553a3a7ce5e13a66c660a5283e97e9a85a $ 1
2 14db95de34b269dbbdae0d6b68d57e737270e98ebc6455716858cecf524fdd1f p:8f5bab218b6bb34476f51ca588e9f4553a3a7ce5e13a66c660a5283e97e9a85a 2
2 14db95de34b269dbbdae0d6b68d57e737270e98ebc6455716858cecf524fdd1f p:8f5bab218b6bb34476f51ca588e9f4553a3a7ce5e13a66c660a5283e97e9a85a 3

So these are the first two blocks, first hash is the wrongly calculated hash, second hash is the (correct) parent hash.

Root cause is likely that the Common instance is not set on all block or header creation occurrences in the client, specifically on receiving block headers in the ETH and LES protocol implementations. Since the clique hash calculation is deviating from the ethash block hash calculation (for clique the seal in extraData is removed from hashing) the hash will be wrong.

Further investigation, can't test so well yet because my Goerli connections are still not that good (my relatively bad internet connection might further contribute to this).

[holgerd77]

Note: I first experimented with passing config to all message DECODE functions to allow for the correct BlockHeader instantiaton on:

decode: (headers: BlockHeaderBuffer[]) =>
      headers.map((h) => BlockHeader.fromValuesArray(h, {})),
  }

But it is likely a lot better/cleaner to just remove this decode function and do the decoding on the 1-2 call occurrences where config is just directly present.

[holgerd77]

I've now added the common instances (locally, will push soon) to the header creation occurrences from above. So hash creation is now taking the clique hash creation version of the BlockHeader.hash() function, this still produces the (same) wrong hash though.

Here is some debug data by adding log.console() messages to BlockFetcher.request(). This is where things currently go wrong since the block bodies are subsequently requested with the wrong hashes which obviously goes wrong.

Here is the raw data from header number 1 retrieved:

[
  <Buffer bf 7e 33 1f 7f 7c 1d d2 e0 51 59 66 6b 3b f8 bc 7a 8a 3a 9e b1 d5 18 96 9e ab 52 9d d9 b8 8c 1a>,
  <Buffer 1d cc 4d e8 de c7 5d 7a ab 85 b5 67 b6 cc d4 1a d3 12 45 1b 94 8a 74 13 f0 a1 42 fd 40 d4 93 47>,
  <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>,
  <Buffer 5d 6c de d5 85 e7 3c 4e 32 2c 30 c2 f7 82 a3 36 31 6f 17 dd 85 a4 86 3b 9d 83 8d 2d 4b 8b 30 08>,
  <Buffer 56 e8 1f 17 1b cc 55 a6 ff 83 45 e6 92 c0 f8 6e 5b 48 e0 1b 99 6c ad c0 01 62 2f b5 e3 63 b4 21>,
  <Buffer 56 e8 1f 17 1b cc 55 a6 ff 83 45 e6 92 c0 f8 6e 5b 48 e0 1b 99 6c ad c0 01 62 2f b5 e3 63 b4 21>,
  <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 206 more bytes>,
  <Buffer 02>,
  <Buffer 01>,
  <Buffer 9f d8 01>,
  <Buffer >,
  <Buffer 5c 53 0f fd>,
  <Buffer 50 61 72 69 74 79 20 54 65 63 68 20 41 75 74 68 6f 72 69 74 79 00 00 00 00 00 00 00 00 00 00 00 2b bf 88 61 81 97 06 54 ed 46 e3 fa e0 de d4 1e e5 3f ... 47 more bytes>,
  <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>,
  <Buffer 00 00 00 00 00 00 00 00>
]

Hash is still: <Buffer e2 6b a5 8f 79 23 69 36 93 f3 b6 27 9b 53 bb 29 e1 7d 6c 7d 17 79 bf 2c 79 3c 14 c9 69 ab f6 60>`` Should be: 0x8f5bab218b6bb34476f51ca588e9f4553a3a7ce5e13a66c660a5283e97e9a85a`

And the created Header object:

BlockHeader {
  _common: Common {
    _supportedHardforks: [],
    _eips: [],
    _customChains: [],
    _chainParams: {
      name: 'goerli',
      chainId: 5,
      networkId: 5,
      defaultHardfork: 'istanbul',
      consensus: [Object],
      comment: 'Cross-client PoA test network',
      url: 'https://github.com/goerli/testnet',
      genesis: [Object],
      hardforks: [Array],
      bootstrapNodes: [Array],
      dnsNetworks: [Array]
    },
    DEFAULT_HARDFORK: 'istanbul',
    _hardfork: 'petersburg'
  },
  parentHash: <Buffer bf 7e 33 1f 7f 7c 1d d2 e0 51 59 66 6b 3b f8 bc 7a 8a 3a 9e b1 d5 18 96 9e ab 52 9d d9 b8 8c 1a>,
  uncleHash: <Buffer 1d cc 4d e8 de c7 5d 7a ab 85 b5 67 b6 cc d4 1a d3 12 45 1b 94 8a 74 13 f0 a1 42 fd 40 d4 93 47>,
  coinbase: Address {
    buf: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>
  },
  stateRoot: <Buffer 5d 6c de d5 85 e7 3c 4e 32 2c 30 c2 f7 82 a3 36 31 6f 17 dd 85 a4 86 3b 9d 83 8d 2d 4b 8b 30 08>,
  transactionsTrie: <Buffer 56 e8 1f 17 1b cc 55 a6 ff 83 45 e6 92 c0 f8 6e 5b 48 e0 1b 99 6c ad c0 01 62 2f b5 e3 63 b4 21>,
  receiptTrie: <Buffer 56 e8 1f 17 1b cc 55 a6 ff 83 45 e6 92 c0 f8 6e 5b 48 e0 1b 99 6c ad c0 01 62 2f b5 e3 63 b4 21>,
  bloom: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 206 more bytes>,
  difficulty: <BN: 2>,
  number: <BN: 1>,
  gasLimit: <BN: 9fd801>,
  gasUsed: <BN: 0>,
  timestamp: <BN: 5c530ffd>,
  extraData: <Buffer 50 61 72 69 74 79 20 54 65 63 68 20 41 75 74 68 6f 72 69 74 79 00 00 00 00 00 00 00 00 00 00 00 2b bf 88 61 81 97 06 54 ed 46 e3 fa e0 de d4 1e e5 3f ... 47 more bytes>,
  mixHash: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>,
  nonce: <Buffer 00 00 00 00 00 00 00 00>
}

[holgerd77]

The Block library actually has not a single dedicated direct test for the hash() function, which is a bit embarrassing. Will add for PoW and PoA.

[holgerd77]

Got it, the hashing was indeed wrong. We misread the spec here, the hash without the extraData seal is just meant for signature creation, the final block hash is just the normal hash created with the complete extraData field (got a confirmation on this on the geth channel). Will push soon.

[holgerd77]

Fixed by #1088