Broken API endpoint /eth/v1/beacon/light_client/updates

[zsfelfoldi]

The updates endpoint seems to be broken:
http://104.248.139.137:9596/eth/v1/beacon/light_client/updates?start_period=10&count=1 returns {"statusCode":500,"error":"Internal Server Error","message":"finalizedBlockRoot not available 0xc23381411ab2aee8a7aa9dc63ebe7faac3220fe53582c03051e5879fb90ecfec"}
(the node on the given address runs Sepolia but I have tried it with another node running Mainnet too and it returns the same error for any period between the Altair fork and the latest period)

[dadepo]

The endpoint returns successful using a beacon node as shown below... using any period from 290 till 581 (the current period)

➜  ~ curl https://lodestar-mainnet.chainsafe.io/eth/v1/beacon/light_client/updates\?start_period\=295\&count\=1 | jq | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 53733  100 53733    0     0  69565      0 --:--:-- --:--:-- --:--:-- 70239
{
  "data": [
    {
      "attested_header": {
        "slot": "2423677",
        "proposer_index": "59881",
        "parent_root": "0x1ad8a96cd5070323f447e4f8f3b70175086eba48518064e46f5188a16f4d35c8",
        "state_root": "0x7a788d08dd8bdc2319f4b751e97690818ed6fe4ea8696cfafca4c3b847d8b223",
        "body_root": "0x62fa4b3dda98a7aa9ae1a9f1e80e7baa4da2e385b69286ef7a78c9f817a29553"
      },
➜  ~ curl https://lodestar-mainnet.chainsafe.io/eth/v1/beacon/light_client/updates\?start_period\=581\&count\=1 | jq | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 53735  100 53735    0     0  72528      0 --:--:-- --:--:-- --:--:-- 72811
{
  "data": [
    {
      "attested_header": {
        "slot": "4759621",
        "proposer_index": "284232",
        "parent_root": "0x0f27f9142da98544be1bbebfea93b1c76300ecfc8df564b9b1ca374d23c9a8eb",
        "state_root": "0xcacc8b81ba825aefb22bf32054e668ffa6a38865eab4edfa6a0488207e30c86d",
        "body_root": "0x93ad52438f0e512c983546ad23554cb34e46f9973e0d54e10c70051c27b58595"
      },

@zsfelfoldi Can you confirm from your end? I also noticed the url you shared has the period has 10, which is wrong...can you confirm also with a correct period. Also if the issue persist, when you start the node before you make the request, do you observe any errors in the logs you can share that can help debug the issue?

[zsfelfoldi]

@dadepo as I wrote my node is running Sepolia on which period 10 should be valid. It looks to me like the updates generated since the node was started are returned fine but if I stop and restart the node it returns the error for those periods too (in case of my Sepolia node periods 81 to 85 are working because it was last started before generating update 81). So there is something that works while in memory but the db store and/or retrieval is wrong.
(I would prefer to not restart that node again because now at least I can use it for testing my syncing code but I also have a mainnet node on my local machine that I restarted multiple times and confirmed the phenomenon described above)

[dadepo]

Hi @zsfelfoldi
Thanks for the clarification. This looks like known issues regarding lodestar regen's capabilities and also this issue noted

• Sync committee witness becomes unavailable after a while #4183

I'll update you once a fix is available.

[zsfelfoldi]

@dadepo I think I'm one step closer to the issue now. When I tried to verify those updates that are returned (that were generated since the node startup) I realized that they are wrong too, at least when they have a finalized header. In this case the finalized header's block root should be proven by the finality branch starting from the main header's state root (this part is correct) and the next sync committee root should be proven by the next sync committee branch, starting from the finalized header's state root. This latter part is not correct, now this proof is also rooted in the main header's state root (making the finalized header pointless). This was also correct earlier so maybe this is where something got broken and the two issues might be related.

[zsfelfoldi]

Looks like this issue has been solved by #4702
Also the issue mentioned in the last comment about the update format is apparently not a bug but a feature: ethereum/consensus-specs#2932