fix: add more historic support for staking-payouts

[TarikGul]

Issue

We are unable to retrieve the payouts from era 0 until 518 in Kusama.
Example endpoint :

http://127.0.0.1:8080/accounts/FXCgfz7AzQA1fNaUqubSgXxGh77sjWVVkypgueWLmAcwv79/staking-payouts?unclaimedOnly=false&at=1000000&depth=21

returns :

{
  "code": 500,
  "message": "apiAt.query.staking.activeEra is not a function",
  "stack": "TypeError: apiAt.query.staking.activeEra is not a function\n    at AccountsStakingPayoutsController ...",
  "level": "error"
}

Root Cause

The available calls before era 518 were different or nonexistent compared to those after era 518. So we need to :

• Find the calls and events that were in place in those runtimes (regarding payouts, activeEra, etc)
• Find the logic behind the payout process during those runtimes.
• Add the calls and logic for those eras

Useful Resource

MIgration Commit that shows the replacement of some of the calls.

Suggested Solution

For the first 517 eras :

• Era Points & Rewards
  • To obtain the points for an era, we reference the end block of the previous era.
  • To obtain the rewards for an era, we reference the start block of the current era.
  • An exception to this rule occurs with era 0, as there is no previous era. To address this, we have simply decided that any queries for blocks before the end of era 0 will yield the same results as queries for blocks of era 1. This means that the query will return the full payouts for era 0, even if the queried block precedes the end of era 0.
• activeEra was not available so we changed the logic on how we calculate it taking into account the SessionIndex and currentEra. Again, an exception case for era 0 was added in this check.
• historyDepth was not available in that runtime so we disabled the depthquery parameter for those eras.
• Implemented a registry for the initial 518 eras (kusamaEarlyErasBlockInfo.json) to track the start and end block of each era.
• Changed the logic on how we retrieve payouts per Validator / Nominator.

Important Note

• era = 517 has no rewards, we believe it's because that's were the runtime change happened and a lot of the staking logic changed.

Sample Test cases

Below are example test cases that refer to eras preceding 518 and now return payout results after implementing the suggested solution:

Example Request 1

/accounts/FXCgfz7AzQA1fNaUqubSgXxGh77sjWVVkypgueWLmAcwv79/staking-payouts?unclaimedOnly=false&at=1341083

We are querying block = 1341083 which according to our registry (kusamaEarlyErasBlockInfo.json) is the start of era 508. Since Sidecar returns by default results with depth era - 1, we expect to see results for era 507.

Result for Request 1

{
  "at": {
    "height": "1341083",
    "hash": "0xaf4f1172117c1a752b24374cc4c9e54af3b1ba996f14c5d029b1ba66eba59ce3"
  },
  "erasPayouts": [
    {
      "era": "507",
      "totalEraRewardPoints": "70660",
      "totalEraPayout": "415081276367227",
      "payouts": [
        {
          "validatorId": "FXCgfz7AzQA1fNaUqubSgXxGh77sjWVVkypgueWLmAcwv79",
          "nominatorStakingPayout": "2114764388461",
          "claimed": true,
          "totalValidatorRewardPoints": "360",
          "validatorCommission": "1000000000",
          "totalValidatorExposure": "29077863011956191",
          "nominatorExposure": "475769231000"
        }
      ]
    }
  ]
}

We can verify this outcome (totalEraPayout = 415081276367227) by comparing it with the result obtained from Subscan event 1341083-2

Example Request 2

/accounts/DSELWxqt4g3THNfg7TAHhU69eQJo8N2kBbAMuaF636eZCX2/staking-payouts?unclaimedOnly=false&at=1057267

We are querying block = 1057267 which according to our registry is the start of era 427. Since Sidecar returns by default results with depth era - 1, we expect to see results for era 426.

Result for Request 2

{
  "at": {
    "height": "1057267",
    "hash": "0xabed9ad49605c7b8c1b933c056bb6dc573b9f02b70b3742eb8f2441af8dc4107"
  },
  "erasPayouts": [
    {
      "era": "426",
      "totalEraRewardPoints": "65880",
      "totalEraPayout": "349336487782285",
      "payouts": [
        {
          "validatorId": "GC8hwHbQ4TdbYJJPDS96G7Uj9bivnW5z56UEkqujjwhQPp5",
          "nominatorStakingPayout": "1378775727261",
          "claimed": true,
          "totalValidatorRewardPoints": "300",
          "validatorCommission": "100000000",
          "totalValidatorExposure": "56175710896906817",
          "nominatorExposure": "54098857016412154"
        },
        {
          "validatorId": "EicrAEbyauqktQpp4CdvsF2CQy3Ju7tGGMohj3h5sAPnKHL",
          "nominatorStakingPayout": "1559649193912",
          "claimed": true,
          "totalValidatorRewardPoints": "340",
          "validatorCommission": "100000000",
          "totalValidatorExposure": "55572949938794175",
          "nominatorExposure": "53416895181965965"
        },
        {
          "validatorId": "HqGhgHg6YvnhaXSnaAUvyTDiR4FirB6Ssh2XNDedTzwCDv2",
          "nominatorStakingPayout": "1961892446690",
          "claimed": true,
          "totalValidatorRewardPoints": "420",
          "validatorCommission": "100000000",
          "totalValidatorExposure": "58550281938137071",
          "nominatorExposure": "57308900193366180"
        }
      ]
    }
  ]
}

We can verify this outcome (totalEraPayout = 349336487782285) by comparing it with the result obtained from Polkadot JS Apps block 1057267 - Reward event

Credits & Thanks

• to @ruiparitydata for bringing this to our attention and providing useful examples and test cases so we can test that the suggested implementation works correctly. Also, thank you for testing our implementation afterward.

[TarikGul]

Calls that I found need to be replaced that the runtimes before era 518 do not have:

apiAt.query.staking:

historyDepth
erasRewardPoints
erasValidatorRewards
erasValidatorPrefs
erasStakersClipped

[TarikGul]

Just want to add a note: Credit to this PR goes completely to @bee344 and @Imod7! Thank you both for tackling this necessary addition.

[marshacb]

commented line

[marshacb]

Really great work on this! LGTM

[TarikGul]

@bee344 @Imod7 👍 From me :)

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/parity-tech-update-for-february/6630/1