Different data returned from debug_traceTransaction/callTracer in Goerli Archive v1.11.1 and v1.10.26

[martinboehm]

System information

Goerli Archive
Geth/v1.11.1-stable-76961066/linux-amd64/go1.20.1 vs Geth/v1.10.26-stable-e5eb32ac/linux-amd64/go1.18.5
Prysm/v3.2.0 (linux amd64)

Description

The call to debug_traceTransaction/callTracer (and debug_traceBlockByHash) returns for some transactions different data in the version v1.11.1 and in the version v1.10.26.

Steps to reproduce the behaviour

Call the tracer, for example curl address:port -X POST --header 'Content-type: application/json' --data '{"jsonrpc":"2.0", "method":"debug_traceTransaction", "params":["0xc0ffcf21dc1881c3f20160b530d76f1d37af7b40552f7c59d3f339bad3023dad", {"tracer": "callTracer"}], "id":1}'.

geth v1.11.1 result:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "from": "0x790ba84d8d1d604f617c0def35e339422ba9a026",
    "gas": "0x2558d",
    "gasUsed": "0x20656",
    "to": "0xe2e52c2d0d64209b8dd1854371a4c673c13448f0",
    "input": "0x0871e97100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000790ba84d8d1d604f617c0def35e339422ba9a026000000000000000000000000000000000000000000000000002386f26fc1000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000",
    "calls": [
      {
        "from": "0xe2e52c2d0d64209b8dd1854371a4c673c13448f0",
        "gas": "0x23020",
        "gasUsed": "0x1931b",
        "to": "0xd08fa8fa2056997bd2ca3ba689c63ddf54f3874f",
        "input": "0x0871e97100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000790ba84d8d1d604f617c0def35e339422ba9a026000000000000000000000000000000000000000000000000002386f26fc1000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000",
        "calls": [
          {
            "from": "0xe2e52c2d0d64209b8dd1854371a4c673c13448f0",
            "gas": "0x14256",
            "gasUsed": "0xaa1f",
            "to": "0x715fc3a1422de7ae2ea483d1da8415e8dd45bad3",
            "input": "0x33d6247d869c7a086d511ffd79d102ba1343faf071bdda48120d2704c53d88a8d81bf748",
            "calls": [
              {
                "from": "0x715fc3a1422de7ae2ea483d1da8415e8dd45bad3",
                "gas": "0x12151",
                "gasUsed": "0x8d8b",
                "to": "0x11dec2c21999c5283db6aa794359fe0a94207dac",
                "input": "0x33d6247d869c7a086d511ffd79d102ba1343faf071bdda48120d2704c53d88a8d81bf748",
                "value": "0x0",
                "type": "DELEGATECALL"
              }
            ],
            "value": "0x0",
            "type": "CALL"
          }
        ],
        "value": "0x2386f26fc10000",
        "type": "DELEGATECALL"
      }
    ],
    "value": "0x2386f26fc10000",
    "type": "CALL"
  }
}

geth v1.10.26 result:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "type": "CALL",
    "from": "0x790ba84d8d1d604f617c0def35e339422ba9a026",
    "to": "0xe2e52c2d0d64209b8dd1854371a4c673c13448f0",
    "value": "0x2386f26fc10000",
    "gas": "0x2558d",
    "gasUsed": "0x1afca",
    "input": "0x0871e97100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000790ba84d8d1d604f617c0def35e339422ba9a026000000000000000000000000000000000000000000000000002386f26fc1000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000",
    "output": "0x",
    "calls": [
      {
        "type": "DELEGATECALL",
        "from": "0xe2e52c2d0d64209b8dd1854371a4c673c13448f0",
        "to": "0xd08fa8fa2056997bd2ca3ba689c63ddf54f3874f",
        "gas": "0x23020",
        "gasUsed": "0x1931b",
        "input": "0x0871e97100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000790ba84d8d1d604f617c0def35e339422ba9a026000000000000000000000000000000000000000000000000002386f26fc1000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000",
        "output": "0x",
        "calls": [
          {
            "type": "CALL",
            "from": "0xe2e52c2d0d64209b8dd1854371a4c673c13448f0",
            "to": "0x715fc3a1422de7ae2ea483d1da8415e8dd45bad3",
            "value": "0x0",
            "gas": "0x14256",
            "gasUsed": "0xaa1f",
            "input": "0x33d6247d869c7a086d511ffd79d102ba1343faf071bdda48120d2704c53d88a8d81bf748",
            "output": "0x",
            "calls": [
              {
                "type": "DELEGATECALL",
                "from": "0x715fc3a1422de7ae2ea483d1da8415e8dd45bad3",
                "to": "0x11dec2c21999c5283db6aa794359fe0a94207dac",
                "gas": "0x12151",
                "gasUsed": "0x8d8b",
                "input": "0x33d6247d869c7a086d511ffd79d102ba1343faf071bdda48120d2704c53d88a8d81bf748",
                "output": "0x"
              }
            ]
          }
        ]
      }
    ]
  }
}

Apart from reordering of the json fields (which is OK) the difference is that the DELEGATECALL on the first level has value 0x2386f26fc10000 in the version v1.11.1 and no value in the version v1.10.26. The top gasUsed value also does not match.

[PPianAIC]

@martinboehm Can you tell me how your geth node runs, because there is a problem with debug_traceTransaction, it always has a error. Thanks!

example:

debug.traceTransaction("0xf978486bed72890591704ce0fe9d802d88b8f74a576b0e3593acfea89151b337", {"tracer": "callTracer"})
Error: required historical state unavailable (reexec=128)
at web3.js:6365:9(45)
at send (web3.js:5099:62(34))
at :1:23(7)

System information

Geth
Version: 1.10.26-stable
Architecture: arm64
Go Version: go1.18.9
Operating System: linux
GOPATH=
GOROOT=go

Command

./geth --datadir /data1/ethereum/ethereum/.ethereum --http --http.port 8454 --http.addr "0.0.0.0" --http.api eth,net,engine,admin,debug,txpool,web3,trace --ws --ws.addr "0.0.0.0" --ws.port 8456 --ws.origins "" --http.corsdomain "" --rpc.allow-unprotected-txs --txlookuplimit 0 --allow-insecure-unlock --maxpeers 1000 --gcmode archive --cache 56000 --authrpc.addr "0.0.0.0" --authrpc.port 8551 --authrpc.vhosts localhost --authrpc.jwtsecret /data1/ethereum/ethereum/consensus/prysm/jwt.hex

[martinboehm]

@PPianAIC Hi, we are running archive settings of geth --gcmode archive like you do, actually our parameters look very similar to yours.
However, you are trying to trace the transaction 0xf978486bed72890591704ce0fe9d802d88b8f74a576b0e3593acfea89151b337, which does not exist in Goerli.

My transaction was 0xc0ffcf21dc1881c3f20160b530d76f1d37af7b40552f7c59d3f339bad3023dad or you can try more recent 0xa03b913c355eb2136e8f3f63705eabfe055977402bf0b43483bebd6be5442922. There are actually many transactions where we observe the issue.

[martinboehm]

@PPianAIC There are multiple people with the same issue #26742

[holiman]

• Regarding the value on DELEGATECALL, that is beacuse of core/vm: set tracer-observable value of a delegatecall to match parent value #26632 . It is an intentional change. It may seem odd, since delegatecall does not transfer any value, however: the evm operation CALLVALUE inside a delegatecall scope does surface it's parent value. So in other words: the value that the tracer sees on a delegatecall matches what the evm sees.
• Regarding the difference in gasUsed

>>> 0x20656-110538 
22156

The difference is 21k + 1156 gas, so it very much looks like it's the intrinsic gas for the transaction plus the input data. This changes comes from #26048

Likewise the gasUsed field of the top call frame of the callTracer will change. The value did not previously account for the intrinsic gas (e.g. 21000 tx cost) and gas refunds.

[martinboehm]

@holiman Thank you for your answer! Just to make it clear to me. When tracking internal transactions (I mean real transfers that change the balance of an address), we should always ignore the DELEGATECALL entries in the trace. We were ignoring them implicitly with the old tracer because they had no value and now we should ignore them explicitly. Is this correct?

[holiman]

If you are tracking the way ether moves across accounts, then yes, you should ignore DELEGATECALL. It can never transfer ether, and it doesn't even change "what address am I at".