v2.1: [rpc] Fatal getSignaturesForAddress() when Bigtable errors (backport of #3700)

[mergify]

Problem

Consider a request to getSignaturesForAddress(). Imagine that there are no signatures in blockstore, but there are signatures in long-term storage (ie. Bigtable).

Currently, if we fail to reach Bigtable – because of a timeout or a connection failure – we return whatever signatures we have. This means that people who query the RPC can't distinguish between the following cases:

• No signatures were found because there are none, or
• No signatures were found because Bigtable was temporarily unavailable

Summary of Changes

1. When doing a range query in get_confirmed_signatures_for_address and the before/until can't be found, throw a SignatureNotFound error instead of RowNotFound.
2. Now that we can match on actual connection errors – separate from SignatureNotFound errors – return a JSON-RPC error in the event that long-term storage errors out.

Test Plan

With the Bigtable emulator

> gcloud beta emulators bigtable start
> ./init-bigtable.sh

> tail -f validator.log | grep [Bb]ig[Tt]able
[2024-11-19T23:06:05.994676924Z INFO  solana_rpc::rpc_service] rpc configuration: JsonRpcConfig { enable_rpc_transaction_history: true, enable_extended_tx_metadata_storage: true, faucet_addr: Some(0.0.0.0:9900), health_check_slot_distance: 0, skip_preflight_health_check: false, rpc_bigtable_config: Some(RpcBigtableConfig { enable_bigtable_ledger_upload: false, bigtable_instance_name: "solana-ledger", bigtable_app_profile_id: "default", timeout: None, max_message_size: 67108864 }), max_multiple_accounts: None, account_indexes: AccountSecondaryIndexes { keys: None, indexes: {} }, rpc_threads: 0, rpc_niceness_adj: 0, full_api: true, rpc_scan_and_fix_roots: false, max_request_body_size: None, disable_health_check: true }
[2024-11-19T23:06:05.995145895Z INFO  solana_storage_bigtable::bigtable] Connecting to bigtable emulator at localhost:8086
[2024-11-19T23:06:05.995411312Z INFO  solana_rpc::rpc_service] BigTable ledger storage initialized

Fetching a signature for an address that does not exist locally

Observe that the RPC goes out to Bigtable and fetches successfully (finds nothing)

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "BzszvNFvokG4v4qb1ipYSDGtKuhen8PMtYjevfZeAkKZ"
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[],"id":"1"}

Fetching a signature for an address and before signature that don't exist locally

Observe that Bigtable can't find the before signature, but doesn't fatal the request

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "BzszvNFvokG4v4qb1ipYSDGtKuhen8PMtYjevfZeAkKZ",
    {"before":"31AUDAUXgD4B5DqqtFZZe8udgHUfbJX9dQbiXQGmNiRqeKRkzRTYitrRbJDtpt4DMd4P3G8haaXMJ8TU6wwQmf3h"}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[],"id":"1"}

Fetching a signature for an address that is completely available locally

Observe Bigtable is never contacted.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {"limit": 1}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[{"blockTime":1732058393,"confirmationStatus":"finalized","err":null,"memo":null,"signature":"3mZUQjhaGA7ytwb3g9PnjYL4d1ErsWf7nheMCqnSuwW6x2hF8tx4DuuvG1u7j6qVCC2J9JqpRHbJ7z57ayhJZyW2","slot":9284}],"id":"1"}

Fetching a signature for an address that does exist but with a bad before signature

Observe that Bigtable can't find the before signature, but doesn't fatal the request

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {"before":"419mQJ2ZpM1uJ2VJyHckWNzmEpRKkyuirvRQH9NfgKvzQJxTcTa5v8U7tXJ4VkqEH8GsTtG7iqSsQBJe8VxxC2XQ","limit": 1}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[],"id":"1"}

Fetching a signature before the last signature, forcing the RPC to go to Bigtable where there is no data

Observe that the RPC goes out to Bigtable and fetches successfully (finds nothing)

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {
      "before":"45z1V6qdGUXRHid4YDzSSqV13pegyP2naBaA3aG4ZoS4xMdKjC3eBvhwTkbXrqEXJT38grfRJ11wCTn5Qop67AEv",
      "limit": 1
    }
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[],"id":"1"}

With the Bigtable emulator shut down, simulating a connection failure

Fetching a signature for an address that does not exist locally

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "BzszvNFvokG4v4qb1ipYSDGtKuhen8PMtYjevfZeAkKZ"
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","error":{"code":-32019,"message":"Failed to query long-term storage; please try again"},"id":"1"}

Fetching a signature for an address and before signature that don't exist locally

Observe that Bigtable experiences a connection error.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "BzszvNFvokG4v4qb1ipYSDGtKuhen8PMtYjevfZeAkKZ",
    {"before":"31AUDAUXgD4B5DqqtFZZe8udgHUfbJX9dQbiXQGmNiRqeKRkzRTYitrRbJDtpt4DMd4P3G8haaXMJ8TU6wwQmf3h"}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","error":{"code":-32019,"message":"Failed to query long-term storage; please try again"},"id":"1"}

Fetching a signature for an address that is completely available locally

Observe Bigtable is never contacted.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {"limit": 1}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":{"jsonrpc":"2.0","result":[{"blockTime":1732058393,"confirmationStatus":"finalized","err":null,"memo":null,"signature":"3mZUQjhaGA7ytwb3g9PnjYL4d1ErsWf7nheMCqnSuwW6x2hF8tx4DuuvG1u7j6qVCC2J9JqpRHbJ7z57ayhJZyW2","slot":9284}],"id":"1"}

Fetching a signature for an address that does exist but with a bad before signature

Observe that Bigtable experiences a connection error.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {"before":"419mQJ2ZpM1uJ2VJyHckWNzmEpRKkyuirvRQH9NfgKvzQJxTcTa5v8U7tXJ4VkqEH8GsTtG7iqSsQBJe8VxxC2XQ","limit": 1}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","error":{"code":-32019,"message":"Failed to query long-term storage; please try again"},"id":"1"}

Fetching a signature before the last signature, forcing the RPC to go to Bigtable where there is no data

Observe that Bigtable experiences a connection error.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {
      "before":"45z1V6qdGUXRHid4YDzSSqV13pegyP2naBaA3aG4ZoS4xMdKjC3eBvhwTkbXrqEXJT38grfRJ11wCTn5Qop67AEv",
      "limit": 1
    }
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","error":{"code":-32019,"message":"Failed to query long-term storage; please try again"},"id":"1"}

Fixes #3696

---

This is an automatic backport of pull request #3700 done by Mergify.

[mergify]

If this PR represents a change to the public RPC API:

1. Make sure it includes a complementary update to rpc-client/ (example)
2. Open a follow-up PR to update the JavaScript client @solana/web3.js (example)

Thank you for keeping the RPC clients in sync with the server API @mergify[bot].

[steveluscher]

Backporting this will allow Quicknode to remove the hack they've put in place to protect against the non-deterministic nature of the RPC response in question here.

[t-nelson]

@KirillLykov are you ok with sme approval of this backport?

[KirillLykov]

as for 2.0