Add cursor support to `/accounts/<addr>/resources` #5313

banool · 2022-10-26T22:04:49Z

Description

This PR makes /accounts/<addr>/resources and /accounts/<addr>/modules return the cursor in the header. This is the easiest way to mitigate this issue quickly in a backwards compatible way.

Remaining work:

~~Add checking in the API layer for the limit value, capped at no higher than 9999 (though we should set it lower than that).~~
~~Add tests for this~~, including the case where an account has > 10k resources (I'll do this in a separate PR).
Use this in the SDK (I'll do this in a separate PR).

Test Plan: Resources

Run a local testnet:

cargo run -p aptos -- node run-local-testnet --force-restart --assume-yes --with-faucet

Get account resources with no limit:

$ curl -s 'http://127.0.0.1:8080/v1/accounts/0x1/resources' | jq '. | length'
31

Get account resources with a limit:

$ curl -s 'http://127.0.0.1:8080/v1/accounts/0x1/resources?limit=10' | jq '. | length'
10

Set limit to zero:

$ curl -s 'http://127.0.0.1:8080/v1/accounts/0x1/resources?limit=0' | jq '. | length'
0

Check with limit > number of resources:

$ curl -s 'http://127.0.0.1:8080/v1/accounts/0x1/resources?limit=100000' | jq '. | length'
31

Paginate with the cursor:

$ curl -v 'http://127.0.0.1:8080/v1/accounts/0x1/resources?limit=8' | jq .[].type
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/accounts/0x1/resources?limit=8 HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< x-aptos-chain-id: 4
< x-aptos-ledger-version: 334
< x-aptos-ledger-oldest-version: 0
< x-aptos-ledger-timestampusec: 1667154971130969
< x-aptos-epoch: 2
< x-aptos-block-height: 166
< x-aptos-oldest-block-height: 0
< x-aptos-cursor: 000000000000000000000000000000000000000000000000000000000000000001390100000000000000000000000000000000000000000000000000000000000000010b73746f726167655f6761730a53746f7261676547617300
< content-length: 352064
< date: Sun, 30 Oct 2022 18:36:15 GMT
<
{ [64792 bytes data]
100  343k  100  343k    0     0  2950k      0 --:--:-- --:--:-- --:--:-- 3069k
* Connection #0 to host 127.0.0.1 left intact
"0x1::account::Account"
"0x1::version::Version"
"0x1::chain_id::ChainId"
"0x1::coin::SupplyConfig"
"0x1::features::Features"
"0x1::stake::ValidatorSet"
"0x1::block::BlockResource"
"0x1::code::PackageRegistry"

$ curl -v 'http://127.0.0.1:8080/v1/accounts/0x1/resources?limit=8&start=000000000000000000000000000000000000000000000000000000000000000001390100000000000000000000000000000000000000000000000000000000000000010b73746f726167655f6761730a53746f7261676547617300' | jq .[].type
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/accounts/0x1/resources?limit=8&start=000000000000000000000000000000000000000000000000000000000000000001390100000000000000000000000000000000000000000000000000000000000000010b73746f726167655f6761730a53746f7261676547617300 HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< x-aptos-chain-id: 4
< x-aptos-ledger-version: 487
< x-aptos-ledger-oldest-version: 0
< x-aptos-ledger-timestampusec: 1667155000175336
< x-aptos-epoch: 3
< x-aptos-block-height: 243
< x-aptos-oldest-block-height: 0
< x-aptos-cursor: 000000000000000000000000000000000000000000000000000000000000000001400100000000000000000000000000000000000000000000000000000000000000010c636861696e5f7374617475731047656e65736973456e644d61726b657200
< content-length: 14471
< date: Sun, 30 Oct 2022 18:36:45 GMT
<
{ [14471 bytes data]
100 14471  100 14471    0     0   470k      0 --:--:-- --:--:-- --:--:--  565k
* Connection #0 to host 127.0.0.1 left intact
"0x1::storage_gas::StorageGas"
"0x1::stake::ValidatorPerformance"
"0x1::account::OriginatingAddress"
"0x1::gas_schedule::GasScheduleV2"
"0x1::stake::AptosCoinCapabilities"
"0x1::version::SetVersionCapability"
"0x1::storage_gas::StorageGasConfig"
"0x1::staking_config::StakingConfig"

$ curl -v 'http://127.0.0.1:8080/v1/accounts/0x1/resources?limit=8&start=000000000000000000000000000000000000000000000000000000000000000001400100000000000000000000000000000000000000000000000000000000000000010c636861696e5f7374617475731047656e65736973456e644d61726b657200' | jq .[].type
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/accounts/0x1/resources?limit=8&start=000000000000000000000000000000000000000000000000000000000000000001400100000000000000000000000000000000000000000000000000000000000000010c636861696e5f7374617475731047656e65736973456e644d61726b657200 HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< x-aptos-chain-id: 4
< x-aptos-ledger-version: 583
< x-aptos-ledger-oldest-version: 0
< x-aptos-ledger-timestampusec: 1667155017557346
< x-aptos-epoch: 3
< x-aptos-block-height: 291
< x-aptos-oldest-block-height: 0
< x-aptos-cursor: 000000000000000000000000000000000000000000000000000000000000000001470100000000000000000000000000000000000000000000000000000000000000011261676772656761746f725f666163746f72791141676772656761746f72466163746f727900
< content-length: 1305
< date: Sun, 30 Oct 2022 18:37:02 GMT
<
{ [1305 bytes data]
100  1305  100  1305    0     0   102k      0 --:--:-- --:--:-- --:--:--  159k
* Connection #0 to host 127.0.0.1 left intact
"0x1::chain_status::GenesisEndMarker"
"0x1::reconfiguration::Configuration"
"0x1::aptos_governance::VotingRecords"
"0x1::state_storage::StateStorageUsage"
"0x1::consensus_config::ConsensusConfig"
"0x1::timestamp::CurrentTimeMicroseconds"
"0x1::aptos_governance::GovernanceConfig"
"0x1::aptos_governance::GovernanceEvents"

$ curl -v 'http://127.0.0.1:8080/v1/accounts/0x1/resources?limit=8&start=000000000000000000000000000000000000000000000000000000000000000001470100000000000000000000000000000000000000000000000000000000000000011261676772656761746f725f666163746f72791141676772656761746f72466163746f727900' | jq .[].type
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/accounts/0x1/resources?limit=8&start=000000000000000000000000000000000000000000000000000000000000000001470100000000000000000000000000000000000000000000000000000000000000011261676772656761746f725f666163746f72791141676772656761746f72466163746f727900 HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< x-aptos-chain-id: 4
< x-aptos-ledger-version: 663
< x-aptos-ledger-oldest-version: 0
< x-aptos-ledger-timestampusec: 1667155032095626
< x-aptos-epoch: 3
< x-aptos-block-height: 331
< x-aptos-oldest-block-height: 0
< content-length: 2183
< date: Sun, 30 Oct 2022 18:37:17 GMT
<
{ [2183 bytes data]
100  2183  100  2183    0     0   110k      0 --:--:-- --:--:-- --:--:--  142k
* Connection #0 to host 127.0.0.1 left intact
"0x1::aggregator_factory::AggregatorFactory"
"0x1::transaction_fee::AptosCoinCapabilities"
"0x1::aptos_governance::ApprovedExecutionHashes"
"0x1::aptos_governance::GovernanceResponsbility"
"0x1::transaction_validation::TransactionValidation"
"0x1::coin::CoinInfo<0x1::aptos_coin::AptosCoin>"
"0x1::voting::VotingForum<0x1::governance_proposal::GovernanceProposal>"

Test Plan: Modules

Paginate with the cursor:

$ curl -v 'http://127.0.0.1:8080/v1/accounts/0x1/modules?limit=30' | jq .[].abi.name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/accounts/0x1/modules?limit=30 HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< x-aptos-chain-id: 4
< x-aptos-ledger-version: 172
< x-aptos-ledger-oldest-version: 0
< x-aptos-ledger-timestampusec: 1667155219929518
< x-aptos-epoch: 2
< x-aptos-block-height: 85
< x-aptos-oldest-block-height: 0
< x-aptos-cursor: 0000000000000000000000000000000000000000000000000000000000000000012a00000000000000000000000000000000000000000000000000000000000000000108706f6f6c5f753634
< content-length: 194813
< date: Sun, 30 Oct 2022 18:40:25 GMT
<
{ [81155 bytes data]
100  190k  100  190k    0     0  3799k      0 --:--:-- --:--:-- --:--:-- 4135k
* Connection #0 to host 127.0.0.1 left intact
"acl"
"any"
"bcs"
"code"
"coin"
"guid"
"hash"
"util"
"block"
"debug"
"error"
"event"
"stake"
"table"
"math64"
"option"
"signer"
"string"
"vector"
"voting"
"account"
"ed25519"
"genesis"
"math128"
"version"
"vesting"
"bls12381"
"chain_id"
"features"
"from_bcs"

$ curl -v 'http://127.0.0.1:8080/v1/accounts/0x1/modules?limit=30&start=0000000000000000000000000000000000000000000000000000000000000000012a00000000000000000000000000000000000000000000000000000000000000000108706f6f6c5f753634' | jq .[].abi.name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/accounts/0x1/modules?limit=30&start=0000000000000000000000000000000000000000000000000000000000000000012a00000000000000000000000000000000000000000000000000000000000000000108706f6f6c5f753634 HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< x-aptos-chain-id: 4
< x-aptos-ledger-version: 244
< x-aptos-ledger-oldest-version: 0
< x-aptos-ledger-timestampusec: 1667155233292499
< x-aptos-epoch: 2
< x-aptos-block-height: 121
< x-aptos-oldest-block-height: 0
< x-aptos-cursor: 00000000000000000000000000000000000000000000000000000000000000000133000000000000000000000000000000000000000000000000000000000000000001117461626c655f776974685f6c656e677468
< content-length: 150314
< date: Sun, 30 Oct 2022 18:40:38 GMT
<
{ [81136 bytes data]
100  146k  100  146k    0     0  3664k      0 --:--:-- --:--:-- --:--:-- 4077k
* Connection #0 to host 127.0.0.1 left intact
"pool_u64"
"secp256k1"
"timestamp"
"type_info"
"aggregator"
"aptos_coin"
"aptos_hash"
"bit_vector"
"capability"
"comparator"
"simple_map"
"storage_gas"
"chain_status"
"copyable_any"
"gas_schedule"
"managed_coin"
"ristretto255"
"aptos_account"
"fixed_point32"
"multi_ed25519"
"staking_proxy"
"state_storage"
"staking_config"
"reconfiguration"
"transaction_fee"
"aptos_governance"
"consensus_config"
"resource_account"
"staking_contract"
"system_addresses"

$ curl -v 'http://127.0.0.1:8080/v1/accounts/0x1/modules?limit=30&start=00000000000000000000000000000000000000000000000000000000000000000133000000000000000000000000000000000000000000000000000000000000000001117461626c655f776974685f6c656e677468' | jq .[].abi.name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /v1/accounts/0x1/modules?limit=30&start=00000000000000000000000000000000000000000000000000000000000000000133000000000000000000000000000000000000000000000000000000000000000001117461626c655f776974685f6c656e677468 HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.84.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< x-aptos-chain-id: 4
< x-aptos-ledger-version: 306
< x-aptos-ledger-oldest-version: 0
< x-aptos-ledger-timestampusec: 1667155244812473
< x-aptos-epoch: 2
< x-aptos-block-height: 152
< x-aptos-oldest-block-height: 0
< content-length: 17388
< date: Sun, 30 Oct 2022 18:40:49 GMT
<
{ [17388 bytes data]
100 17388  100 17388    0     0   570k      0 --:--:-- --:--:-- --:--:--  679k
* Connection #0 to host 127.0.0.1 left intact
"table_with_length"
"aggregator_factory"
"governance_proposal"
"optional_aggregator"
"transaction_context"
"transaction_validation"

This change is

lightmark

lgtm now. Wait for @gregnazario to fix the ordering issue brought up by pagination.

banool · 2022-11-01T03:41:59Z

@lightmark you're saying we shouldn't land this and wait for @gregnazario to come back? What issue did he have?

lightmark · 2022-11-01T03:44:00Z

I mean we land first. the issue is now he sorts the query result. For everything it is fine, but for a page, it seems weird as the returned results page by page are not total ordered anyway. I'll let him decide on that. Not a big deal here.

banool · 2022-11-01T23:07:55Z

Okay I just added limit checking and unit tests for that. We still don't have unit tests for the 10k+ case though, I need to figure out a good way to get 10k resources on an account.

github-actions · 2022-11-02T17:12:58Z

✅ Forge suite `compat` success on `2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `e415fa8ceffa126b2905717976a052b999ad3e72`

Compatibility test results for 2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> e415fa8ceffa126b2905717976a052b999ad3e72 (PR)
1. Check liveness of validators at old version: 2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7437 TPS, 5220 ms latency, 8200 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: e415fa8ceffa126b2905717976a052b999ad3e72
compatibility::simple-validator-upgrade::single-validator-upgrade : 4764 TPS, 8527 ms latency, 11700 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: e415fa8ceffa126b2905717976a052b999ad3e72
compatibility::simple-validator-upgrade::half-validator-upgrade : 4878 TPS, 8623 ms latency, 10900 ms p99 latency,no expired txns
4. upgrading second batch to new version: e415fa8ceffa126b2905717976a052b999ad3e72
compatibility::simple-validator-upgrade::rest-validator-upgrade : 6736 TPS, 5769 ms latency, 10800 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for 2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> e415fa8ceffa126b2905717976a052b999ad3e72 passed
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions · 2022-11-02T17:14:46Z

✅ Forge suite `land_blocking` success on `e415fa8ceffa126b2905717976a052b999ad3e72`

performance benchmark with full nodes : 6853 TPS, 5792 ms latency, 12900 ms p99 latency,no expired txns
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

msmouse

this is an irresponsible stamp

banool changed the base branch from main to 1026alden-page October 26, 2022 22:05

banool changed the title ~~banool/cursor account resources~~ Add cursor support to /accounts/<addr>/resources Oct 26, 2022

banool force-pushed the banool/cursor_account_resources branch from 80a45b2 to 68d52be Compare October 26, 2022 22:52

lightmark force-pushed the 1026alden-page branch 2 times, most recently from b1b3928 to 6951322 Compare October 27, 2022 00:45

banool force-pushed the 1026alden-page branch from 6951322 to 4169021 Compare October 27, 2022 02:40

banool force-pushed the banool/cursor_account_resources branch 2 times, most recently from a9ed47e to dc13fad Compare October 27, 2022 03:02

lightmark force-pushed the 1026alden-page branch from 4169021 to c3cc03e Compare October 27, 2022 03:41

banool mentioned this pull request Oct 27, 2022

[Bug][API] Get account resources API returns 500 #5298

Closed

banool force-pushed the banool/cursor_account_resources branch 4 times, most recently from 1b7cc4d to 9127725 Compare October 30, 2022 18:21

banool changed the base branch from 1026alden-page to add_prefix_iter October 30, 2022 18:21

banool force-pushed the banool/cursor_account_resources branch 4 times, most recently from 314b975 to 624a5f6 Compare October 30, 2022 18:50

banool requested review from lightmark, davidiw and msmouse October 30, 2022 18:54

banool force-pushed the banool/cursor_account_resources branch from 624a5f6 to b2b5ded Compare October 30, 2022 18:55

banool marked this pull request as ready for review October 30, 2022 18:55

banool requested review from jjleng and gregnazario as code owners October 30, 2022 18:55

lightmark force-pushed the add_prefix_iter branch 4 times, most recently from 6b89524 to 71efff2 Compare October 31, 2022 22:07

lightmark approved these changes Oct 31, 2022

View reviewed changes

Base automatically changed from add_prefix_iter to main November 1, 2022 03:38

banool force-pushed the banool/cursor_account_resources branch from b2b5ded to 28c21c0 Compare November 1, 2022 03:40

banool force-pushed the banool/cursor_account_resources branch from 28c21c0 to e86da79 Compare November 1, 2022 17:25

banool force-pushed the banool/cursor_account_resources branch from e86da79 to 0683e06 Compare November 1, 2022 23:08

Add cursor support to /accounts/<addr>/resources

e415fa8

banool force-pushed the banool/cursor_account_resources branch from 0683e06 to e415fa8 Compare November 1, 2022 23:33

banool enabled auto-merge (squash) November 2, 2022 16:14

This comment has been minimized.

Sign in to view

msmouse approved these changes Nov 2, 2022

View reviewed changes

banool merged commit 5ff03ee into main Nov 2, 2022

banool deleted the banool/cursor_account_resources branch November 2, 2022 18:23

banool mentioned this pull request Nov 2, 2022

[TS][SDK] Use pagination in getAccountResources and getAccountModules #5405

Merged

banool added a commit that referenced this pull request Nov 16, 2022

Add cursor support to /accounts/<addr>/resources (#5313)

c3a3469

banool added a commit that referenced this pull request Nov 16, 2022

Add cursor support to /accounts/<addr>/resources (#5313)

ebcfb83

banool mentioned this pull request Nov 17, 2022

Fix functions for getting all resources / modules on an account by using pagination #5600

Merged

wrwg pushed a commit to wrwg/aptos-core that referenced this pull request Dec 6, 2022

Add cursor support to /accounts/<addr>/resources (aptos-labs#5313)

f2a5ede

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cursor support to `/accounts/<addr>/resources` #5313

Add cursor support to `/accounts/<addr>/resources` #5313

banool commented Oct 26, 2022 •

edited

Loading

lightmark left a comment

banool commented Nov 1, 2022

lightmark commented Nov 1, 2022 •

edited

Loading

banool commented Nov 1, 2022

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Nov 2, 2022

github-actions bot commented Nov 2, 2022

msmouse left a comment

Add cursor support to /accounts/<addr>/resources #5313

Add cursor support to /accounts/<addr>/resources #5313

Conversation

banool commented Oct 26, 2022 • edited Loading

Description

Test Plan: Resources

Test Plan: Modules

lightmark left a comment

Choose a reason for hiding this comment

banool commented Nov 1, 2022

lightmark commented Nov 1, 2022 • edited Loading

banool commented Nov 1, 2022

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Nov 2, 2022

✅ Forge suite compat success on 2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> e415fa8ceffa126b2905717976a052b999ad3e72

github-actions bot commented Nov 2, 2022

✅ Forge suite land_blocking success on e415fa8ceffa126b2905717976a052b999ad3e72

msmouse left a comment

Choose a reason for hiding this comment

Add cursor support to `/accounts/<addr>/resources` #5313

Add cursor support to `/accounts/<addr>/resources` #5313

banool commented Oct 26, 2022 •

edited

Loading

lightmark commented Nov 1, 2022 •

edited

Loading

✅ Forge suite `compat` success on `2d8b1b57553d869190f61df1aaf7f31a8fc19a7b` ==> `e415fa8ceffa126b2905717976a052b999ad3e72`

✅ Forge suite `land_blocking` success on `e415fa8ceffa126b2905717976a052b999ad3e72`