xdr: changes for auth and removal of mutli-invoke (wip) #4900

chowbao · 2023-06-05T20:36:47Z

PR Checklist

PR Structure

This PR has reasonably narrow scope (if not, break it down into smaller PRs).
This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
This PR's title starts with name of package that is most changed in the PR, ex.
services/friendbot, or all or doc if the changes are broad or impact many
packages.

Thoroughness

This PR adds tests for the most critical parts of the new functionality or fixes.
I've updated any docs (developer docs, .md
files, etc... affected by this change). Take a look in the docs folder for a given service,
like this one.

Release planning

I've updated the relevant CHANGELOG (here for Horizon) if
needed with deprecations, added features, breaking changes, and DB schema changes.
I've decided if this PR requires a new major/minor version according to
semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.

What

#4895
Updates to support auth and removal of multi-invoke

xdr changes up to stellar/stellar-xdr@de264d8
Core version 19.11.1-1328.cfa6a8918.focal~authchanges
All instances of InvokeHostFunctions changed to InvokeHostFunction with only single HostFunction
txnbuild updates
- Added Auth []xdr.SorobanAuthorizationEntry to InvokeHostFunction struct
Ingest
- Replace GetOperationEvents with GetDiagnosticEvents to return diagnostic events
Update Horizon operation_processor_tests for new InvokeHostFunction format
sac_test updated to use ResultMetaXdr to get the InvokeHostFunction return ScVal

Why

Updates needed for preview 10

Known limitations

[TODO or N/A]

…tellar#4884)

…lar#4887)

* services/horizon: Suppress Core timeout error (stellar#4860) * services/horizon: Protect 'currentState' variable using Mutex to prevent race condition. (stellar#4889) * services/horizon: Modify the tests due to changes in the Begin function signature.

chowbao · 2023-06-05T20:39:29Z

@tsachiherman I pulled in the xdr changes needed to update the generated xdr.
I did this basically by updating/copying from stellar-xdr. Is there anything else y'all usually do to update xdr changes?

Currently trying to get/build the new stellar-core image

Next will try to update the horizon ingest processors

chowbao · 2023-06-05T20:41:12Z

services/horizon/internal/integration/contracts/Cargo.toml

@@ -22,6 +22,7 @@ codegen-units = 1
 lto = true

 [workspace.dependencies.soroban-sdk]
-version = "0.8.0"
+# TODO: Does a new version need to be released to update to 8a7d2c3c8c5bc1664ab38f977ff947cc2a26d694


@tsachiherman Unsure if there is any change needed to https://github.com/stellar/rs-soroban-sdk to update the version.
From looking at this these changes are just used to generate wasm testdata

…ellar#4892)

sreuland · 2023-06-12T15:53:53Z

Makefile

@@ -15,7 +15,7 @@ xdr/Stellar-internal.x \
 xdr/Stellar-contract-config-setting.x

 XDRGEN_COMMIT=80e38ef2a96489f6b501d4db3a350406e5aa3bab
-XDRNEXT_COMMIT=2f16687fdf6f4bcfb56805e2035f69997f4b34c4
+XDRNEXT_COMMIT=de264d81ea87dfaa19e089b60fed8eb0219b2ab4


@chowbao , did you run make xdr-update after this change to create the updates on .x files and generated code, or did you manually copy those .x over and run generate, probably is same end result, was just curious which approach.

Yeah I updated the XDRNEXT_COMMIT then ran make xdr-update. I guess copying the files would have worked too but I still would have had to run like make xdr to get the generated xdr

…lar#4903)

chowbao · 2023-06-13T12:34:57Z

services/horizon/internal/ingest/processors/asset_stats_processor_test.go

@@ -1122,7 +1122,7 @@ func (s *AssetStatsProcessorTestSuiteLedger) TestUpdateContractIDError() {

 	s.Assert().EqualError(
 		s.processor.Commit(s.ctx),
-		"attempting to set contract id b729e13867d5c4b2d161574e00854fd41bbba3e3b0e31d36c64904414a862fa7 but row credit_alphanum4/EUR/GBRPYHIL2CI3FNQ4BXLFMNDLFJUNPU2HY3ZMFSHONUCEOASW7QC7OX2H already has contract id set: 6645621141097c0f88b99360ce267c3396bcfd8cfdbe9c462b0dc167b72ecdc4",
+		"attempting to set contract id 67b1f192e30d8cd56dcb103c783cfee753588a434ad1092ef8a39375c9738bab but row credit_alphanum4/EUR/GBRPYHIL2CI3FNQ4BXLFMNDLFJUNPU2HY3ZMFSHONUCEOASW7QC7OX2H already has contract id set: c9cd6bbf712cfd4e086f0e25f90722c1af24416c0303728ad4527a6967ddc51b",


Not sure if these changes to the contract Ids are correct or not. These were all updated to the results from running the tests

chowbao · 2023-06-13T12:36:35Z

services/horizon/internal/integration/contracts/Cargo.toml

 git = "https://github.com/stellar/rs-soroban-sdk"
-rev = "090f5c88e8f8b67f4d02010665aedfea6ee43db3"
+rev = "8a7d2c3c8c5bc1664ab38f977ff947cc2a26d694"


Currently not able to cargo build due to rs-soroban-sdk differences

…es-for-auth-and-remove-multi-invoke

tsachiherman

rebasing to soroban-xdr-next-next so that we can keep working on this one while doing changes to soroban-rpc.

* all: enforce simplified Golang code (#4852) * Update completed sprint on issue/pr closed (#4857) * Bump core image to latest stable release v19.10.0 * Add a simple test for asset case sorting in ascii (#4876) * services/horizon: Suppress Core timeout error (#4860) Suppress Core timeout error when ingestion state machine is in build state. * Update CHANGELOG.md for latest release (#4828) * Bump core image to latest release v19.11.0 (#4885) * services/horizon: Protect 'currentState' variable using Mutex to prevent race condition. (#4889) * services/horizon: Update default for --captive-core-use-db to true (#4877) * 4856: Update default for --captive-core-use-db to true * Update CHANGELOG.md * xdr: changes for auth and removal of mutli-invoke (wip) (#4900) * updated core git ref for tests (#4879) * update toml to support LIMIT_TX_QUEUE_SOURCE_ACCOUNT (#4882) * horizon/ingest/processors: SAC storage entry by different key name (#4884) * services/horizon: Reenable InvokeHostFunction integration tests (#4887) * horizon: 4446/supress core timeout error (#4894) * services/horizon: Suppress Core timeout error (#4860) * services/horizon: Protect 'currentState' variable using Mutex to prevent race condition. (#4889) * services/horizon: Modify the tests due to changes in the Begin function signature. * xdr: changes for auth and removal of mutli-invoke (wip) * services/horizon: Reenable SAC integration tests (#4891) * services/horizon: optionally add soroban-rpc to integration tests (#4892) * Update core image * Update stellar-xdr next commit * Update horizon ingestion * update txnbuild * Updates * Update fmt * fix txnbuild tests * updates * Update horizon tests * add ingest test * Update operation processor test * Update tests * Update tests * Remove sac test changes to resolve merge conflicts for now * Formatting * 4902 Add mutex for concurrent access in GetLatestLedgerSequence (#4903) * Update tests * Update sac test * update tests --------- Co-authored-by: shawn <sreuland@users.noreply.github.com> Co-authored-by: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> Co-authored-by: Alfonso Acosta <alfonso@stellar.org> Co-authored-by: urvisavla <urvi.savla@stellar.org> * services/horizon: Improve error handling for when stellar-core crashes (#4893) * Parse LIMIT_TX_QUEUE_SOURCE_ACCOUNT in core config * updated changelog for 2.26.0 release notes * Pinning and updates golang and ubuntu images * xdr: update per 7b403105 (#4923) * Simplify ScError.Equals * Add missing case for ScVal.Equals * Add LedgerEntry.SetContractData and LedgerEntry.SetContractCode methods * xdr updates * fix formatting * update * update transfer_event_xdr.bin * update * update * update * bugfix * update * update * update * handle nil entries * fix few linting issues * additional linting * Fix missing err check * fix missing nil check * Minor code checker fix * Fix minor naming * fix hard-coded values. * warn and don't set expiration ledger * Warn and don't set expirationLedger * update new xdr * update per peer review. * Add effect for bumpFootprintExpirationOp * Fix govet-caught bug * update per 7b403105788e33044e089c4c2f957df8ddabaca8 * update fmt * fix unit test * Update core image --------- Co-authored-by: Paul Bellamy <paul@stellar.org> Co-authored-by: Simon Chow <simon.chow@stellar.org> * services/horizon: Fix ledger endpoint url in HAL (#4928) * Don't crash on LedgerCloseMetaV2 ingestion (#4927) * xdr: update per xdr version e372df9 (#4930) * update * update * update * bugfix * update * update per linter * update per peer review. * Update to use 19.11.1-1349.fae91b092.focal~soroban * Goreplay middleware (#4932) * tools/goreplay-middleware: Add goreplay middleware * Fix linter errors --------- Co-authored-by: Bartek Nowotarski <bartek@nowotarski.info> * Update to final core preview 10 image * all: Fix improper use of errors.Wrap (#4926) * all: Fix improper use of errors.Wrap `errors.Wrap` method returns nil if the first argument passed is also nil. If `errors.Wrap` is copied from a condition like `if err != nil` to another one which also returns `errors.Wrap` but does not overwrite `err` before the returned value will always be `nil`. * Update services/horizon/internal/db2/history/claimable_balances.go Co-authored-by: George <Shaptic@users.noreply.github.com> --------- Co-authored-by: George <Shaptic@users.noreply.github.com> Co-authored-by: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> * fix apt repo reference to focal now (#4929) * LedgerChangeReader: Support State Expiration & Eviction (#4941) * Simplify ScError.Equals * Add missing case for ScVal.Equals * Add LedgerEntry.SetContractData and LedgerEntry.SetContractCode methods * update generated xdr to 0f5e556 * Add LedgerCloseMetaV2 support * Update ledgerTransaction.GetOperationEvents * Make LedgerChangeReader emit evictions * Fixing up after merge * Add test for LedgerChangeReader extensions and evictions * Fix typo * Add xdr.LedgerEntryData.ExpirationLedgerSeq helper * review feedback * merge latest master to soroban-xdr-next-next (#4937) * all: enforce simplified Golang code (#4852) * Update completed sprint on issue/pr closed (#4857) * Bump core image to latest stable release v19.10.0 * Add a simple test for asset case sorting in ascii (#4876) * services/horizon: Suppress Core timeout error (#4860) Suppress Core timeout error when ingestion state machine is in build state. * Update CHANGELOG.md for latest release (#4828) * Bump core image to latest release v19.11.0 (#4885) * services/horizon: Protect 'currentState' variable using Mutex to prevent race condition. (#4889) * services/horizon: Update default for --captive-core-use-db to true (#4877) * 4856: Update default for --captive-core-use-db to true * Update CHANGELOG.md * services/horizon: Improve error handling for when stellar-core crashes (#4893) * Parse LIMIT_TX_QUEUE_SOURCE_ACCOUNT in core config * updated changelog for 2.26.0 release notes * Pinning and updates golang and ubuntu images * services/horizon: Fix ledger endpoint url in HAL (#4928) * Goreplay middleware (#4932) * tools/goreplay-middleware: Add goreplay middleware * Fix linter errors --------- Co-authored-by: Bartek Nowotarski <bartek@nowotarski.info> * all: Fix improper use of errors.Wrap (#4926) * all: Fix improper use of errors.Wrap `errors.Wrap` method returns nil if the first argument passed is also nil. If `errors.Wrap` is copied from a condition like `if err != nil` to another one which also returns `errors.Wrap` but does not overwrite `err` before the returned value will always be `nil`. * Update services/horizon/internal/db2/history/claimable_balances.go Co-authored-by: George <Shaptic@users.noreply.github.com> --------- Co-authored-by: George <Shaptic@users.noreply.github.com> Co-authored-by: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> * fix apt repo reference to focal now (#4929) * fixed go fmt on bindata * fixed merge conflict snippet * fixed manual merge commit omition --------- Co-authored-by: Alfonso Acosta <alfonso@stellar.org> Co-authored-by: Paul Bellamy <paul@stellar.org> Co-authored-by: Mehmet <119539688+mbsdf@users.noreply.github.com> Co-authored-by: mlo <marta.lohova@gmail.com> Co-authored-by: urvisavla <urvi.savla@stellar.org> Co-authored-by: stellarsaur <126507441+stellarsaur@users.noreply.github.com> Co-authored-by: Molly Karcher <molly@stellar.org> Co-authored-by: Bartek Nowotarski <bartek@nowotarski.info> Co-authored-by: George <Shaptic@users.noreply.github.com> Co-authored-by: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> * Refactor `xdr.LedgerEntry.LedgerKey` method (#4942) * Simplify ScError.Equals * Add missing case for ScVal.Equals * Add LedgerEntry.SetContractData and LedgerEntry.SetContractCode methods * update generated xdr to 0f5e556 * Add LedgerCloseMetaV2 support * Update ledgerTransaction.GetOperationEvents * Make LedgerChangeReader emit evictions * Fixing up after merge * Add test for LedgerChangeReader extensions and evictions * Fix typo * Add xdr.LedgerEntryData.ExpirationLedgerSeq helper * review feedback * Refactor LedgerEntry.LedgerKey method to avoid panics, and only have a single copy * Fix govet.sh * Fixing govet * Fix bug * Remove unneeded code bits * s/marshalling/marshaling * Review feedback * integration tests: Add horizon test support for new bump/restore footprint ops (#4944) * use the new core image for local integration tests * Add bump/restoreFootprint ops to the horizon reingestion integration tests * services/horizon: Remove command line flag --remote-captive-core-url (#4940) * txnbuild: Make bump and restore footprint soroban operations (#4946) * integration tests: fix integration tests for preview 10 (#4938) this pr fixes the invokehostfunciton_test only, sac_test and contract event tests will be addressed in separate, follow-onw pr. * fix sac tests for preview 10 data model (#4951) * fix merge issue. --------- Co-authored-by: Alfonso Acosta <alfonso@stellar.org> Co-authored-by: Paul Bellamy <paul@stellar.org> Co-authored-by: Mehmet <119539688+mbsdf@users.noreply.github.com> Co-authored-by: mlo <marta.lohova@gmail.com> Co-authored-by: urvisavla <urvi.savla@stellar.org> Co-authored-by: stellarsaur <126507441+stellarsaur@users.noreply.github.com> Co-authored-by: chowbao <simon.chow765@gmail.com> Co-authored-by: shawn <sreuland@users.noreply.github.com> Co-authored-by: Molly Karcher <molly@stellar.org> Co-authored-by: Shawn Reuland <shawn@stellar.org> Co-authored-by: Simon Chow <simon.chow@stellar.org> Co-authored-by: Bartek Nowotarski <bartek@nowotarski.info> Co-authored-by: George <Shaptic@users.noreply.github.com>

sreuland and others added 6 commits May 22, 2023 14:15

updated core git ref for tests (stellar#4879)

7393068

update toml to support LIMIT_TX_QUEUE_SOURCE_ACCOUNT (stellar#4882)

7f0c9ea

horizon/ingest/processors: SAC storage entry by different key name (s…

2ef0183

…tellar#4884)

services/horizon: Reenable InvokeHostFunction integration tests (stel…

317e74e

…lar#4887)

xdr: changes for auth and removal of mutli-invoke (wip)

31c953e

chowbao requested a review from tsachiherman June 5, 2023 20:36

chowbao commented Jun 5, 2023

View reviewed changes

2opremio and others added 15 commits June 6, 2023 17:08

services/horizon: Reenable SAC integration tests (stellar#4891)

3c0ccbe

services/horizon: optionally add soroban-rpc to integration tests (st…

9ff77c2

…ellar#4892)

Update core image

d896924

Update stellar-xdr next commit

889ab99

Update horizon ingestion

5ca87bf

update txnbuild

aab1b89

Updates

32a15a8

Update fmt

913451c

fix txnbuild tests

2c9e137

updates

1b65208

Update horizon tests

883bfaf

add ingest test

39927c4

Update operation processor test

2b2661f

Update tests

4513e38

Update tests

bb81d5d

tsachiherman assigned chowbao Jun 12, 2023

tsachiherman added preview-10 soroban-scrum labels Jun 12, 2023

sreuland reviewed Jun 12, 2023

View reviewed changes

chowbao added 2 commits June 12, 2023 13:55

Remove sac test changes to resolve merge conflicts for now

c8ea693

Formatting

d3a1c46

urvisavla and others added 2 commits June 13, 2023 00:08

4902 Add mutex for concurrent access in GetLatestLedgerSequence (stel…

e20e8bc

…lar#4903)

Update tests

83967f8

chowbao commented Jun 13, 2023

View reviewed changes

chowbao added 3 commits June 13, 2023 08:50

Update sac test

5e1ea98

Merge remote-tracking branch 'origin/soroban-xdr-next' into xdr-chang…

c838629

…es-for-auth-and-remove-multi-invoke

update tests

bc35c89

tsachiherman changed the base branch from soroban-xdr-next to soroban-xdr-next-next June 13, 2023 18:14

tsachiherman approved these changes Jun 13, 2023

View reviewed changes

tsachiherman marked this pull request as ready for review June 13, 2023 18:17

tsachiherman merged commit 67d249d into stellar:soroban-xdr-next-next Jun 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

xdr: changes for auth and removal of mutli-invoke (wip) #4900

xdr: changes for auth and removal of mutli-invoke (wip) #4900

chowbao commented Jun 5, 2023 •

edited

Loading

chowbao commented Jun 5, 2023 •

edited

Loading

chowbao Jun 5, 2023

sreuland Jun 12, 2023

chowbao Jun 12, 2023 •

edited

Loading

chowbao Jun 13, 2023

chowbao Jun 13, 2023

tsachiherman left a comment

xdr: changes for auth and removal of mutli-invoke (wip) #4900

xdr: changes for auth and removal of mutli-invoke (wip) #4900

Conversation

chowbao commented Jun 5, 2023 • edited Loading

PR Structure

Thoroughness

Release planning

What

Why

Known limitations

chowbao commented Jun 5, 2023 • edited Loading

chowbao Jun 5, 2023

Choose a reason for hiding this comment

sreuland Jun 12, 2023

Choose a reason for hiding this comment

chowbao Jun 12, 2023 • edited Loading

Choose a reason for hiding this comment

chowbao Jun 13, 2023

Choose a reason for hiding this comment

chowbao Jun 13, 2023

Choose a reason for hiding this comment

tsachiherman left a comment

Choose a reason for hiding this comment

chowbao commented Jun 5, 2023 •

edited

Loading

chowbao commented Jun 5, 2023 •

edited

Loading

chowbao Jun 12, 2023 •

edited

Loading