Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

services/horizon/internal: Fix configuration of captive core in reingest command #5069

Merged
merged 4 commits into from
Oct 3, 2023

Conversation

tamirms
Copy link
Contributor

@tamirms tamirms commented Sep 27, 2023

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with name of package that is most changed in the PR, ex.
    services/friendbot, or all or doc if the changes are broad or impact many
    packages.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated any docs (developer docs, .md
    files, etc... affected by this change). Take a look in the docs folder for a given service,
    like this one.

Release planning

  • I've updated the relevant CHANGELOG (here for Horizon) if
    needed with deprecations, added features, breaking changes, and DB schema changes.
  • I've decided if this PR requires a new major/minor version according to
    semver, or if it's mainly a patch change. The PR is targeted at the next
    release branch if it's not a patch change.

What

When running horizon --db-url postgres://postgres@localhost:5432?sslmode=disable --apply-migrations db reingest range 48263265 48263265 with the following environment variables:

HISTORY_ARCHIVE_URLS=https://history.stellar.org/prd/core-live/core_live_001;INGEST=true
NETWORK_PASSPHRASE=Public Global Stellar Network \; September 2015
STELLAR_CORE_BINARY_PATH=/usr/local/bin/stellar-core

I encountered the error listed below:

2023/09/27 12:51:23 Applying DB migrations...
2023/09/27 12:51:24 successfully applied 66 horizon migrations
2023/09/27 12:51:24 Checking DB migrations...
2023/09/27 12:51:24 Preparing captive core...
INFO[2023-09-27T12:51:24.383+01:00] Ingestion system initial state                current_state="reingestHistoryRange(fromLedger=48263265, toLedger=48263265, force=false)" pid=65279 service=ingest
INFO[2023-09-27T12:51:24.385+01:00] Preparing ledger backend to retrieve range    from=48263265 pid=65279 service=ingest to=48263265
INFO[2023-09-27T12:51:25.212+01:00] Warning: soroban-env-host-curr is running a pre-release version 20.0.0-rc1  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.215+01:00] default: Config from /Users/tamir/work/gopath/src/github.com/stellar/go/captive-core-09ac164f/stellar-core.conf  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.215+01:00] default: 'RUN_STANDALONE' enabled in configuration file - node will not function properly with most networks  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.216+01:00] default: Using QUORUM_SET: {                  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.216+01:00] "t" : 1,                                      pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.216+01:00] "v" : [ "GCZBO" ]                             pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.216+01:00] }                                             pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.217+01:00] Database: Connecting to: sqlite3://stellar.db  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.221+01:00] SCP: LocalNode::LocalNode@GA4HX qSet: 1dcacc  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.224+01:00] default: *                                    pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.224+01:00] default: * The database has been initialized  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.224+01:00] default: *                                    pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.225+01:00] Database: Applying DB schema upgrade to version 14  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.225+01:00] Database: Applying DB schema upgrade to version 15  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.225+01:00] Database: Applying DB schema upgrade to version 16  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.225+01:00] Database: Applying DB schema upgrade to version 17  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.225+01:00] Database: Applying DB schema upgrade to version 18  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.225+01:00] Database: Applying DB schema upgrade to version 19  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.226+01:00] Database: Applying DB schema upgrade to version 20  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.226+01:00] Database: Applying DB schema upgrade to version 21  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.226+01:00] Database: DB schema is in current version     pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.226+01:00] default: Dropping accounts                    pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.226+01:00] default: Dropping trustlines                  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.226+01:00] default: Dropping offers                      pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.226+01:00] default: Dropping accountdata                 pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.232+01:00] default: Dropping claimablebalances           pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.232+01:00] default: Dropping liquiditypools              pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.232+01:00] default: Dropping contractdata                pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.232+01:00] default: Dropping contractcode                pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.232+01:00] default: Dropping configsettings              pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.232+01:00] default: Dropping expiration                  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.233+01:00] Ledger: Established genesis ledger, closing   pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.233+01:00] Ledger: Root account: GC7JA62LVSCP5ZOORAI5WLPPZG7QWKRKFO6D2VGYUISX5TLQIQMWEIY3  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.233+01:00] Ledger: Root account seed: SDR3BRCCTD6BYFE27P2MRGLPXESCPLSB4RSJXE2MUSKZSG3YKK4FL6HO  pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.235+01:00] default: Application destructing              pid=65279 service=ingest subservice=stellar-core
INFO[2023-09-27T12:51:25.235+01:00] default: Application destroyed                pid=65279 service=ingest subservice=stellar-core
ERRO[2023-09-27T12:51:25.237+01:00] default: Got an exception: NETWORK_PASSPHRASE not configured  pid=65279 service=ingest subservice=stellar-core
ERRO[2023-09-27T12:51:25.237+01:00] default: Please report this bug along with this log file if this was not expected  pid=65279 service=ingest subservice=stellar-core
ERRO[2023-09-27T12:51:25.241+01:00] Error in ingestion state machine              current_state="reingestHistoryRange(fromLedger=48263265, toLedger=48263265, force=false)" error="error preparing range: error starting prepare range: opening subprocess: error running stellar-core: error initializing core db: exit status 1" next_state=stop pid=65279 service=ingest
INFO[2023-09-27T12:51:25.241+01:00] Shut down                                     pid=65279 service=ingest
INFO[2023-09-27T12:51:25.241+01:00] Shutting down ingestion system...             pid=65279 service=ingest
error preparing range: error starting prepare range: opening subprocess: error running stellar-core: error initializing core db: exit status 1
Exiting.

It looks like this regression was introduced in https://github.com/stellar/go/pull/4949/files . Prior to #4949 we were overriding config.CaptiveCoreTomlParams.CoreBinaryPath , config.CaptiveCoreTomlParams.HistoryArchiveURLs, and config.CaptiveCoreTomlParams.NetworkPassphrase before constructing the captive core toml. But, it seems like we missed one of the cases in
It looks like this regression was introduced in https

@tamirms tamirms requested a review from urvisavla September 27, 2023 12:03
 - Include validation for an empty captive-core-config-path in the regular ingestion
process with captive core.
 - Update unit tests to cover different scenarios of captive-core-config-paths, for both
ingestion where "RequireCaptiveCoreConfig" is True and reingestion where "RequireCaptiveCoreConfig" is False.
@urvisavla urvisavla requested a review from sreuland September 28, 2023 16:13
@urvisavla urvisavla merged commit facabfc into stellar:master Oct 3, 2023
2opremio pushed a commit that referenced this pull request Oct 16, 2023
…est command (#5069)

* Fix configuration of captive core in reingest command
* Add validation for an empty --captive-core-config-path.
---------

Co-authored-by: Urvi <urvi.savla@stellar.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants