exp/ingest: Process ledger upgrade meta

[bartekn]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

Summary

This commits adds support for getting ledger upgrade changes from exp/ingest/io.LedgerReader and updates ledger processors in Horizon to process ledger upgrade meta.

Requirement for #1848.
Close #1470.

Goal and scope

As described in Integration doc in stellar-core the ledger meta changes need to be processed in the following order:

• transaction set "meta" (fee, txmeta)
• ledger upgrades "meta".

This commit adds missing ledger upgrade processing by adding a new method on LedgerReader that streams io.Change objects connected to a given ledger upgrades. Because processing ledger upgrade meta is essential to update the state correctly, LedgerReader.Close() will return an error in case some ledger upgrades have not been read.

Summary of changes

• Added ReadUpgradeChange to io.LedgerReader.
• Updated pipeline wrappers to pass reference to upgrades to the next reader in the pipeline.
• Updated Horizon processors to process ledger upgrade meta.

Known limitations & issues

• LedgerEntryChanges connected to ledger upgrade can be of arbitrary size. Currently all ledger upgrade changes are read fully into memory because they are stored in a single field in stellar-core DB (streaming them would require an awkward code that calls substr on an upgrade DB row). This should be solved when stellar-core ledger protocol is finished. LedgerReader API is designed to support streaming upgrade changes.

What shouldn't be reviewed

• Tests were not updated yet to check if upgrade meta processing works.

[tamirms]

I think you need to initialize reader. upgradeChanges to []Change{}. Otherwise it will not be possible to distinguish an empty list from nil in GetLedgerUpgradeChangesFromContext()

[bartekn]

That's interesting! I started wondering how is it possible that it works right now. It seems that the nil slice returned by interface{} return values is returned as empty slice (not nil): https://play.golang.org/p/1PHT_WH7URR. I think the answer is here.

[tamirms]

why is this separate from the LedgerReader interface?

[bartekn]

I didn't want to add GetUpgradeChanges() []Change to LedgerReader because users shouldn't get all upgrade changes, instead they should stream them. This is because upgrade changes can be even 1 GB according to stellar-core devs (but so far the largest were just a few MBs). It's hard to implement streaming now because all upgrade changes are in a single field in a DB. I guess the new stellar-core ledger protocol should solve it.

I created a new interface because a reader in LedgerPipeline can be one of: io.DBLedgerReader, readerWrapperLedger or reporterLedgerReader. Instead of checking the types separately, they all implement this interface.

[bartekn]

Added tests in 2ac7fe3. EDIT: and in: a573436.

[bartekn]

@tamirms I realized that there are several things I needed to fix:

• The most important thing was that if ledger upgrade changes length was 0, Close() didn't report any errors so it was easy to forget about ReadUpgradeChange().
• Added LedgerReader.IgnoreUpgradeChanges() method to allow ignoring ledger upgrade changes (ex. if processor doesn't care about metas).
• No tests for account processing.

Can you check 74478d5 and 👍 if it's OK?

[tamirms]

@bartekn the commit looks good