services/horizon: Add accounts and account data to ingestion pipeline

[bartekn]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

Summary

This commit adds accounts and account data to ingestion pipeline.

Summary of changes

• Adds a new migration file adding a new accounts and accounts_data tables.
• Updates db2/history package with accounts and data related interfaces and structs.
• Updates DatabaseProcessor to process accounts and data in state and ledger pipelines.

[abuiles]

👍 -- I think this will be useful too for #442

[bartekn]

TODO state verification fails:

ERRO[2019-10-12T02:27:07.174+02:00] STATE IS INVALID!                             err="addAccountsToStateVerifier failed: Entry does not match the fetched entry. Expected: AZC2YQAAAAAAAAAANamIrB+fg1XeekYeLuM8ZRkDHiXdoULzK1TiyJlZNTsAAAAAALOBQgFZVWYAAA9ZAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA== (pretransform = AZC2YQAAAAAAAAAANamIrB+fg1XeekYeLuM8ZRkDHiXdoULzK1TiyJlZNTsAAAAAALOBQgFZVWYAAA9ZAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==), actual: AZCjfQAAAAAAAAAANamIrB+fg1XeekYeLuM8ZRkDHiXdoULzK1TiyJlZNTsAAAAAALOBpgFZVWYAAA9YAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==" pid=44881 service=expingest

[bartekn]

Fixed an issue connected to state verification in 19187bc09648b9e3c950c745fc34df6ffeecac9f but this PR is currently blocked by #1836.

[abuiles]

@bartekn why did we have to change to .Post.Data?

[bartekn]

Good question, I forgot to mention this. It's because I changed /exp/ingest/io.Change:

type Change struct {
	Type xdr.LedgerEntryType
-	Pre  *xdr.LedgerEntryData
+	Pre  *xdr.LedgerEntry
-	Post *xdr.LedgerEntryData
+	Post *xdr.LedgerEntry
}

LedgerEntry is LedgerEntryData plus LastModifiedLedgerSeq. I had to do this because stellar-core sometimes generates a change in metas even though ledger entry actually doesn't change (ex. core loads an entry to update the XLM balance but it doesn't change because amount sold and bought are both zeros due to rounding). You can see this for GAXB325M4CYY5IYFEKOPH5H6LLILECK7Y5RAVTB3THN3LKMGIVDJCICU in this transaction meta: https://horizon.stellar.org/transactions/f87b374b377ca66d763dafb9ef408726ce28d90b9994fc81ceb767c1a20d2cdb. It's actually going to be changed in stellar-core (CAP-0017) one day.

Without changing this we wouldn't know that we need to update LastModifiedLedgerSeq in a DB and state verifier would fail (it's actually how I found this issue).

[abuiles]

👍

[abuiles]

@bartekn oh I missed this, it seems like we have been using accountid in other places. Not a big issue though.

[bartekn]

Good point, we should make it consistent: #1858