exp/ingest: verify-range Horizon command and RangeSession

[bartekn]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

Adds horizon expingest verify-range command and exp/ingest.RangeSession.

Close #1848.

Why

In order to ensure that new ingestion system works correctly we plan to reingest pubnet history from the first ledger. In order to achieve this we add horizon expingest verify-range command that accepts required two params: --from and --to that describe the ledger range. When started it:

• It ingests ledger entries of from ledger (must be a checkpoint ledger).
• Ingests ledgers [from+1, to].
• Runs verifyState function on to ledger (must be a checkpoint ledger when --verify-state).

When merged we can start multiple workers ingesting full history in parallel like:

horizon expingest verify-range --from 9983 --to 16383 --verify-state
horizon expingest verify-range --from 16383 --to 22783 --verify-state
...

Internally, it's using a new RangeSession from exp/ingest package that process a given range of ledger. RangeSession will be also used in horizon reingest commands.

It's also helpful for finding performance issues as it's easy to reingest a given range with a debug server enabled.

runs ingestion pipeline between X and Y sequence number (inclusive)

Usage:
  horizon expingest verify-range [flags]

Flags:
      --debug-server-port uint32   [optional] opens a net/http/pprof server at given port
      --from uint32                first ledger of the range to ingest
      --to uint32                  last ledger of the range to ingest
      --verify-state               [optional] verifies state at the last ledger of the range when true

Known limitations

A small update to db.Session was needed. When using Session.BeginTx, Ctx on Session must be set, otherwise it panics with panic: runtime error: invalid memory address or nil pointer dereference in database/sql/sql.go:1140. db.Open function does not set Ctx.

[abuiles]

Suggested change

	// initState initialilizes the state using given arguments. Common code for
	// initState initializes the state using given arguments. Common code for

[abuiles]

LGTM!

[abuiles]

@bartekn it seems like this could be nil shall we check if this was passed or not before calling process?

[abuiles]

what's the reason for this update?

[abuiles]

should we reframe this a little bit like:

"It will exit if another go routine is already running or the ledger sequence is old (ex. catching up)"?

[bartekn]

Closed in favour of #2086 (easier to do when expingest.System is a state machine).