xdrill for ledgerCloseMeta

[chowbao]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've reviewed the changes in this PR and if I consider them worthwhile for being mentioned on release notes then I have updated the relevant CHANGELOG.md within the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

github issue

xdrill functions for LedgerCloseMeta

Why

First commit for xdrill with the low level helper functions from the processors/transforms from stellar-etl for history_ledgers found here

Known limitations

• Refactor of the processor/transforms to be done in separate ticket/pr

[chowbao]

How exhaustive do we want the testing to be?

[tamirms]

this struct definition would not work outside of the xdrill package. Although the Ledger struct is exported, the ledger field remains private to external packages. So, if you would not be able to construct a Ledger instance outside of the xdrill package because you would not be able to reference the ledger field.

To fix this issue you can make the ledger field public:

type Ledger struct {
	Ledger *xdr.LedgerCloseMeta
}

Alternatively, you could create a type definition:

type Ledger xdr.LedgerCloseMeta

func (l Ledger) Sequence() uint32 {
	return uint32(l.ledger.LedgerHeaderHistoryEntry().Header.LedgerSeq)
}

Then to call the helper function you would cast your xdr.LedgerCloseMeta value into an xdrill.Ledger value so you could call the helper function:

package app

import (
	"github.com/stellar/go/exp/xdrill"
	"github.com/stellar/go/xdr"
)

func doSomething(ledger xdr.LedgerCloseMeta) {
  l := xrill.Ledger(ledger)
  seq := l.Sequence()
  ...
}

Or you could make the helper functions take the ledger parameter as the first argument:

func LedgerSequence(ledger xdr.LedgerCloseMeta) uint32 {
	return uint32(ledger.LedgerHeaderHistoryEntry().Header.LedgerSeq)
}

IMO the cleanest API would be define helper functions like these in the xdr package directly on the LedgerCloseMeta type. I think I missed the discussion about this decision. What was the motivation for defining a type equivalent to xdr.LedgerCloseMeta in the xdrill package?

[chowbao]

Ah oops yeah it should be public

[tamirms]

I forgot to mention one other possible solution, you could leave the ledger field as private and expose a public constructor:

type Ledger struct {
	ledger *xdr.LedgerCloseMeta
}

func NewLedger(ledger xdr.LedgerCloseMeta) Ledger {
   return Ledger{ledger: &ledger}
}

[chowbao]

What was the motivation for defining a type equivalent to xdr.LedgerCloseMeta in the xdrill package?

Talked in slack. tl;dr - it's nicer for users as a separate package because they would be given a subset of only XDRill functions instead of all the functions/fields within xdr and ingest

[chowbao]

Updated functions to accept lcm as a param instead of the Ledger struct
d45b0fd

[chowbao]

Moved xdrill ledger functions to go/ingest/ledger
40d5057

[tamirms]

utils packages are generally discouraged in go:

https://google.github.io/styleguide/go/decisions.html#package-names

[chowbao]

Ah ok. What is the correct way to organize shared "utils"/"common" functions?

Would just renaming the package to xdrillutils be good?
Or should I split it out into more descriptive packages like xdrillschemas, xdrillconversions?

[tamirms]

I think xdrillutils is basically equivalent to utils. In general, package names should be related to what the code actually does and if there is no unifying theme for all the code in the package then it could be that the code should be separated into multiple packages

for example, CreateSampleTx() is used to generate to testing fixtures. you could argue that it doesn't need to even be exported outside of the testing files which use it.

[chowbao]

Yeah makes sense. I'll try splitting them up into more relevantly named packages. Maybe ChatGPT can offer some suggestions lol

[chowbao]

Moved util functions to relevant places
40d5057