Integrate latest ledger dependencies

[lehins]

The goal of this PR is to integrate latest changes in the cardano-ledger, which accrued over the period of the last few months. Most notable breaking changes in ledger are:

• Addition of Conway era
• Implementation of versioned CBOR serialization
• Switching to lenses for manipulating of PParams
• Changes to CostModel serialization
• Many more ...

Besides integrating ledger this PR also:

• Accounts for recent changes in consensus
• Changes representation of CostModel
• Fixes a bug where only metadata from TxAuxData was hashed upon transaction body creation with createTransactionBody
• Reduces a whole lot of duplication
• Moves all orphan instances for types defined in cardano-base and cardano-ledger
• Fixes issues with calculateMinimumUTxO funciton:
  • Proper fix for minutxo computation: [BUG] - min required UTxO reported by CLI is not correct #4061
  • Makes that function total

[newhoggy]

LGTM

[Jimbo4350]

Thank you @lehins for propagating the changes from ledger. I just have a couple comments

1. I think the ProtocolParameter related improvments that are not directly related to ledger changes should be in a separate PR.
2. I'd prefer to not use lenses if we don't have to. We're not updating the fields of deeply nested structures and I think the lenses further obfuscate the code vs record accessors which immediately give me information/context about the data structure I am updating.

There are many good refactors here 👍 . We should definitely organize a call next week so you can take us through the changes.

[lehins]

I think the ProtocolParameter related improvements that are not directly related to ledger changes should be in a separate PR.

They are all related to the ledger changes, so they can't be extracted into a separate PR. PParams and PParamsUpdate have been completely redone in the latest version of ledger. Here is the relevant PR: IntersectMBO/cardano-ledger#3242

I'd prefer to not use lenses if we don't have to.

We have to. Every single field in almost every ledger type family is now to be manipulated by a dedicated lens.

We're not updating the fields of deeply nested structures and I think the lenses further obfuscate the code vs record accessors which immediately give me information/context about the data structure I am updating.

In the matter of fact we are. This is how you'd add inputs to a transaction:

> let addInputs tx ins = tx & bodyTxL . inputsTxBodyL <>~ ins
> :t addInputs
addInputs
  :: EraTx era => Tx era -> Set (TxIn (EraCrypto era)) -> Tx era

It is important to note that this function here will work for all eras. This is the power of lenses from ledger, not only that they allow to modify nested structure.

Moreover. Both Tx and TxBody are nested structures themselves for every era. We historically provided pattern synonyms with record fields, that allowed us to pretend like they are not nested structures. But this is a maintenance burden and it does not help us with avoiding duplication. Lenses do. For example, I'd love to see an alternative approach that will let you compute a total output of any transaction in any era:

> let txTotalAda tx = fold $ fmap (\txOut -> txOut ^. coinTxOutL) (tx ^. bodyTxL . outputsTxBodyL)
> :t txTotalAda
txTotalAda :: EraTx era => Tx era -> Coin

Same applies to PParams. For example here is a lens that will let you set and get protocol version from PParams for any era:

> :t ppProtocolVersionL 
ppProtocolVersionL :: EraPParams era => Lens' (PParams era) ProtVer

If a field was added at a later era then it will be guarded by a specific type class that allows using the lens from specific era onwards, eg ppCostModelsL can be used in Alonzo, Babbage..., but not with PParams from eras prior to Alonzo:

> :t ppCostModelsL 
ppCostModelsL :: AlonzoEraPParams era => Lens' (PParams era) CostModels

Before that we were using PParams with the hacky getField @"_protocolVersion" pp approach, but it was too ugly and limited in functionality. It only allowed for accessing, not modifying. And with respect to other types, like Tx, TxBody, TxOut we had no way to deal in an era agnostic fashion.

Long story short, lenses are the new designated way to deal with ledger types that can change across eras.

There are many good refactors here +1 . We should definitely organize a call next week so you can take us through the changes.

Thank you. Sounds good with me.

[newhoggy]

FWIW, I think lenses are way easier than type families provided the lens API is discoverable.

In your example, how easy is it to find all the accessors for the expression:tx ^. bodyTxL?

For example you have outputsTxBodyL. What other fields are there?

[lehins]

In your example, how easy is it to find all the accessors for the expression: tx ^. bodyTxL?
For example you have outputsTxBodyL. What other fields are there?

Those are great questions! We have a proper cardano-ledger-api in the works. For now we export all of the lenses, but there will be a lot more exported later with proper documentation and examples. For instance here are all the lenses that are available for TxBody: https://github.com/input-output-hk/cardano-ledger/blob/302e46f4d83e5ad8dafec68130ea5b3c07942f52/libs/cardano-ledger-api/src/Cardano/Ledger/Api/Tx/Body.hs#L12-L56