CIP for ledger evolution

CIP-ledger-evolution/README.md

@@ -0,0 +1,302 @@
+---
+CIP: XXXX
+Title: Cardano Ledger Evolution
+Authors: Jared Corduan <jared.corduan@iohk.io>
+Comments-Summary: No comments
+Comments-URI:
+Status: Active
+Type: Process
+Created: 2022-12-XX
+License: CC-BY-4.0
+---
+
+# Evolution of the Cardano ledger
+
+## Abstract
+
+This CIP proposes a process for proposing changes to the Cardano ledger.
+
+## Motivation
+
+The ledger is responsible for processing transactions and updating the shared state of the network.
+It also processes block headers and handles the state transformation from one epoch to the next
+(e.g. computing the staking rewards).
+It is a very user-facing component;
+adding new functionality to the Cardano network often involves adding features to the ledger.
+
+Most of the state maintained by the ledger relates to the
+[Extended UTxO accounting model](https://iohk.io/en/research/library/papers/the-extended-utxo-model/) and
+support for [Ouroboros](https://iohk.io/en/research/library/papers/ouroboros-a-provably-secure-proof-of-stake-blockchain-protocol/),
+the proof-of-stake consensus mechanism used in Cardano.
+
+This CIP aims to give guidance for future CIPs related to the ledger,
+making it a registered category of the CIP process[^1].
+
+Many thanks Michael Peyton Jones to writing [CIP-35] and forging this path for us.
+Some of the language here has been taken directly from it.
+
+## Background
+
+### Terminology
+
+Context for the terminology used in this document is given in [CIP-59].
+
+### Specifications
+
+The ledger is specified as a state transition system using a
+[small step operational semantics](STS.md).
+We refer to this framework as the *Small Step Semantics for Cardano*, or the *STS* for short.
+An understanding of the existing STS specifications for the
+[existing ledger eras](https://github.com/input-output-hk/cardano-ledger#cardano-ledger)
+is often required to fully understand the implications of changes to the ledger
+(though an understanding of the Haskell implementation is a fair substitute).
+
+The STS framework leaves both cryptographic primitives and the serialization format abstract.
+These specifications need to be complete enough to realize a full implementation of the ledger
+given the cryptographic primitives and serialization format.
+The cryptographic primitives are described as appendices to the STS specifications,
+and the serialization format is given as a
+[CDDL file](https://www.rfc-editor.org/rfc/rfc8610).
+There SHOULD be one STS specification per ledger era
+(the Allegra ledger era is currently combined with the Mary era).
+
+From the Byron to the Babbage ledger eras, the STS frameworks were written in $\LaTeX$.
+Starting in the Conway ledger era, literate Agda will be used.
+During the transition from $\LaTeX$ to literate Agda, we will take advantage
+of the ability to substitute $\LaTeX$ in place of Agda code when needed for expedience.
+With time, the Agda specification will gloriously blur the line between specification
+and reference implementation,
+just as the type system upon which Agda is built blurs the line between proof and program.
+
+### Ledger eras
+
+A ledger era is a collection of features added to the ledger which are introduced at a hard fork.
+The existing ledger eras, with a very simplistic description is given below.
+
+|name|new features|
+| --- | --- |
+|Byron|initial UTxO ledger|
+|Shelley|decentralized block production, stake delegation|
+|Allegra|timelock scripts|
+|Mary|multi-assets|
+|Alonzo|Plutus scripts|
+|Babbage|improved Plutus script contexts|
+|Conway|constitutional committee replacement, SPO voting on hard forks|
+
+
+### Ledger state
+
+The ledger state is the source of truth from which the network derives meaning.
+For performance reasons, we require it to be the minimal state needed
+to derive this value and validate blocks.
+
+There is currently one exception to the minimality, which will be deprecated in the days ahead,
+namely the stake pool ranking information which is used by Daedalus.
+
+#### Ledger events
+
+Some provenance about the ledger calculations is provided by ledger events.
+These events come with zero cost to a running node if not used and are not stored in the ledger state.
+
+### Soft forks and Hard forks
+
+We make the following definitions:
+
+**Hard fork** - A *hard fork* is a change to the protocol (not restricted to the ledger)
+resulting in a single new (CBOR) block being valid.
+
+**Soft fork** - A *soft fork* is a change to the protocol (not restricted to the ledger)
+resulting in fewer blocks being valid.
+A good way to think about soft forks is the property that a node which only validates
+(and does not produce blocks) does not need a software upgrade in order to continue following the block chain.
+Soft forks do not seem to come up often in practice.
+
+### Serialization
+
+Transactions and blocks are serialized with
+[CBOR](https://www.rfc-editor.org/rfc/rfc7049)
+and specified with
+[CDDL file](https://www.rfc-editor.org/rfc/rfc8610).
+
+The ledger state is also currently also serialized with CBOR,
+but this may change in the future.
+
+### Plutus script context
+
+The ledger and Plutus scripts have a common interface, namely the script context.
+See [CIP-35] for more information.
+Note that CIPs relating to the script context are relevant to both the ledger and to the Plutus CIP categories.
+
+## Specification
+
+This proposal deals only with the types of change listed in
+[Types of change](#types-of-change), all others are out of scope.
+
+### Changes that require a CIP
+
+This proposal recommends that some of the changes listed in "Types of change" (specified below) should:
+
+* Be proposed in a CIP.
+* Go through additional process in addition to the usual CIP process.
+
+The additional process mostly takes the form of additional information that should be present in
+the CIP before it moves to particular stages. As such, it is up to the CIP Editors to enforce this.
+
+The requirement to propose a change via a CIP is, as all CIPs are, advisory.
+In exceptional circumstances or where swift action is required, we expect that changes may still
+be made without following this process. In such circumstances,
+a retrospective CIP SHOULD be made after the fact to record the changes and the rationale for them.
+
+Changes that require a CIP do not have to each be in an individual CIP, they can be included in batches or in other CIPs.
+So, for example, a single CIP could propose multiple new related fields in the transaction.
+
+It is very likely the case that if a change does not require a soft or hard fork,
+it does not require a CIP, except in the case of bugs.
+
+A "bug fix" is a change to behavior where:
+- The implemented behavior does not match the specification; or
+- The specified behavior is clearly wrong (in the judgment of relevant experts)
+
+#### Changes that *do not* require a CIP
+
+* performance improvements
+* bug fixes
+* ledger events
+* changes to the protocol parameter values (this is a governance issue, see [CIP-1694])
+
+### Types of change
+
+- [Ledger features](#ledger-features)
+  - [Ledger-script interface](#ledger-script-interface)
+- [Ledger intra-era switch](#ledger-intra-era-switch)
+- [Ledger protocol](#ledger-protocol)
+- [Ledger eras](#ledger-eras-1)
+- [Serialization](#serialization)
+
+#### Ledger features
+
+This is a very broad category.
+Any conceptual group of changes to the ledger semantics,
+involving a single idea or a group of closely related ideas
+(such as adding, modifying, or removing existing features)
+is a `ledger-feature`.
+This category excludes bug fixes and changes to the [ledger protocol](#ledger-protocol).
+Ledger features are assumed to be included in a future ledger era.
+
+To become active, the ledger feature must be included in an active [ledger era](#ledger-eras-1).
+
+##### Ledger-script interface
+
+Any conceptual group of changes to the script context is a `ledger-script-interface`.
+This is a shared type with Plutus, so the process in [CIP-35] MUST also be followed.
+
+To become active, the interface change must be included in an active [ledger era](#ledger-eras-1).
+
+#### Ledger intra-era switch
+
+Sometimes a ledger change is so small and targeted that it can be implemented by
+switching on the major protocol version (and not introducing a new ledger era).
+These must be used *very* sparingly, as they can quickly lead to unintended complexity and
+unclear semantics[^2].
+A ledger intra-era switch does not necessarily require a CIP;
+sometimes it might be more appropriate to just include it in a [ledger era](#ledger-eras-1) CIP.
+
+When a hard fork involves only intra-era switching, a new ledger era can be avoided and we
+refer to the hard fork as an intra-era hard fork.
+
+To become active, the ledger intra-era switch must be included in an active [ledger era](#ledger-eras-1).
+
+#### Ledger protocol
+
+The `ledger-protocol` category corresponds closely to block header validation.
+It corresponds precisely to all the functionality in the Shelley ledger specification
+not captured by the `BBODY` and `TICK` transitions.
+This mostly involves handling the nonces for Ouroboros, the VRF checks,
+the KES certificates, and block leadership checks.
+
+Historically, the ledger protocols have been specified alongside the ledger eras.
+The `TPraos` ledger protocol was specified alongside Shelley, and
+`Praos` was specified alongside Babbage.
+
+To become active, a ledger protocol change MUST be
+specified with the STS framework and MUST be included in the software update for a soft/hard fork.
+If the fork coincides with a ledger era, the specifications can be combined.
+
+#### Ledger eras
+
+A `ledger-era` is a collection of ledger-features, bug fixes, intra-era switches, and other minor changes
+grouped together with a name and activated together at a fork.
+In the event of interaction between the features, details may need to be provided.
+
+To become active, the ledger era MUST:
+* specify the semantics with the [STS](STS.md) framework
+* specify cryptographic details in an appendix to the STS specification
+* specify the CBOR serialization schema with a CDDL file
+* be included in the software update for a soft/hard fork.
+
+### Serialization changes
+
+See [Serialization](Serialization.md).
+
+### The Small Step Semantics Framework for Cardano
+
+See [STS](STS.md).
+
+### Implementing and releasing changes
+
+This CIP does not cover the process of implementing changes.
+As usual, the CIP process covers the design phase, and it is up to the implementer to ensure that
+their proposal is implemented, which may require additional work to meet the requirements of the
+maintainers of the Cardano code repositories (testing, implementation quality, approach), and so on.
+
+Changes can be released after their CIPs have reached Active status.
+Different changes will require different releases as described in "Types of release".
+This CIP does not cover the process by which changes are actually incorporated into releases after having been implemented.
+In particular, there is NO assumption that a feature which requires a particular release will be
+included in the next such release, even after it has been implemented.
+
+### Ledger Core CIP registry
+
+Any CIP which proposes a type of change listed in "Types of change" MUST also add itself to this registry (in addition to the main registry).
+
+| #  | Title             | Type of change          | Status |
+|----|-------------------|-------------------------|--------|
+| 31 | Reference inputs  | Ledger-script interface | Draft  |
+| 32 | Inline datums     | Ledger-script interface | Draft  |
+| 33 | Reference scripts | Ledger-script interface | Draft  |
+
+## Rationale
+
+### Why have a public process for changes?
+
+Cardano is continuing to move towards decentralized governance within the Voltaire phase of development.
+Historically, key development and implementation decisions have been made by the core development team.
+This was important in the earliest stages of the platform’s evolution.
+However, this becomes less so as the platform starts to mature and is neither sustainable nor desirable in the long term.
+
+Furthermore, while many changes to Cardano are obscure or not of interest to many community members,
+there is a much larger community who have a keen interest in changes to Plutus Core: dApp developers.
+Hence it is especially important to have a clear way for this community to be able to propose changes and see how they are progressing.
+
+### Why include a CIP registry?
+
+This is just to make it easy for those considering proposing a CIP following this process to see which CIPs have already been submitted.
+An alternative would be a standard title for CIPs, or perhaps some kind of CIP metadata to indicate that it follows the process in this CIP.
+
+### Out of scope
+
+* Changes that cover _using_ the ledger, but not _changing_ the ledger.
+* Changes to the node-to-client queries, which are used to get information from the ledger.
+* Changes about how clients consume the ledger state, unless it involves changes to the ledger state.
+* Unless something exceptional arises,
+  storing additional information in the ledger state that is not needed for block validation.
+
+[^1]: See [CIP-1](https://github.com/cardano-foundation/CIPs/blob/cip-cps-rework/CIP-0001/README.md#categories).
+[^2]: On an early testnet for the Mary era, the network managed to enter the Mary era while remaining
+on major protocol version 2 (the mainnet version for the Shelley era).
+The network exhibited strange behavior due to this fact, which was perplexing to understand until
+we realized the mismatch. We dubbed it the Mary Shelley's Frankenstein era.
+
+[CIP-35]: https://github.com/cardano-foundation/CIPs/tree/master/CIP-0035
+[CIP-59]: https://github.com/cardano-foundation/CIPs/tree/master/CIP-0059
+[CIP-1694]: https://github.com/cardano-foundation/CIPs/tree/master/CIP-1694