Add Support for BlockchainTests

README.md

@@ -1,19 +1,36 @@
-# Testing Tools
+# Execution Spec Tests
 
 This repository provides tools and libraries for generating cross-client
 Ethereum tests.
 
 ## Quick Start
 
-Relies on Python `3.10.0`, `geth` `v1.10.13`, `solc` `v0.8.5` or later. 
+Relies on Python `3.10.0`, `geth` `v1.10.13`, `solc` `v0.8.17` or later. 
 
 ```console
-$ git clone https://github.com/lightclient/testing-tools
-$ cd testing-tools
+$ git clone https://github.com/ethereum/execution-spec-tests
+$ cd execution-spec-tests
 $ pip install -e .
 $ tf --output="fixtures"
 ```
 
+It is recommended to use a virtual environment to run the tests:
+```console
+$ git clone https://github.com/ethereum/execution-spec-tests
+$ cd execution-spec-tests
+$ python -m venv ./venv/
+$ source ./venv/bin/activate
+$ pip install -e .
+$ tf --output="fixtures"
+```
+
+Go-ethereum's `evm` command must be accessible in the `PATH`
+to be able to successfully produce the tests. See
+https://github.com/ethereum/go-ethereum#building-the-source for information on
+how to build go-ethereum utilities.
+
+`solc` compiler must also be accessible in the `PATH`.
+
 ## Overview 
 
 ### `ethereum_test`
@@ -42,29 +59,60 @@ Contains all the Ethereum consensus tests available in this repository.
 
 ## Writing Tests
 
+### Purpose of test specs in this repository
+
+The goal of the test specs included in this repository is to generate test vectors that can be consumed by any Execution client, and to verify that all of the clients agree on the same output after executing each test.
+
+Consensus is the most important aspect of any blockchain network, therefore, anything that modifies the state of the blockchain must be tested by at least one test in this repository.
+
+The tests focus on the EVM execution, therefore before being able to properly write a test, it is important to understand what the Ethereum Virtual Machine is and how it works.
+
+
+### Types of tests
+
+At the moment there are only two types of tests that can be produced by each test spec:
+
+- State Tests
+- Blockchain Tests
+
+The State tests span a single block and, ideally, a single transaction.
+
+Examples of State tests:
+
+- Test a single opcode behavior
+- Verify opcode gas costs
+- Test interactions between multiple smart contracts
+- Test creation of smart contracts
+
+The Blockchain tests span multiple blocks which can contain or not transactions, and mainly focus on the block to block effects to the Ethereum state.
+
+- Verify system-level operations such as coinbase balance updates or withdrawals
+- Verify fork transitions
+
 ### Adding a New Test
 
 All currently implemented tests can be found in the `src/ethereum_tests`
 directory, which is composed of many subdirectories, and each one represents a
 different test category.
 
-Source files included in each category contain one or multiple test functions,
-and each can in turn create one or many test vectors.
+Source files included in each category contain one or multiple test specs
+represented as python functions, and each can in turn produce one or many test
+vectors.
 
 A new test can be added by either:
 
-- Adding a new `test_` function to an existing file in any of the existing
-  category subdirectories within `src/ethereum_tests`.
+- Adding a new `test_` python function to an existing file in any of the
+  existing category subdirectories within `src/ethereum_tests`.
 - Creating a new source file in an existing category, and populating it with
   the new test function(s).
 - Creating an entirely new category by adding a subdirectory in
   `src/ethereum_tests` with the appropriate source files and test functions.
 
-### Test Generator Functions
+### Test Spec Generator Functions
 
-Every test function is a generator which can perform a single or multiple
-`yield` operations during its runtime to each time yield a single `StateTest` 
-object.
+Every test spec is a python generator function which can perform a single or
+multiple `yield` operations during its runtime to each time yield a single
+`StateTest`/`BlockchainTest` object.
 
 The test vector's generator function _must_ be decorated by only one of the
 following decorators:
@@ -93,6 +141,17 @@ following attributes:
     created or modified after all transactions are executed.
 - txs: All transactions to be executed during the test vector runtime.
 
+### `BlockchainTest` Object
+
+The `BlockchainTest` object represents a single test vector that evaluates the
+Ethereum VM by attempting to append multiple blocks to the chain:
+
+- pre: Pre-State containing the information of all Ethereum accounts that exist
+    before any block is executed.
+- post: Post-State containing the information of all Ethereum accounts that are
+    created or modified after all blocks are executed.
+- blocks: All blocks to be appended to the blockchain during the test.
+
 
 ### Pre/Post State of the Test
 

setup.cfg

@@ -48,7 +48,7 @@ test =
 
 lint =
     isort>=5.8,<6
-    mypy==0.910; implementation_name == "cpython"
+    mypy==0.982; implementation_name == "cpython"
     black==22.3.0; implementation_name == "cpython"
     flake8-spellcheck>=0.24,<0.25
     flake8-docstrings>=1.6,<2

src/ethereum_test/__init__.py

@@ -2,17 +2,27 @@
 Library for generating cross-client Ethereum tests.
 """
 
+from .blockchain_test import BlockchainTest
 from .code import Code
 from .common import TestAddress
 from .decorators import test_from, test_only
-from .fill import fill_state_test
+from .fill import fill_test
 from .helpers import to_address
 from .state_test import StateTest
-from .types import Account, Environment, JSONEncoder, Storage, Transaction
+from .types import (
+    Account,
+    Block,
+    Environment,
+    JSONEncoder,
+    Storage,
+    Transaction,
+)
 from .yul import Yul
 
 __all__ = (
     "Account",
+    "Block",
+    "BlockchainTest",
     "Code",
     "Environment",
     "JSONEncoder",
@@ -21,7 +31,7 @@
     "TestAddress",
     "Transaction",
     "Yul",
-    "fill_state_test",
+    "fill_test",
     "test_from",
     "test_only",
     "to_address",

src/ethereum_test/base_test.py

@@ -0,0 +1,110 @@
+"""
+Generic Ethereum test base class
+"""
+from abc import abstractmethod
+from typing import Any, Callable, Dict, Generator, List, Mapping, Tuple, Union
+
+from evm_block_builder import BlockBuilder
+from evm_transition_tool import TransitionTool
+
+from .types import Account, FixtureBlock, FixtureHeader, Transaction
+
+
+def normalize_address(address: str) -> str:
+    """
+    Normalizes an address to be able to look it up in the alloc that is
+    produced by the transition tool.
+    """
+    address = address.lower()
+    if address.startswith("0x"):
+        address = address[2:]
+    address.rjust(40, "0")
+    if len(address) > 40:
+        raise Exception("invalid address")
+
+    return "0x" + address
+
+
+def verify_transactions(
+    txs: Union[List[Transaction], None], result
+) -> List[int]:
+    """
+    Verify rejected transactions (if any) against the expected outcome.
+    Raises exception on unexpected rejections or unexpected successful txs.
+    """
+    rejected_txs: Dict[int, Any] = {}
+    if "rejected" in result:
+        for rejected_tx in result["rejected"]:
+            if "index" not in rejected_tx or "error" not in rejected_tx:
+                raise Exception("badly formatted result")
+            rejected_txs[rejected_tx["index"]] = rejected_tx["error"]
+
+    if txs is not None:
+        for i, tx in enumerate(txs):
+            error = rejected_txs[i] if i in rejected_txs else None
+            if tx.error and not error:
+                raise Exception("tx expected to fail succeeded")
+            elif not tx.error and error:
+                raise Exception(f"tx unexpectedly failed: {error}")
+
+            # TODO: Also we need a way to check we actually got the
+            # correct error
+    return list(rejected_txs.keys())
+
+
+def verify_post_alloc(post: Mapping[str, Account], alloc: Mapping[str, Any]):
+    """
+    Verify that an allocation matches the expected post in the test.
+    Raises exception on unexpected values.
+    """
+    for address, account in post.items():
+        address = normalize_address(address)
+        if account is None:
+            # If an account is None in post, it must not exist in the
+            # alloc.
+            if address in alloc:
+                raise Exception(f"found unexpected account: {address}")
+        else:
+            if address in alloc:
+                account.check_alloc(address, alloc[address])
+            else:
+                raise Exception(f"expected account not found: {address}")
+
+
+class BaseTest:
+    """
+    Represents a base Ethereum test which must return a genesis and a
+    blockchain.
+    """
+
+    pre: Mapping[str, Account]
+
+    @abstractmethod
+    def make_genesis(
+        self,
+        b11r: BlockBuilder,
+        t8n: TransitionTool,
+        fork: str,
+    ) -> FixtureHeader:
+        """
+        Create a genesis block from the test definition.
+        """
+        pass
+
+    @abstractmethod
+    def make_blocks(
+        self,
+        b11r: BlockBuilder,
+        t8n: TransitionTool,
+        genesis: FixtureHeader,
+        fork: str,
+        chain_id: int = 1,
+        reward: int = 0,
+    ) -> Tuple[List[FixtureBlock], str]:
+        """
+        Generate the blockchain that must be executed sequentially during test.
+        """
+        pass
+
+
+TestSpec = Callable[[str], Generator[BaseTest, None, None]]