protocol upgrade activation mechanism (also implements PREACTIVATE_FEATURE and ONLY_LINK_TO_EXISTING_PERMISSION protocol features)

[arhag]

Change Description

Resolves #6429, #6431, #6437, and #6333.

This PR builds the foundations needed to enable consensus protocol upgrades (also referred to as protocol features throughout the code and documentation). These foundations consist of the following three pillars:

1. The core mechanism to recognize and allow activation (via block headers) of protocol features (Consensus protocol upgrade activation mechanism #6429). This is also accompanied by new RPC endpoints in producer_api_plugin to allow a block producer (BP) to activate a protocol feature (specifically PREACTIVATE_FEATURE which cannot be pre-activated) and another new RPC endpoint in chain_api_plugin to allow users to see which protocol features have been activated as of the current state of a node.
2. A modifiable dynamic whitelist of intrinsic names that are tracked as part of state and restrict which intrinsics a contract can link with during an eosio::setcode (Dynamic whitelist of intrinsics #6437).
3. Two new intrinsics, preactivate_feature and is_feature_activated, which become linkable (via the mechanism of pillar 2 above) on activation of the PREACTIVATE_FEATURE intrinsic, enable, respectively, pre-activating a protocol feature from within a privileged contract and checking if a particular protocol feature has been activated from within any contract (Consensus protocol upgrade to enable protocol feature pre-activation #6431).

PRs #6588 and #6624 made changes to critical blockchain data structures which made it not possible to replay with a state directory generated by v1.7.x or earlier of nodeos nor recover from a snapshot generated by those earlier versions. This PR continues to take advantage of the freedom of not being constrained by replay-ability concerns by making necessary changes to the block header state and state data structures to support the three pillars of the protocol feature foundations.

Changes to block header state

The block header state structures were modified to include a new shared pointer field providing access to the set of digests (each protocol feature is represented by a 256-bit digest) for each of the protocol features that are active as of that block. If a new block building upon another does not activate any new protocol features, the activated_protocol_features of the corresponding block_header_state of the new block will point to the same data structure as the pointed to by activated_protocol_features of the block_header_state of the prior block. However, if the new block activates new protocol features, the activated_protocol_features of its corresponding block_header_state will point to a new set which is the union of the one for the prior block and any new protocol features activated in the new block.

New protocol features are signaled for activation through a block header extension. pending_block_header_state::finish_block uses the protocol_feature_activation extracted from the block header extension to first determine whether they are considered valid (via a passed in validator function) and then add them to the protocol feature activation set. Details of this process will be discussed under the pillar 1 section below.

block_header_state also has a new header_exts field which is merely a cache of the vector of block_header_extensions extracted from a block header using the new block_header::validate_and_extract_header_extensions method.

There are three other changes added to block_header_state.cpp. One is an optimization to block_header_state::calc_dpos_last_irreversible that uses std::nth_element to get the desired element rather than unnecessarily sorting the full sequence. The other two are fixes to bugs introduced in the refactoring effort of PR #6588 that would break existing consensus if left unfixed: the hash of the pending schedule was incorrectly computed when promoted from proposed; a quirk in updating producer_to_last_produced in the existing consensus algorithm was not properly replicated in the refactor (required adding the new_producer_to_last_produced[prokey.producer_name] = result.block_num; line). The unit test producer_schedule_tests/producer_watermark_test was added in response to that second bug fix.

Changes to fork database

Additional minor changes were made to fork_database beyond PR #6624. An explicit open method was added to decouple constructing a fork_database instance from actually opening and loading the database it points to; it also better complements the existing close method. fork_database::open takes a validator function which controller_impl provides as one that utilizes the controller_impl::check_protocol_features method. For safety and clarity, I wanted to ensure the fields of controller_impl (specifically the new protocol_features field) are fully constructed prior to utilizing a validator function that calls the controller_impl::check_protocol_features method, since that method expects the protocol_features to be constructed. Adding a new block_state_ptr to fork_database using the add method will now allow (optionally) validating any new protocol features signaled for activation in the block header. The validation is enforced when opening a fork_database to ensure that any blocks retrieved from the serialized fork database file are still consistent with the protocol features locally recognized by the node.

Pillar 1: The core protocol feature activation mechanism

The original EOSIO consensus mechanism disallows having any block header extensions (i.e. having a non-empty header_extensions field). However, this field was added to the block header data structure explicitly to support adding data in there in future upgrades. The core protocol feature activation mechanism introduced in this PR takes advantage of this extension field to signal activation of protocol features.

It is acceptable that non-empty fields containing protocol feature activations will rejected by nodeos v1.7.x and earlier since the activation of a protocol feature not supported by an older version of nodeos is intended to be rejected. Old nodes will be unable to follow the blockchain branch utilizing the new features. With a two-thirds supermajority of BPs upgraded to support the activation of the first protocol feature, any old nodes will remain stuck on a part of the blockchain that cannot advance the last irreversible block until they upgrade their nodes to support the protocol feature(s).

However, there are strict requirements for what the data in header_extensions must look like to be accepted as a signal to activate protocol features. This was described in issue #6429 with the phrase "well-formed protocol feature activation". These rules have been implemented in the code through the new protocol_feature_activation type and the validation rules in block_header::validate_and_extract_header_extensions which actually deserializes the data in header_extensions into a more useable form: vector<block_header_extensions> where block_header_extensions is a static_variant of the supported forms of block header extensions (currently only protocol_feature_activation).

The discussion of block header state changes above discussed how the set of activated protocol features is tracked per block_header_state (which are managed as a tree of possible blockchain branches to switch to in fork_database). Some validation also occurs at the stage of creating a new block_header_state. For example, the validation guarantees that no protocol features are repeated. Furthermore, by passing in an appropriate validator function into the constructor of block_state, it is also able to validate (via the controller_impl::check_protocol_features method which the validator function calls) that all of the new protocol features to activate satisfy the following: are recognized by the software; are subjectively enabled; are ready for activation according to the subjectively set earliest_allowed_activation_time for that protocol feature; and, all of their dependencies have either already been activated or are earlier in the activation sequence for the current block.

These validations are also done through the other entry point of getting controller to process a new block: controller::start_block used by the producer_plugin (or tester). These checks are also necessary for blocks that were already validated before (e.g. when replaying blocks or opening an existing fork database from disk) because nodeos have started up with a new configuration of supported protocol features that is more restrictive than its configuration when these blocks were first validated and stored to disk.

However, any state manipulations by the activated protocol features, and any further validation that depends on state must wait until controller_impl::start_block. In controller_impl::start_block, the new protocol features to activate are traversed in the order specified in the block header and for each: if the protocol feature is subjectively configured to require pre-activation, assert that the feature has already been pre-activated (see pillar 3 for details); run the trigger function (if one exists) for the protocol feature under the context that only new protocol features traversed prior to this point are considered activated; and then, consider that new protocol feature activated by adding it to the activated_protocol_features vector which is a field of the new protocol_state_object singleton added to the chain state (also activate the feature in protocol_feature_manager which is discussed in detail below).

At the end of traversing through the new protocol features, if any pre-activated features exist, the code ensures that all were activated in that block, otherwise it reject the block. It also then clears the existing pre-activated features since they were all activated. Again, for more information on pre-activation, see the section on pillar 3.

So far there has been no mention of what these protocol features actually are. At a high-level, controller just considers a protocol feature to be some unique thing (uniquely identified by a digest) that can be activated (at most once) assuming its validation requirements (discussed above) are met. Some of those validation requirements, like dependencies, are objective and are actually committed to by the digest that refers to the protocol feature. Other validation requirements are subjective and locally determined by the configuration of the node (the only three subjective validation requirements thus far are whether the protocol feature is enabled, whether it require pre-activation; and the earliest allowed time, as measured by the block timestamp, it can be activated); these subjective requirements are not included in the calculation of the digest. These protocol features may optionally trigger some code to run at the start of the block (prior to processing the onblock transaction). However, in many cases the change in behavior of the code occurs elsewhere simply by checking whether a particular protocol feature has been activated or not, and conditionally adjusting behavior as appropriate.

Issue #6429 discusses different classes of protocol features that could be supported by the design. Currently, the implementation only supports the builtin class of protocol features, which are ones associated with a hard-coded codename which the nodeos code is written to understand how to handle through custom trigger functions and/or referencing whether the particular builtin protocol feature is activated from elsewhere in the code. (Note that a builtin codename cannot be repeated across multiple protocol features.) However, this framework does support adding future classes of protocol features without too much difficulty. The specification describing what the protocol feature does would be committed to in the digest representing the protocol feature. A generic trigger function applicable to all protocol features of a specific class would read the specification of the protocol feature and adjust state appropriately.

Nodeos is designed to populate the config/protocol_features directory with JSON files specifying the default configuration of any builtin protocol features that version of nodeos understands but is not represented by an existing JSON file in that directory. With sensible defaults (plus the pre-activation featured discussed in the pillar 3 section), this enables users to not need to change the configuration files of any protocol features in typical scenarios. However, a community can choose, if they so desire, to customize one or more of the supported protocol features for their EOSIO blockchain to a certain degree without changing code. For example, the feature_digest or dependencies could be changed (which would affect the digest representing the protocol feature). More typically, the subjective_restrictions might be adjusted to control if and when a protocol feature can be activated. However, with pre-activate feature functionality enabled, there is unlikely to be any need to modify the JSON files generated in the config/protocol_features directory.

The only real exception with the current state of the code is that the block producers will likely choose to modify the earliest_allowed_activation_time subjective restriction of the PREACTIVATE_FEATURE protocol feature to have better control over when the first protocol feature is activated. But due to the nature of earliest_allowed_activation_time, other users can keep the default permissive value without breaking away from consensus when PREACTIVATE_FEATURE is activated. More exceptions will likely come into play in the future if other classes of protocol features (beyond builtin) are added. In a case of a future community-initiated protocol feature that does not require any code changes to nodeos, users would have to manually include the appropriately configured JSON file into config/protocol_features (since nodeos would not know how to create such a thing) in order to configure their local nodeos to support the protocol feature and not break away from consensus when it is activated.

A new accessor is_protocol_feature_activated is added to controller to allow any part of the code (either internal or external to controller) to determine if a particular protocol feature (referenced by digest) is activated as of the current state the controller is in. However, a more efficient accessor has been added to check if a builtin protocol feature is activated. Since critical parts of the code will be checking if certain builtin protocol features are activated to adjust behavior appropriately, it is important to ensure that this accessor is fast. So controller::is_builtin_activated is added in this PR to enable fast constant-time lookups (by codename rather than digest) of whether a particular builtin protocol feature is activated or not. That function is actually a relatively simple wrapper over a method is_builtin_activated in the newly added protocol_feature_manager class.

The protocol_feature_manager instance is included as a field in controller_impl. It is constructed with a protocol_feature_set which wrapper over a set of recognized protocol_features (with an accompanying vector index to speed up lookups of builtin protocol features by their codename) that only allows mutation through its add_feature method. A protocol_feature type holds the objective specifications of a protocol feature as well as the subjective restrictions in a convenient format for internal use.

The protocol_feature_set is passed into one of the constructors of controller (the other constructor just default constructs protocol_feature_set) and so it is created by chain_plugin during initialization (or tester in the case of unit tests) by instantiating a default protocol_feature_set and then adding recognized protocol features through the add_feature method. During initialization of chain_plugin, it goes through a series of steps in the initialize_protocol_features function to read the configuration/specification of protocol features specified in the config/protocol_features and add them to the protocol_feature_set (in the right order to respect any dependency ordering as the add_feature method requires that) before then iterating through builtin protocol features recognized by nodeos that were not reflected in the config/protocol_features directory and adding them with their default configuration (again in right order) to the protocol_feature_set while also (by default) writing out the configuration/specification JSON for those missing builtin protocol features to the config/protocol_features directory.

Once a protocol_feature_manager is instantiated with a protocol_feature_set it will not allow that protocol_feature_set to be mutated. (Note this preserves the requirements controller_impl::check_protocol_features, which uses protocol_feature_manager, needs to remain thread-safe.) It allows read-only access to the wrapped protocol_feature_set instance. The main goal of protocol_feature_manager is to track the mutating state of which of the recognized protocol features (included with their specifications in the wrapped protocol_feature_set instance) are actually activated at the moment. To enable this it has two mutators: activate_feature which is called from controller_impl::start_block when a protocol feature is activated; and, popped_blocks_to which is called when a pending block is aborted/rejected or prior applied blocks are popped during a fork switch so that controller ensures the state of protocol_feature_manager remains consistent with the protocol features considered activated according to the chain state (again see the activated_protocol_features vector of the protocol_state_object singleton). The state information of activation is duplicated and kept in sync in this manner purely to make it very quick to lookup whether a built protocol feature has been activated. There is another non-const method in protocol_feature_manager: init. Again to guarantee synchronization of the activation state, the protocol_feature_manager::init method is called from controller_impl::init on nodeos startup to ensure the protocol_feature_manager is initialized to reflect the state of protocol feature activations as determined by the activated_protocol_features stored in the chain state Chainbase database.

Pillar 2: Dynamic whitelist of intrinsics

The new protocol_state_object singleton in the chain state also includes a field called whitelisted_intrinsics that represents the set of names for the intrinsic against which a contract deployed with eosio::setcode is allowed to link.

The type of whitelisted_intrinsics (referenced by type alias whitelisted_intrinsics_type) is actually a multimap (specifically a boost::interprocess::flat_multimap<uint64_t, shared_string>) for performance reasons. While a boost::interprocess::set<shared_string> would work, it would require contract validation to do log(N) string comparisons (where N is the number of intrinsics in the whitelist) for each of the M unique intrinsics used in a contract that is being set each time eosio::setcode is called. So it is important to improve the performance of this validation step. One major gain is significantly reducing the number of string comparisons necessary. By hashing the string to a uint64_t value, the log(N) comparison become fast integer comparisons, and since there should be a low likelihood of collision, it is unlikely that more than one string comparison would be necessary in practice. Furthermore, since the multimap is in a flat structure (where the actual intrinsic name strings are not stored in the contiguous memory region of the multimap unless very short) it improves the cache properties of the lookup process.

While whitelisted_intrinsics within protocol_state_object is of type boost::interprocess::flat_multimap<uint64_t, shared_string>, it is not serialized that way in the portable snapshots. It is converted into a std::set<std::string> (which is what it logically represents) when writing it out to the portable snapshot (and vice versa when reading from portable snapshot) so that the portable snapshot always contains the list of whitelisted intrinsics in the deterministic lexicographical order (and so that the redundant and implementation-specific uint64_t hash is not stored in the portable snapshot).

Three helper functions (see whitelisted_intrinsics.hpp and whitelisted_intrinsics.cpp) that accept a reference to a whitelisted_intrinsics_type enable: checking if a particular intrinsic name is in the whitelist (is_intrinsic_whitelisted); and, adding (add_intrinsic_to_whitelist) or (remove_intrinsic_from_whitelist) a intrinsic name to/from the whitelist. Two additional helper functions (again see same files mentioned before) allow converting to and from the std::set<std::string> type: reset_intrinsic_whitelist allows converting from the std::set<std::string> type into the whitelisted_intrinsics_type and convert_intrinsic_whitelist_to_set allows converting from the whitelisted_intrinsics_type to the std::set<std::string>.

Checking whether an intrinsic name exists in the current whitelist is used by the eosio::chain::webassembly::common::root_resolver::resolve (see wasm_interface.hpp) to validate that a new contract to be set does not try to use intrinsics that it should not have access to at the current time (specifically, intrinsics recognized by the nodeos software but not yet enabled via a corresponding protocol feature activation). Note that this validation is only done on a eosio::setcode (which is when eosio::chain::wasm_interface::validate is called). That validation is not done when intrinsics simply need to be resolved when instantiating the WASM cache of an already set contract; in that case the default constructor of resolver is called (see wavm_runtime::instantiate_module in wavm.cpp) setting up the instance to skip validation in the resolve method.

Adding intrinsics to the whitelist is what the trigger function of a protocol feature must do in order for the protocol feature to enable those intrinsics. Removing intrinsics from the whitelist is also allowed in the trigger function of a protocol feature, but it only affects future eosio::setcode actions after the protocol feature is activated. Existing contracts that use the "removed" intrinsics are effectively grandfathered in, as long as their code is never updated.

When initializing the chain state to the genesis state (see controller_impl::initialize_database), the protocol_state_object is constructed with the whitelisted_intrinsics field set up with the initial set of intrinsics supported in the original EOSIO consensus protocol (called the "genesis intrinsics"). The genesis intrinsics are stored in the genesis_intrinsics global variable of type const std::vector<const char*> (see genesis_intrinsics.cpp).

Pillar 3: Pre-activation of protocol features

The third pillar is actually a protocol feature (codename: PREACTIVATE_FEATURE) made possible using the prior pillars. However, it is a requirement for the way all other protocol features are intended to be activated, so it is considered a foundational feature to the protocol feature activation mechanism.

Pre-activation of a protocol feature in block N forces the activation of the protocol feature in block N+1. Furthermore, the subjective restrictions of a protocol feature can be configured to require pre-activation. So, a protocol feature can be setup to only allow activation by a privileged contract exactly when desired. This allows switching from a subjective policy by BPs of when to activate a protocol feature and making it an objective on-chain policy controlled by, for example, the eosio account. Typically the eosio account would still be controlled by a two-thirds supermajority of the active BPs, but now because of pre-activation, it might require an on-chain multisig to trigger that pre-activation and thus subsequent activation of the protocol feature.

The implementation ensures that a protocol feature that is configured locally to require pre-activation cannot be activated in controller_impl::start_block if that feature is not referenced in the new preactivated_protocol_features vector of the protocol_state_object singleton. This prevents an eager BP from activating a protocol feature (that requires pre-activation) sooner than it was intended to be by the objective on-chain policy. Also, by asserting that all the protocol features referenced in the preactivated_protocol_features vector were activated and then clearing preactivated_protocol_features at the start of the block, the implementation ensures that BPs cannot arbitrarily delay the activation of a pre-activated protocol feature.

It is allowed to pre-activate a protocol feature that does not require pre-activation. However, in this case, the protocol rules do still require that the pre-activated protocol feature is activated in the next block.

The code implementing the pre-activation of a new protocol feature is primarily in controller::preactivate_feature.

Nodes should not accept a transaction that attempts to pre-activates a protocol feature that it would not accept the activation of in the next block. Obviously this means the node should reject a transaction that attempts to pre-activate a protocol feature that it does not recognize or that it recognizes but is configured to be disabled. But it also means that if it is too early to activate a recognized and enabled protocol feature (because of the earliest_allowed_activation_time parameter) in the next block, then the node should not accept the transaction. Since it is impossible to know what the timestamp of the next block will be, the rules simply require that the current block's timestamp is sufficient to meet the time requirements of the protocol feature. So even if the next block's timestamp is exactly the earliest_allowed_activation_time of the protocol feature, the attempt to pre-activate that protocol feature will fail in the current block since the current block's timestamp is strictly less than the earliest_allowed_activation_time.

Notice that the decisions to reject the pre-activation attempt described in the paragraph above were made based on subjective information: the locally configured subjective_restrictions values, such as enabled or earliest_allowed_activation_time; or whether the node even recognized the protocol feature. Care must be taken when making processing decisions for a block or transaction based on subjective information to avoid breaking consensus. If the local node is processing a block containing a transaction that attempts a pre-activation that the local node cannot accept according to its local configuration rules, the node is forced to reject the block. So, if the node cannot accept the pre-activation due to the reasons above, it throws a protocol_feature_bad_block_exception which forces it to propagate all the way up until it rejects the block; it does not allow it to get caught and re-direct processing to an error handler of a deferred transaction, for example. However, when producing a block, we do not want the transaction author to be able to abort the speculative block being produced (even if it requires a privileged contract which presumably would require special trusted authorizations prior to getting to the point of the preactivate_feature intrinsic call). So, when producing a block, a subjective_block_production_exception is thrown instead to just reject the transaction and prevent it from being included in the block in the first place. Using a subjective exception type is critical here because of deferred transactions: throwing a regular exception would have lead to the deferred transaction retiring as either soft_fail or hard_fail which is an outcome that would be based on subjective local information and therefore not necessarily repeatable on other nodes (with other subjective configurations) replaying the blockchain.

However, once the subjective assertions mentioned above pass in the controller::preactivate_feature implementation, the rest of the function can throw regular exceptions on failure because at that point it is safely in an objective context. All nodes that reach that point and are also not referring to the subjective_restrictions of protocol features should be working with the same deterministic data and therefore have the same deterministic results (whether in the form of failure or success). So, for example, if the code detects that the feature to pre-activate is already in the preactivated_protocol_features vector of the protocol_state_object singleton, it will throw a regular exception complaining about the attempt to pre-activate an already pre-activated protocol feature. If all the checks pass, the code will then add the digest of the protocol feature to pre-activate to the end of the preactivated_protocol_features vector.

A helpful accessor get_preactivated_protocol_features is also added to controller which is used by the producer_plugin (and tester in the case of unit tests) to get the sequence of protocol feature digests that it must include in the header extensions of the next block produced to ensure the block is accepted. However, the producer_plugin is free to inject the digests of other ready-to-activate protocol features into the sequence. The current implementation in this PR takes any digests in an internally tracked _protocol_features_to_activate vector and prepends them to the sequence obtained from get_preactivated_protocol_features before then including that combined sequence into the header extensions of the next block it creates. The _protocol_features_to_activate vector can be modified with the new schedule_protocol_feature_activations RPC endpoint added to the producer_api_plugin (read more about it under the API changes section).

Implemented ONLY_LINK_TO_EXISTING_PERMISSION protocol feature

In order to facilitate tests of the pre-activate functionality, another protocol feature needed to be implemented. The ONLY_LINK_TO_EXISTING_PERMISSION protocol feature was a relatively simple one that was already planned for eventual implementation.

So, this PR implements the ONLY_LINK_TO_EXISTING_PERMISSION protocol feature, which when activated ensures that a eosio::linkauth action cannot be used to link an action to a non-existent permission (see #6333 for details). The behavior of eosio::linkauth of course remains exactly the same prior to ONLY_LINK_TO_EXISTING_PERMISSION activation in order to not break consensus.

Changes to test framework and new tests

To facilitate testing the new protocol feature activation functionality, changes have been made to both the unit testing framework (tester) and the integration testing framework (the Python scripts in the tests directory).

Furthermore, the eosio.bios contract was upgraded to support a new active preactivate which enables an outside client, such as a testing script, to actually pre-activate a particular protocol feature. The eosio.bios contract with the modified code was compiled (see code changes that were used here and its compiled WASM and ABI replaced the ones in the unittests/contracts/eosio.bios directory. The compiled WASM and ABI of the version of the bios contract without the preactivate action addition (specifically the one from v1.6.0-rc3 of eosio.contracts) were also added to to the unittests/contracts/old_versions/1.6.0-rc3/eosio.bios directory. This old version (as in prior to the preactive addition) of the eosio.bios contract is necessary to keep around because some tests will need a bios contract around in an environment where the PREACTIVATE_FEATURE protocol feature has not yet been activated. But the new eosio.bios contract with the preactivate action requires linking to intrinsics (specifically preactivate_feature and is_feature_activated) which are not allowed unless PREACTIVATE_FEATURE has already been activated.

tester has been modified to support different policies for how to initialize the fresh blockchain it creates on init. One of the overloads of base_tester::init which used to take a push_genesis boolean (and one of the constructors of tester which used to optionally take the push_genesis boolean) now take a new enum class called setup_policy instead of push_genesis. The setup_policy enum can take one of the following values:

• none: nothing is done after starting up the chain (the same as what happened before when push_genesis was false);
• old_bios_only: set the old eosio.bios contract (prior to the preactivate action addition) on the eosio account after starting up the chain (similar to the behavior before when push_genesis was true);
• preactivate_feature_only: after starting up the chain, create a single block that activates PREACTIVATE_FEATURE and do nothing else;
• preactivate_feature_and_new_bios: after starting up the chain, create a single block that activates PREACTIVATE_FEATURE and then set the new eosio.bios contract (with the new preactivate action) on the eosio account.
• full: do all of the steps of the preactivate_feature_and_new_bios setup policy but then also pre-activate all the support builtin protocol features before then producing another block so that they can all become activated.

The default value for the setup_policy field for both the init method overload and the tester constructor is setup_policy::full. Furthermore, the the constructor of validating_tester which takes one optional argument calls init. This means that a default constructed tester and default constructed validating_tester (which covers most of the existing unit tests) will automatically choose the full policy and thus now operate in an environment where all builtin protocol features have been activated. If any tests need to start running in an environment where some protocol features are not activated (for example to test those protocol features), then they will need to start tester with an alternate setup_policy.

Additional methods have been added to base_tester to facilitate testing the protocol features. These include:

• schedule_protocol_features_wo_preactivation: takes a vector of protocol digests to include in the activations in the next block produced;
• preactivate_protocol_features: pushes the eosio::preactivate action for each protocol feature digest in the vector passed into the method;
• preactivate_all_builtin_protocol_features: a convenience method to automatically pre-activate all the support builtin protocol features.

Seven new unit tests were added to the new protocol_feature_tests test suite (see unittests/protocol_feature_tests.cpp):

• activate_preactivate_feature: Tests that the PREACTIVATE_FEATURE protocol feature can be properly activated, and that the new bios contract can only be deployed after PREACTIVATE_FEATURE is activated.
• activate_and_restart: Ensures that the fact that protocol feature was activated persists after a restart.
• double_preactivation: Verifies that double pre-activation (in the same block) is not allowed.
• double_activation: Verifies that signaling activation of the same protocol feature twice within a block is not allowed. It also verifies that pre-activating another protocol feature (ONLY_LINK_TO_EXISTING_PERMISSION in this case) done the proper way will work fine.
• require_preactivation_test: Verifies that it is not possible to activate a protocol feature that requires pre-activation (ONLY_LINK_TO_EXISTING_PERMISSION in this case) without first going through pre-activation.
• only_link_to_existing_permission_test: Tests the behavior related to the ONLY_LINK_TO_EXISTING_PERMISSION protocol feature for correctness both before and after activation of ONLY_LINK_TO_EXISTING_PERMISSION.
• subjective_restrictions_test: Tests that the subjective_restrictions of a protocol feature (specifically the enabled and earliest_allowed_activation_time parameters) are properly enforced for both a protocol feature that does not require pre-activation (PREACTIVATE_FEATURE) and a protocol feature that does require pre-activation (ONLY_LINK_TO_EXISTING_PERMISSION).

The Python testing scripts have also been modified to support different setup policies. In this case, it is a little more simplified:

• NONE: do nothing like the old behavior (on top of this the old eosio.bios contract can be deployed through the regular bootstrapping processes);
• PREACTIVATE_FEATURE_ONLY: produce a block which activates the PREACTIVATE_FEATURE protocol feature after starting up the blockchain (again a eosio.bios contract can then be deployed on top of this);
• FULL: do the same thing as PREACTIVATE_FEATURE_ONLY but then during the bootstrapping process, after the new eosio.bios contract has been deployed, pre-activate all the recognized builtin protocol features and ensure a block is produced so that they can all be activated.

Note that when the setup policy is FULL, the pre-activation of builtin protocol features will not occur if the test does not go through the bootstrapping process (for example if dontBootstrap is true). Furthermore, when the setup policy is PREACTIVATE_FEATURE_ONLY or FULL, the eosio.bios contract which is deployed during the bootstrapping process is the new one with the preactivate action. However, if the setup policy is NONE, the bootstrapping process (if it occurs) will use the old bios contract that does not have the preactivate action.

tests/Node.py also adds some convenience methods to schedule protocol features to be activated, pre-activate specific protocol features (or all the builtin ones), and get the protocol features supported by the local node through the new get_supported_protocol_features RPC endpoint of the producer_api_plugin.

Two new smoke tests were added:

• tests/prod_preactivation_test.py: Simulates activating the PREACTIVATE_FEATURE protocol feature in a two producer blockchain test.
• tests/nodeos_protocol_feature_test.py: Verifies that changes to the JSON files in config/protocol_features are properly read by nodeos on startup.

Consensus Changes

Yes... definitely. Read above for details.

In addition to the protocol feature foundations (include the PREACTIVATE_FEATURE protocol feature), this PR also includes an additional protocol feature: ONLY_LINK_TO_EXISTING_PERMISSION (#6333).

API Changes

The producer_api_plugin has added three new RPC endpoints:

• get_supported_protocol_features: Returns the protocol features recognized by the local nodeos.
  • Optional boolean parameter exclude_disabled (defaults to false) determines whether to filter out any protocol features that are not enabled.
  • Optional boolean parameter exclude_unactivatable (defaults to false) determines whether to filter out any protocol features that could not be activated at the present time (either because they are disabled or because the earliest_allowed_activation_time for the protocol feature has not yet been reached).
  • Returns a vector of JSON objects containing the following fields:
    • feature_digest: Hexadecimal string representation of the digest of the protocol feature.
    • subjective_restrictions: A JSON object containing the following fields:
      • enabled: A boolean that specifies whether the protocol feature is enabled.
      • preactivation_required: A boolean that specifies whether the protocol feature require pre-activation before it can be activated.
      • earliest_allowed_activation_time: The earliest block timestamp at which the protocol feature can be pre-activated or activated.
    • description_digest: Hexadecimal string representation of the digest of the textual description for the protocol feature.
    • dependencies: An array of hexadecimal strings representing the digests of any protocol features this protocol feature depends on (can be an empty array and usually would be).
    • protocol_feature_type: A string representing the class of protocol features. Currently only the builtin class is supported which would be represented here with the string "builtin".
    • specification: An array of JSON objects each containing a string-valued field called name and any-valued field called value which provide the specification for the protocol feature. In the case of builtin protocol features, there should be only one JSON object in the array that has a name field with value "builtin_feature_codename" and a value field with value equal to a string representing the codename of the builtin protocol feature.
• schedule_protocol_feature_activations: Sets the vector of protocol feature digests to activate for the next time the local node can produce a block.
  • Takes one parameter protocol_features_to_activate which is the vector of hexadecimal strings representing the digests of protocol features to activate (in the specified order).
  • Returns nothing.
• get_scheduled_protocol_feature_activations: Returns the vector of protocol feature digests that are currently scheduled to be activated the next time the local node can produce a block. This is the vector that was last set by schedule_protocol_feature_activations, unless the activation attempt already happened in which case the returned vector will be empty.
  • Takes no arguments.
  • Returns the vector of hexadecimal strings representing the digest of protocol features that are scheduled to be activated the next time the local node can produce a block.

The chain_api_plugin has added one new RPC endpoint:

• get_activated_protocol_features: Returns the protocol features that have been activated as of the current state of the blockchain (ordered according to order of activation, unless reversed by request). It also includes metadata like the ordinal of the activation and the block in which it was activated.
  • Optional integer parameter lower_bound acts as the lower bound to filter the returned results (providing no lower_bound avoids putting a lower bound filter on the results which acts equivalent to passing a lower_bound value of 0).
  • Optional integer parameter upper_bound acts as the upper bound to filter the returned results (providing no upper_bound avoids putting a upper bound filter on the results which acts equivalent to passing a upper_bound value equivalent to the largest value representable by a C++ uint32_t).
  • Optional integer parameter limit (defaults to 10) puts a maximum limit on the number of protocol features returned.
  • Optional boolean search_by_block_num (defaults to false) determines how the lower_bound and upper_bound values will be treated in the filtering process. If search_by_block_num is false, those values act as bounds on the activation ordinal (which can act as a unique lookup key for protocol features and can support pagination). If search_by_block_num is true, those values act as bounds on the activate block number (the number of the block in which the protocol features were activated).
  • Optional boolean reverse (defaults to false) determines whether to reverse the order of the returned results.
  • Returns an object containing an array field named activated_protocol_features and optionally including an integer field named more which if present means that more results of the query exist than what was returned (due to time constraints) and in that case the value of more is the activation ordinal of the next protocol feature that should have been returned if time allowed. The activate_protocol_features array contains JSON objects (representing the protocol features) that contain the following fields:
    • feature_digest: Hexadecimal string representation of the digest of the protocol feature.
    • activation_ordinal: An integer value of the activation ordinal of the protocol feature, where the activation ordinal is the sequence number of ordered protocol feature activations that have occurred on the chain since genesis.
    • activation_block_num: An integer value of the number of the block in which the protocol feature was activated.
    • description_digest: Hexadecimal string representation of the digest of the textual description for the protocol feature.
    • dependencies: An array of hexadecimal strings representing the digests of any protocol features this protocol feature depends on (can be an empty array and usually would be).
    • protocol_feature_type: A string representing the class of protocol features. Currently only the builtin class is supported which would be represented here with the string "builtin".
    • specification: An array of JSON objects each containing a string-valued field called name and any-valued field called value which provide the specification for the protocol feature. In the case of builtin protocol features, there should be only one JSON object in the array that has a name field with value "builtin_feature_codename" and a value field with value equal to a string representing the codename of the builtin protocol feature.

Documentation Additions

In addition to the API changes discussed above, the documentation will likely need updates to reflect the addition of the config/protocol_features directory and the format of the JSON files contained within. Also, the whole concept of protocol features and how they work will likely need to be documented elsewhere in a manner more accessible to users, contract developers, and block producers than what is in this PR.

[taokayan]

I've briefly go through it and can't see any vulnerability. But since the changes are more than 4000 lines of code, it is unlikely to be checked very deeply. I suggest to merge the code first and then we can start testing.