[Tracking & Design]: FT benchmark workload generation and tooling improvements

[mooori]

Goals

• Running the workload should require executing only 2 commands:
  • one command to initialize state
  • one command to run the workload using that state
• State initialized for a given network should be re-usable in subsequent benchmark runs.
• Errors and transaction failures should be handled.
• The code generating the workload should be easy to understand and extend.
• Modular building blocks reduce the effort of generating new workloads.

Building blocks

The following functionality should be generic enough to allow re-using it for other workloads:

• Create an account, enable creating multiple in batches
• Deploy a contract to an account
• Call a contract method
• Get the execution outcome of a transaction

FT workload implementation

Above building blocks should be sufficient to implement the ft workload consisting of:

• Deploy and initialize FT contract(s)
• Create user accounts
• Register them with the FT contract
  • Modify the smart contract to enable registering users in batches, i.e. n > 1 users per transaction
• Mint FT for users
• Transfer tokens between users with ft_transfer

Principles

Sequential initialization

Contract and state initialization should be sequential, to dedicate all resources to it.

• After its finalization, all transactions sent to the node should be included in tps measurements. The sequential approach avoids tainting tps measurements.
• Avoid congestion that currently happens due to the Locust workload sending initialization transactions and ft_transfers at the same time.

Tested workload generation

Executing a command like cargo test should suffice to invoke a test suite for the code that generates the workload.

• If some functionality is broken, a test failure should pinpoint the problem. Making the test pass again allows to verify a fix works.
• Working on workload generation becomes more convenient when new functionality can be tested automatically, without requiring manual setup and full workload invocations.
• Avoid a situation like with the current Locust ft workload that continuously reports errors during "successful" execution.

Use a well maintained RPC library

There are frequent changes to NEAR RPC standards and methods. A well maintained RPC library should handle such changes and minimize the work required to adapt workload generation to RPC changes.

Auto generate tps numbers and further data

After running a workload, a report should be printed containing:

• tps numbers
• data regarding node health: e.g. skipped chunks, failed transactions, etc.
• data regarding node load: e.g. metrics for congestion and factors that limit including transactions or receipts

[mooori]

A status update is available here.