Add metrics for gas price / usage #700

nytzuga · 2023-06-26T20:54:42Z

Why this should be merged

Adds more metrics requested in #662

How this works

New metrics has been added to extend the visibility of nodes.

How this was tested

To be updated

How is this documented

It is a minor request, maybe the documents for the metrics should be updated.

aaronbuchwald · 2023-06-27T01:52:02Z

core/blockchain.go

+	if baseFee := block.BaseFee(); baseFee != nil {
+		blockGasPriceGauge.Update(baseFee.Int64())
+	}
+
+	totalFees := big.NewInt(0)
+	for _, tx := range block.Transactions() {
+		gasPrice := tx.GasPrice()
+		gasUsed := new(big.Int).SetUint64(tx.Gas())
+		txFees := new(big.Int).Mul(gasPrice, gasUsed)
+		totalFees = new(big.Int).Add(totalFees, txFees)
+	}
+	blockTotalFeesGauge.Update(totalFees.Int64())


Can we move these metrics into Accept here: https://github.com/ava-labs/subnet-evm/blob/master/core/blockchain.go#L1055 ?

We should also update the totalFees calculation to account for the base fee and effective gas price. Unfortunately, GasPrice got a bit more complicated with EIP-1559: https://timroughgarden.org/papers/eip1559.pdf.

The transaction's GasPrice() function returns the gas price for legacy transactions (https://github.com/ava-labs/subnet-evm/blob/master/core/types/legacy_tx.go#L109) and the GasFeeCap for transactions added post-EIP1559 (https://github.com/ava-labs/subnet-evm/blob/master/core/types/dynamic_fee_tx.go#L102), which does not account for the actual fee paid anymore.

We actually do this incorrectly in the logs here too: https://github.com/ava-labs/subnet-evm/blob/master/miner/worker.go#L391, so I think we can leave this in this PR for now and make a ticket to correct this and factor out the code so that it can be shared across these two places.

Thanks for the feedback @aaronbuchwald, I moved the changes to the place you suggested I also update the total Fees calculation to include GasFeeCap() and GasTipCap().

darioush · 2023-06-27T16:34:26Z

core/blockchain.go

+		gasFeeCap := tx.GasFeeCap()
+		gasTipCap := tx.GasTipCap()
+		gasUsed := new(big.Int).SetUint64(tx.Gas())
+		if gasUsed != nil && gasTipCap != nil && gasFeeCap != nil {


Suggested change

if gasUsed != nil && gasTipCap != nil && gasFeeCap != nil {

if gasTipCap != nil && gasFeeCap != nil {

darioush · 2023-06-27T16:36:37Z

core/blockchain.go

+		gasUsed := new(big.Int).SetUint64(tx.Gas())
+		if gasUsed != nil && gasTipCap != nil && gasFeeCap != nil {
+			txFees := new(big.Int).Mul(new(big.Int).Add(gasFeeCap, gasTipCap), gasUsed)
+			totalFees = new(big.Int).Add(totalFees, txFees)


nit: wonder if totalFees.Add(totalFees, txFees) saves an allocation.

darioush · 2023-06-27T16:42:10Z

core/blockchain.go

+	for _, tx := range block.Transactions() {
+		gasFeeCap := tx.GasFeeCap()
+		gasTipCap := tx.GasTipCap()
+		gasUsed := new(big.Int).SetUint64(tx.Gas())


I could be wrong.. but I believe that tx.Gas() returns the gas supplied and the gas used is available in receipts. eg:
https://github.com/ava-labs/coreth/blob/master/miner/worker.go#L370
I think we should follow the same computation with tx.EffectiveGasTipValue(baseFee) as well.

That's correct

aaronbuchwald · 2023-06-29T18:12:09Z

core/blockchain.go

+	return feesWei, nil
+}
+
+func TotalFeesFloat(block *types.Block, receipts []*types.Receipt) *big.Float {


Could we change the return type here to return the error instead of a debug log and zero?

aaronbuchwald · 2023-06-29T18:12:58Z

core/blockchain.go

@@ -81,6 +81,8 @@ var (
 	acceptorQueueGauge           = metrics.NewRegisteredGauge("chain/acceptor/queue/size", nil)
 	acceptorWorkTimer            = metrics.NewRegisteredCounter("chain/acceptor/work", nil)
 	acceptorWorkCount            = metrics.NewRegisteredCounter("chain/acceptor/work/count", nil)
+	blockGasPriceGauge           = metrics.NewRegisteredGauge("chain/block/gas/price", nil)


Suggested change

blockGasPriceGauge = metrics.NewRegisteredGauge("chain/block/gas/price", nil)

lastAcceptedBlockBaseFeeGauge = metrics.NewRegisteredGauge("chain/block/gas/basefee", nil)

aaronbuchwald · 2023-06-29T18:13:53Z

core/blockchain.go

@@ -81,6 +81,8 @@ var (
 	acceptorQueueGauge           = metrics.NewRegisteredGauge("chain/acceptor/queue/size", nil)
 	acceptorWorkTimer            = metrics.NewRegisteredCounter("chain/acceptor/work", nil)
 	acceptorWorkCount            = metrics.NewRegisteredCounter("chain/acceptor/work/count", nil)
+	blockGasPriceGauge           = metrics.NewRegisteredGauge("chain/block/gas/price", nil)
+	blockTotalFeesGauge          = metrics.NewRegisteredGauge("chain/block/fees", nil)


Suggested change

blockTotalFeesGauge = metrics.NewRegisteredGauge("chain/block/fees", nil)

blockTotalFeesGauge = metrics.NewRegisteredGauge("chain/block/gas/fees", nil)

aaronbuchwald · 2023-06-29T18:15:41Z

core/blockchain.go

+	}
+	total, err := TotalFees(block, bc.GetReceiptsByHash(block.Hash()))
+	if err != nil {
+		log.Debug(fmt.Sprintf("TotalFees %s", err))


Can we change this to Error level? This error should never happen since the block has been accepted and we should have the same number of transactions and receipts for the indexed block.

aaronbuchwald · 2023-06-29T18:16:31Z

core/blockchain.go

+func TotalFeesFloat(block *types.Block, receipts []*types.Receipt) *big.Float {
+	total, err := TotalFees(block, receipts)
+	if err != nil {
+		log.Debug(fmt.Sprintf("TotalFees %s", err))


this should be an Error log as well, since it should never happen

Log errors instead of returning if there is a mismatch between the number of transactions and receipts (this would mean an issue at dev time instead of runtime)

ceyonur · 2023-06-29T20:27:04Z

core/blockchain.go

+	}
+	total, err := TotalFees(block, bc.GetReceiptsByHash(block.Hash()))
+	if err != nil {
+		log.Error(fmt.Sprintf("TotalFees error: %s", err))


Did we specifically not returned any error here? If so I think this should be a warn instead of error

I think error is the right level here since this should never happen

Why dont we return the error?

Could go either way. The desire was not to treat this as fatal since it's only used to add an extra metric.

core/blockchain.go

ceyonur · 2023-06-29T20:57:04Z

core/blockchain.go

 	return nil
 }

+// TotalFees computes total consumed fees in ETH. Block transactions and receipts have to have the same order.


I think it's in wei rather than eth

ceyonur · 2023-06-29T20:58:34Z

core/blockchain.go

+	return feesWei, nil
+}
+
+func TotalFeesFloat(block *types.Block, receipts []*types.Receipt) *big.Float {


I feel like this is a helper function and not need to be exported. core.TotalFees handles most of the calculations so we might not require this at all.
If not required elsewhere, can we remove this function altogether and use it as a variable in worker.go. If you prefer to keep this function, can we keep it as an unexported function in worker.go?

ceyonur · 2023-06-29T21:08:51Z

core/blockchain.go

+		} else {
+			minerFee = new(big.Int).Add(baseFee, tx.EffectiveGasTipValue(baseFee))
+		}
+		feesWei.Add(feesWei, new(big.Int).Mul(new(big.Int).SetUint64(receipts[i].GasUsed), minerFee))


This is a bit hard to read, can we split this into two different variables?

I think this is fine for this PR since it's just copied from miner/worker.go

Co-authored-by: aaronbuchwald <aaron.buchwald56@gmail.com> Signed-off-by: Cesar <137245636+nytzuga@users.noreply.github.com>

* Rename test_ to step_ for clarity (ava-labs#636) * Codeowners fix (specific dir overrides global) (ava-labs#639) * rename state upgrade test (ava-labs#641) * plugin/evm: fix godoc typo (ava-labs#647) Signed-off-by: Gyuho Lee <gyuho.lee@avalabs.org> * plugin/evm: remove unncessary tx.Hash op in map iter (ava-labs#648) Signed-off-by: Gyuho Lee <gyuho.lee@avalabs.org> * deployExampleRewardManager.ts hardhat script fixed (ava-labs#650) * Migrate HardHat tests to DS-tests (Solidity tests) (ava-labs#601) * Increase trie dirty default cache size to 512MB (ava-labs#656) * export gingkgo suites (ava-labs#659) * export gingkgo suites * remove precompiles description from ping tests * update package.json * fix url in package json * change package name * Publish npm module for contract-examples (ava-labs#663) * export gingkgo suites * remove precompiles description from ping tests * update package.json * fix url in package json * change package name * prepare for module publishing * use npm ci * preload ethers * nits * Fix broken path to docs in README. (ava-labs#665) * Fix contract examples package json (ava-labs#672) * rename contract-examples to contracts & fix pack issue * allow custom CMD to be run on blockchains for e2e tests * move default hardhat test to utils * fix comment * add more comments * change folder reference * Bump got and ethereum-waffle in /contracts (ava-labs#680) Removes [got](https://github.com/sindresorhus/got). It's no longer used after updating ancestor dependency [ethereum-waffle](https://github.com/EthWorks/Waffle). These dependencies need to be updated together. Removes `got` Updates `ethereum-waffle` from 3.4.4 to 4.0.10 - [Release notes](https://github.com/EthWorks/Waffle/releases) - [Commits](https://github.com/EthWorks/Waffle/compare/ethereum-waffle@3.4.4...ethereum-waffle@4.0.10) --- updated-dependencies: - dependency-name: got dependency-type: indirect - dependency-name: ethereum-waffle dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump flat and hardhat in /contracts (ava-labs#679) Bumps [flat](https://github.com/hughsk/flat) to 5.0.2 and updates ancestor dependency [hardhat](https://github.com/nomiclabs/hardhat). These dependencies need to be updated together. Updates `flat` from 4.1.1 to 5.0.2 - [Commits](hughsk/flat@4.1.1...5.0.2) Updates `hardhat` from 2.6.1 to 2.15.0 - [Release notes](https://github.com/nomiclabs/hardhat/releases) - [Commits](https://github.com/nomiclabs/hardhat/compare/hardhat-core-v2.6.1...hardhat@2.15.0) --- updated-dependencies: - dependency-name: flat dependency-type: indirect - dependency-name: hardhat dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix core benchmarks (ava-labs#697) * remove unused line in test (ava-labs#695) * bump version to v0.5.2 (ava-labs#681) * bump version to v0.5.2 * downgrade avalanchego to 1.10.2 * update compatibility * Remove unused ethereum-waffle and ethereumjs-tx pkgs (ava-labs#685) * move state upgrade related functions to own file (ava-labs#684) * Update load test to use ANR for network setup (ava-labs#579) * Update load test to use ANR for network setup * Remove ToWebsocketURI that relies on strings feature added in go1.20 * Update load test to use ENV var from CI if populated * Move env var logic into NewDefaultANRConfig * Add init functions and clean up comments * Remove unnecessary helper * Remove unneeded cmd * Cleanup * Address PR comments * Move 2 min timeout for SetupNetwork to the start of the function * Turn list of txs into tx sequence (ava-labs#683) * Turn list of txs into tx sequence * Add agent comments * Remove unused code and address comments * add some more comments * address more comments' * lint * Remove :::warning --------- Co-authored-by: Anusha <anushachillara@gmail.com> * Minor simulator fixes (ava-labs#703) * add benchmark to template (ava-labs#705) * add benchmark to template * fix comment * Add metrics for gas price / usage (ava-labs#700) Co-authored-by: aaronbuchwald <aaron.buchwald56@gmail.com> * added config setting to prune warpdb (ava-labs#702) * added config setting to prune warpdb * refactored warp prune uint test * message * fixed up nits * fixed nits * fixed nits * Update plugin/evm/config.go * Add wrapped validator state for primary to subnet warp messages (ava-labs#708) * Add wrapped validator state for primary to subnet warp message verification * Address PR comment * upload network runner artifact after e2e test run (ava-labs#709) * Differentiate mandatory/optional Subnet-EVM Network Upgrades (ava-labs#640) * remove network upgrades from upgrade configs and enforce them in vm init * fix tests * remove unused var * remove extra shutdown * copy test chain config when modify * use SimulatedTestChainConfig for simulated tests * copy modified test chain config * add optional upgrades * Add tests * fix comments * use unmarshal in test * unexport functions * add commented out test * add TODO comment * add blockchain.Stop as defer to test (ava-labs#714) Co-authered-by: najeal <haim.nathan@icloud.com> Co-authored-by: Ceyhun Onur <ceyhun.onur@avalabs.org> * Add support for predicate gas into IntrinsicGas (ava-labs#713) * Add support for predicate gas * Address review * Fix test and add test for PredicateGas * Address comments * Add signature aggregation helpers (ava-labs#711) * Add signature aggregation helpers * Fix * Move WarpQuorumDenominator into new file * Address comments * fix var rename * Improve comments * Address comments * Add predicate packing helper (ava-labs#710) * Add predicate packing helper * move predicate code and add readme * fix moved import * fix merge * Run ginkgo precompile tests in parallel (ava-labs#712) Use SynchronizedBeforeSuite to create the node and all the subnet that are needed to run tests. Pass the blockchainID to each work processor and run the tests in parallel to reduce runtime * Fix BlockContext comment (ava-labs#731) * precompile accepter should take logIdx (ava-labs#728) * precompile accepter should take logIdx * minimize mock * copyright yr * add comments * Improve tests (ava-labs#733) Improve tests based on feedback from ava-labs#712 * Fix precompile test gen template (ava-labs#737) * fix Benchmark * fix spelling/references in Benchmark * Clean up gossip (ava-labs#744) * Remove call from precompile (ava-labs#748) * Fix nil types tmpl (ava-labs#751) * add more nil types and fix int types * readd hash type * remove convertToNil * Sync coreth v0.12.4 rc0 (ava-labs#694) * apply diff to rc0 * replace coreth with subnet-evm * go mod tidy * remove must commit * remove apricots * replace coreth with subnet-evm * rename banff and cortina * test memory db * add dUpgrade * fix errors * sync with v0.12.4-rc0 * more diffs * change templates * readd preparePredicateSlots * run goimports * readd tx allow list check to txpool * Fix tests * readd subnet evm check for tests * fix tests * add gas limit & coinbase fakers * Fix timestamp types * check error from newblockchain * fix gas limit in simulated genesis * downgrade ci to go 1.19 * gofmt files * rebump go to 1.20 * bump anr to latest * bump anr to latest * prepare predicate storage slots in Subnet-EVM * uncomment tests * change order of airdrop memory cleanup * add utils import * Coreth 0.12.4 x apply (ava-labs#720) Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> * move instruction sets * revert unpack changes * reduce diffs * move genesis verify to vm (ava-labs#753) * move genesis verify to vm * fix comment * set a correct default from genesis * fix log * revert change to vm.go --------- Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> * avoid adding/removing GasLimits in tests (ava-labs#755) * move genesis verify to vm * fix comment * set a correct default from genesis * fix log * revert change to vm.go * avoid adding/removing GasLimits in tests * nit --------- Co-authored-by: Ceyhun Onur <ceyhun.onur@avalabs.org> --------- Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> * Bump avalanchego to v1.10.5 and bump Subnet-EVM for v0.5.3 release (ava-labs#757) * Bump avalanchego to v1.10.5 and bump Subnet-EVM for v0.5.3 release * bump anr version to v1.7.1 * Migrate ABI unpack fix (ava-labs#750) * change sprintf verb and use values (ava-labs#760) * Generalize precompile test suite (ava-labs#759) * restore old functions, generalize paths in sync subnet creation * add subnet suit struct and methods * move constants * fix reviews * add comment * Update subnet.go Signed-off-by: Ceyhun Onur <ceyhun.onur@avalabs.org> --------- Signed-off-by: Ceyhun Onur <ceyhun.onur@avalabs.org> * Add warp contract implementation (ava-labs#718) * Add warp contract implementation * Cleanup predicate test * Fix new function signature * Replace invalid fuzz test with unit test * Add chain config to enable warp API for warp e2e test * remove unused var * Add experimental warning and move warp precompile to x/ package * fix warning label * Fix warning * vm test nits * Improve sendWarpMessenger sol comment * more vm warp test nits * Move warp params into params package * More vm warp test nits * Address more PR comments * Remove triggerTx2 * Add check for expected topics from sendWarpMessage log * Fix config test * Fix incorrect replace * remove unnecessary echo * Address comments * Address comments * Address PR comments * Improve comments * Convert [32]byte type to common.Hash * Add base cost for getVerifiedWarpMessage * fix require equal type check * Fix updated awm message format * Update warp message format * Move IWarpMessenger.sol to interfaces/ * Add newline to warp genesis * uncomment assertion * Fix broken links in README * Add metrics to simulator (ava-labs#706) * add tps * get block build metrics * Add issuance time, confirmed time, issued-> acceptance time, verify time * Log times at end of batch * cleaner * address comments * remove unused code block * avoids taking len of channel * nits * pass in stringID * move to loader * remove unused field * revert file back * cleaner * lint * make it work for ws or for rpc * protect * endpoint * no return on defer * sep to a funciton * have blockchainidstr passed in * typo * pass in metrics through config * address comments * address more comments and edit err policy of metrics functions * add more logging to load_test * typo * better check * fix endpoints * typo: * individual * histogram * address feedback: * remove metrics from default * address comments * simplify time metrics * better explanation * address comments * address comments * cleanup * more cleanup * rename vars for clarity * ws * cleanup * address comments * ws * expose metrics add flag * fix blocking issue of http server and gracefully stop it * cleanup * use constant * add issuance to confirmation metrics * ws * simplify metrics server * Bump avalanchego to v1.10.5 and bump Subnet-EVM for v0.5.3 release (ava-labs#757) * Bump avalanchego to v1.10.5 and bump Subnet-EVM for v0.5.3 release * bump anr version to v1.7.1 * handle control c * print out output * clean up * clean up * remove go routines to close client * address comments * memory leak * fix * print * e2e test: connect to appropriate chain (ava-labs#771) * Bump avalanchego dep and update version for next release (ava-labs#770) * Bump avalanchego dep and update version for next release * Update cache sizes * Update flag in run script to replace staking-enabled=false * Update block builder to enforce predicate one tx at a time (ava-labs#773) * Update block builder to enforce predicate one tx at a time * remove dead code * Update miner/worker.go Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com> --------- Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com> Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> * update iocorders address * remove state-sync-enabled from chain config * 1s block ticker * flush hubbleFeed when len > 512 * fix compilation err --------- Signed-off-by: Gyuho Lee <gyuho.lee@avalabs.org> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Ceyhun Onur <ceyhun.onur@avalabs.org> Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com> Co-authored-by: Richard Pringle <rpring9@gmail.com> Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> Co-authored-by: Ceyhun Onur <ceyhun.onur@avalabs.org> Co-authored-by: Gyuho Lee <gyuho.lee@avalabs.org> Co-authored-by: İzzet Emre Demir <izzetemredemir@gmail.com> Co-authored-by: aaronbuchwald <aaron.buchwald56@gmail.com> Co-authored-by: Gabriel Cardona <gabriel@avalabs.org> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Anusha <anushachillara@gmail.com> Co-authored-by: Cesar <137245636+nytzuga@users.noreply.github.com> Co-authored-by: morrisettjohn <60852062+morrisettjohn@users.noreply.github.com> Co-authored-by: nathan haim <nathan.haim@free.fr> Co-authored-by: Martin Eckardt <m.eckardt@outlook.com> Co-authored-by: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Co-authored-by: Anusha <63559942+anusha-ctrl@users.noreply.github.com>

nytzuga requested review from ceyonur, darioush and aaronbuchwald as code owners June 26, 2023 20:54

nytzuga force-pushed the add-metrics-gas-price-usage branch 2 times, most recently from a8b33d7 to 89ae56a Compare June 26, 2023 21:30

aaronbuchwald reviewed Jun 27, 2023

View reviewed changes

nytzuga force-pushed the add-metrics-gas-price-usage branch from 89ae56a to 6b5b2ff Compare June 27, 2023 15:10

darioush reviewed Jun 27, 2023

View reviewed changes

nytzuga force-pushed the add-metrics-gas-price-usage branch 3 times, most recently from 4893f1a to e9e771f Compare June 29, 2023 16:12

aaronbuchwald reviewed Jun 29, 2023

View reviewed changes

nytzuga force-pushed the add-metrics-gas-price-usage branch from e9e771f to 9b731f8 Compare June 29, 2023 20:21

Add metrics for gas price / usage

6a21b9a

Log errors instead of returning if there is a mismatch between the number of transactions and receipts (this would mean an issue at dev time instead of runtime)

nytzuga force-pushed the add-metrics-gas-price-usage branch from 9b731f8 to 6a21b9a Compare June 29, 2023 21:02

ceyonur reviewed Jun 29, 2023

View reviewed changes

nytzuga and others added 2 commits June 30, 2023 10:10

Update core/blockchain.go

9d8d323

Co-authored-by: aaronbuchwald <aaron.buchwald56@gmail.com> Signed-off-by: Cesar <137245636+nytzuga@users.noreply.github.com>

Fix comments

cf0b1a6

aaronbuchwald approved these changes Jun 30, 2023

View reviewed changes

darioush approved these changes Jun 30, 2023

View reviewed changes

darioush merged commit a221f63 into master Jun 30, 2023
7 checks passed

darioush deleted the add-metrics-gas-price-usage branch June 30, 2023 15:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add metrics for gas price / usage #700

Add metrics for gas price / usage #700

nytzuga commented Jun 26, 2023

aaronbuchwald Jun 27, 2023

nytzuga Jun 27, 2023

darioush Jun 27, 2023

darioush Jun 27, 2023

darioush Jun 27, 2023

aaronbuchwald Jun 27, 2023

aaronbuchwald Jun 29, 2023

aaronbuchwald Jun 29, 2023

aaronbuchwald Jun 29, 2023

aaronbuchwald Jun 29, 2023

aaronbuchwald Jun 29, 2023

ceyonur Jun 29, 2023

aaronbuchwald Jun 29, 2023

ceyonur Jun 30, 2023

aaronbuchwald Jun 30, 2023

ceyonur Jun 29, 2023

ceyonur Jun 29, 2023

ceyonur Jun 29, 2023

aaronbuchwald Jun 29, 2023

	if gasUsed != nil && gasTipCap != nil && gasFeeCap != nil {
	if gasTipCap != nil && gasFeeCap != nil {

	blockGasPriceGauge = metrics.NewRegisteredGauge("chain/block/gas/price", nil)
	lastAcceptedBlockBaseFeeGauge = metrics.NewRegisteredGauge("chain/block/gas/basefee", nil)

	blockTotalFeesGauge = metrics.NewRegisteredGauge("chain/block/fees", nil)
	blockTotalFeesGauge = metrics.NewRegisteredGauge("chain/block/gas/fees", nil)

Add metrics for gas price / usage #700

Add metrics for gas price / usage #700

Conversation

nytzuga commented Jun 26, 2023

Why this should be merged

How this works

How this was tested

How is this documented

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment