[Feature] Shared replay buffers

[vmoens]

Description

Allows a replay buffer with memmap storage to be shared across processes.

Example:

import pickle

from torchrl.data import TensorDictReplayBuffer, LazyMemmapStorage
import torch
from torch import multiprocessing as mp
from tensordict import TensorDict

def worker(rb):
    td = TensorDict({"a": torch.ones(10)}, [10])
    rb.extend(td)

if __name__ == "__main__":
    rb = TensorDictReplayBuffer(storage=LazyMemmapStorage(21))
    td = TensorDict({"a": torch.zeros(10)}, [10])
    rb.extend(td)
    
    proc = mp.Process(target=worker, args=(rb,))
    proc.start()
    proc.join()
    assert len(rb) == 20

TODO:

• Raise exception if assumption are not met (eg, ListStorage)
• Document the feature

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1724

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (4 Unrelated Failures)

As of commit ab463f0 with merge base 6c27bdb ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Build M1 Wheels / pytorch/rl / wheel-py3_8-cpu (gh)
• Unit-tests on Linux CPU / tests (3.9) / linux-job (gh)

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Habitat Tests on Linux / tests (3.9, 11.6) / linux-job (gh)
• Unit-tests on Windows GPU / unittests / windows-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}5$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	67.4719ms	63.8922ms	15.6514 Ops/s	15.2075 Ops/s	$\color{#35bf28}+2.92\%$
test_sync	40.6903ms	34.9258ms	28.6321 Ops/s	28.7671 Ops/s	$\color{#d91a1a}-0.47\%$
test_async	59.7037ms	31.6839ms	31.5617 Ops/s	30.3871 Ops/s	$\color{#35bf28}+3.87\%$
test_simple	0.4876s	0.4332s	2.3084 Ops/s	2.2959 Ops/s	$\color{#35bf28}+0.54\%$
test_transformed	0.6526s	0.5933s	1.6856 Ops/s	1.6357 Ops/s	$\color{#35bf28}+3.05\%$
test_serial	1.3930s	1.3379s	0.7475 Ops/s	0.7417 Ops/s	$\color{#35bf28}+0.78\%$
test_parallel	1.3237s	1.2837s	0.7790 Ops/s	0.7758 Ops/s	$\color{#35bf28}+0.42\%$
test_step_mdp_speed[True-True-True-True-True]	0.1787ms	22.8566μs	43.7510 KOps/s	44.3228 KOps/s	$\color{#d91a1a}-1.29\%$
test_step_mdp_speed[True-True-True-True-False]	50.2510μs	13.8186μs	72.3660 KOps/s	73.0125 KOps/s	$\color{#d91a1a}-0.89\%$
test_step_mdp_speed[True-True-True-False-True]	55.5030μs	13.8403μs	72.2527 KOps/s	73.1816 KOps/s	$\color{#d91a1a}-1.27\%$
test_step_mdp_speed[True-True-True-False-False]	46.9370μs	8.2099μs	121.8048 KOps/s	121.0161 KOps/s	$\color{#35bf28}+0.65\%$
test_step_mdp_speed[True-True-False-True-True]	52.7280μs	24.6622μs	40.5478 KOps/s	42.0671 KOps/s	$\color{#d91a1a}-3.61\%$
test_step_mdp_speed[True-True-False-True-False]	37.8200μs	15.0519μs	66.4369 KOps/s	66.6956 KOps/s	$\color{#d91a1a}-0.39\%$
test_step_mdp_speed[True-True-False-False-True]	62.6260μs	15.2292μs	65.6633 KOps/s	66.2575 KOps/s	$\color{#d91a1a}-0.90\%$
test_step_mdp_speed[True-True-False-False-False]	30.0560μs	9.4817μs	105.4660 KOps/s	102.7961 KOps/s	$\color{#35bf28}+2.60\%$
test_step_mdp_speed[True-False-True-True-True]	0.1088ms	25.7267μs	38.8701 KOps/s	39.7521 KOps/s	$\color{#d91a1a}-2.22\%$
test_step_mdp_speed[True-False-True-True-False]	55.0020μs	16.2944μs	61.3707 KOps/s	60.9110 KOps/s	$\color{#35bf28}+0.75\%$
test_step_mdp_speed[True-False-True-False-True]	46.4560μs	15.0418μs	66.4816 KOps/s	66.8221 KOps/s	$\color{#d91a1a}-0.51\%$
test_step_mdp_speed[True-False-True-False-False]	57.5070μs	9.4993μs	105.2709 KOps/s	103.1970 KOps/s	$\color{#35bf28}+2.01\%$
test_step_mdp_speed[True-False-False-True-True]	67.3650μs	26.8393μs	37.2588 KOps/s	37.5255 KOps/s	$\color{#d91a1a}-0.71\%$
test_step_mdp_speed[True-False-False-True-False]	71.9440μs	17.5736μs	56.9034 KOps/s	57.1083 KOps/s	$\color{#d91a1a}-0.36\%$
test_step_mdp_speed[True-False-False-False-True]	55.4940μs	16.4020μs	60.9682 KOps/s	61.1222 KOps/s	$\color{#d91a1a}-0.25\%$
test_step_mdp_speed[True-False-False-False-False]	62.6870μs	10.7172μs	93.3077 KOps/s	92.4406 KOps/s	$\color{#35bf28}+0.94\%$
test_step_mdp_speed[False-True-True-True-True]	54.9830μs	25.6189μs	39.0337 KOps/s	39.0963 KOps/s	$\color{#d91a1a}-0.16\%$
test_step_mdp_speed[False-True-True-True-False]	85.2580μs	16.4129μs	60.9276 KOps/s	60.8261 KOps/s	$\color{#35bf28}+0.17\%$
test_step_mdp_speed[False-True-True-False-True]	48.8310μs	17.4545μs	57.2918 KOps/s	56.7680 KOps/s	$\color{#35bf28}+0.92\%$
test_step_mdp_speed[False-True-True-False-False]	35.3460μs	10.6811μs	93.6230 KOps/s	91.4259 KOps/s	$\color{#35bf28}+2.40\%$
test_step_mdp_speed[False-True-False-True-True]	0.1101ms	26.3295μs	37.9802 KOps/s	37.5755 KOps/s	$\color{#35bf28}+1.08\%$
test_step_mdp_speed[False-True-False-True-False]	62.5560μs	17.6289μs	56.7251 KOps/s	56.9237 KOps/s	$\color{#d91a1a}-0.35\%$
test_step_mdp_speed[False-True-False-False-True]	74.1690μs	18.5147μs	54.0110 KOps/s	53.6828 KOps/s	$\color{#35bf28}+0.61\%$
test_step_mdp_speed[False-True-False-False-False]	59.9310μs	12.1271μs	82.4596 KOps/s	83.2475 KOps/s	$\color{#d91a1a}-0.95\%$
test_step_mdp_speed[False-False-True-True-True]	84.2770μs	27.9258μs	35.8092 KOps/s	35.7283 KOps/s	$\color{#35bf28}+0.23\%$
test_step_mdp_speed[False-False-True-True-False]	61.9850μs	18.7518μs	53.3283 KOps/s	52.8453 KOps/s	$\color{#35bf28}+0.91\%$
test_step_mdp_speed[False-False-True-False-True]	74.8590μs	18.5846μs	53.8081 KOps/s	54.0538 KOps/s	$\color{#d91a1a}-0.45\%$
test_step_mdp_speed[False-False-True-False-False]	40.7060μs	11.9135μs	83.9386 KOps/s	82.8075 KOps/s	$\color{#35bf28}+1.37\%$
test_step_mdp_speed[False-False-False-True-True]	79.7090μs	29.4704μs	33.9324 KOps/s	34.9252 KOps/s	$\color{#d91a1a}-2.84\%$
test_step_mdp_speed[False-False-False-True-False]	68.1370μs	19.7808μs	50.5541 KOps/s	50.0031 KOps/s	$\color{#35bf28}+1.10\%$
test_step_mdp_speed[False-False-False-False-True]	69.8900μs	19.8701μs	50.3268 KOps/s	51.1208 KOps/s	$\color{#d91a1a}-1.55\%$
test_step_mdp_speed[False-False-False-False-False]	61.0540μs	13.1627μs	75.9724 KOps/s	76.5935 KOps/s	$\color{#d91a1a}-0.81\%$
test_values[generalized_advantage_estimate-True-True]	12.7018ms	11.9458ms	83.7114 Ops/s	83.1197 Ops/s	$\color{#35bf28}+0.71\%$
test_values[vec_generalized_advantage_estimate-True-True]	34.1571ms	26.5234ms	37.7026 Ops/s	37.9956 Ops/s	$\color{#d91a1a}-0.77\%$
test_values[td0_return_estimate-False-False]	0.2409ms	0.2004ms	4.9911 KOps/s	5.2366 KOps/s	$\color{#d91a1a}-4.69\%$
test_values[td1_return_estimate-False-False]	49.2118ms	26.4886ms	37.7520 Ops/s	38.4923 Ops/s	$\color{#d91a1a}-1.92\%$
test_values[vec_td1_return_estimate-False-False]	33.7013ms	26.4854ms	37.7566 Ops/s	38.0754 Ops/s	$\color{#d91a1a}-0.84\%$
test_values[td_lambda_return_estimate-True-False]	53.5390ms	36.2499ms	27.5863 Ops/s	27.6335 Ops/s	$\color{#d91a1a}-0.17\%$
test_values[vec_td_lambda_return_estimate-True-False]	34.5158ms	26.6477ms	37.5267 Ops/s	37.6678 Ops/s	$\color{#d91a1a}-0.37\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	7.9600ms	7.8377ms	127.5889 Ops/s	123.0562 Ops/s	$\color{#35bf28}+3.68\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	10.5278ms	1.8429ms	542.6248 Ops/s	529.3785 Ops/s	$\color{#35bf28}+2.50\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	8.5893ms	0.4269ms	2.3422 KOps/s	2.2897 KOps/s	$\color{#35bf28}+2.30\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	41.3817ms	39.5629ms	25.2762 Ops/s	25.2642 Ops/s	$\color{#35bf28}+0.05\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	2.6725ms	2.4795ms	403.3015 Ops/s	395.0562 Ops/s	$\color{#35bf28}+2.09\%$
test_dqn_speed	11.8649ms	1.6370ms	610.8571 Ops/s	602.8243 Ops/s	$\color{#35bf28}+1.33\%$
test_ddpg_speed	75.1824ms	3.8599ms	259.0754 Ops/s	272.2033 Ops/s	$\color{#d91a1a}-4.82\%$
test_sac_speed	19.1185ms	10.4389ms	95.7952 Ops/s	88.8782 Ops/s	$\textbf{\color{#35bf28}+7.78\%}$
test_redq_speed	30.3533ms	19.7979ms	50.5105 Ops/s	50.6629 Ops/s	$\color{#d91a1a}-0.30\%$
test_redq_deprec_speed	27.8640ms	16.0260ms	62.3987 Ops/s	61.8139 Ops/s	$\color{#35bf28}+0.95\%$
test_td3_speed	19.1797ms	10.6583ms	93.8236 Ops/s	94.0548 Ops/s	$\color{#d91a1a}-0.25\%$
test_cql_speed	50.1846ms	41.2564ms	24.2387 Ops/s	25.7928 Ops/s	$\textbf{\color{#d91a1a}-6.03\%}$
test_a2c_speed	16.1196ms	8.1330ms	122.9556 Ops/s	121.0620 Ops/s	$\color{#35bf28}+1.56\%$
test_ppo_speed	16.3789ms	8.7587ms	114.1728 Ops/s	115.7953 Ops/s	$\color{#d91a1a}-1.40\%$
test_reinforce_speed	15.5453ms	7.4601ms	134.0457 Ops/s	136.8631 Ops/s	$\color{#d91a1a}-2.06\%$
test_iql_speed	42.0677ms	35.2449ms	28.3729 Ops/s	29.1730 Ops/s	$\color{#d91a1a}-2.74\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.0265ms	1.9209ms	520.5811 Ops/s	523.7265 Ops/s	$\color{#d91a1a}-0.60\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.0551ms	2.1221ms	471.2229 Ops/s	492.7825 Ops/s	$\color{#d91a1a}-4.38\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	3.7295ms	2.1715ms	460.5086 Ops/s	467.0424 Ops/s	$\color{#d91a1a}-1.40\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.6149ms	2.0994ms	476.3367 Ops/s	491.9174 Ops/s	$\color{#d91a1a}-3.17\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.9848ms	1.9975ms	500.6358 Ops/s	465.7728 Ops/s	$\textbf{\color{#35bf28}+7.48\%}$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3.1174ms	2.0931ms	477.7517 Ops/s	457.3835 Ops/s	$\color{#35bf28}+4.45\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.8729ms	1.9773ms	505.7363 Ops/s	480.6359 Ops/s	$\textbf{\color{#35bf28}+5.22\%}$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	3.6642ms	2.1390ms	467.5073 Ops/s	462.4392 Ops/s	$\color{#35bf28}+1.10\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	3.8470ms	2.1721ms	460.3925 Ops/s	462.1168 Ops/s	$\color{#d91a1a}-0.37\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.7899ms	1.9119ms	523.0407 Ops/s	505.2123 Ops/s	$\color{#35bf28}+3.53\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.8199ms	2.1357ms	468.2252 Ops/s	478.1744 Ops/s	$\color{#d91a1a}-2.08\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	3.8644ms	2.1759ms	459.5788 Ops/s	470.2160 Ops/s	$\color{#d91a1a}-2.26\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.0104ms	1.9968ms	500.7935 Ops/s	508.5685 Ops/s	$\color{#d91a1a}-1.53\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3.9626ms	2.1344ms	468.5094 Ops/s	427.8825 Ops/s	$\textbf{\color{#35bf28}+9.49\%}$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	2.9227ms	2.0010ms	499.7450 Ops/s	477.0261 Ops/s	$\color{#35bf28}+4.76\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.0737ms	1.8560ms	538.8007 Ops/s	515.1271 Ops/s	$\color{#35bf28}+4.60\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	3.1722ms	2.1089ms	474.1880 Ops/s	420.7880 Ops/s	$\textbf{\color{#35bf28}+12.69\%}$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	3.2785ms	2.1245ms	470.7025 Ops/s	465.9094 Ops/s	$\color{#35bf28}+1.03\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1927s	18.7751ms	53.2619 Ops/s	52.7799 Ops/s	$\color{#35bf28}+0.91\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	0.1222s	16.9913ms	58.8536 Ops/s	60.4699 Ops/s	$\color{#d91a1a}-2.67\%$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	0.1101s	16.4444ms	60.8111 Ops/s	58.7786 Ops/s	$\color{#35bf28}+3.46\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1189s	17.1475ms	58.3177 Ops/s	59.3969 Ops/s	$\color{#d91a1a}-1.82\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	0.1233s	19.4634ms	51.3784 Ops/s	58.9722 Ops/s	$\textbf{\color{#d91a1a}-12.88\%}$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	0.1251s	17.3282ms	57.7094 Ops/s	58.4879 Ops/s	$\color{#d91a1a}-1.33\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1193s	17.0855ms	58.5293 Ops/s	59.2623 Ops/s	$\color{#d91a1a}-1.24\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	0.1094s	16.4318ms	60.8575 Ops/s	65.8102 Ops/s	$\textbf{\color{#d91a1a}-7.53\%}$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	0.1253s	16.9805ms	58.8910 Ops/s	58.8579 Ops/s	$\color{#35bf28}+0.06\%$

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}12$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1228s	0.1209s	8.2723 Ops/s	8.2077 Ops/s	$\color{#35bf28}+0.79\%$
test_sync	0.1026s	0.1024s	9.7667 Ops/s	9.6860 Ops/s	$\color{#35bf28}+0.83\%$
test_async	0.2789s	0.1011s	9.8901 Ops/s	9.9217 Ops/s	$\color{#d91a1a}-0.32\%$
test_single_pixels	0.1323s	0.1317s	7.5908 Ops/s	6.9181 Ops/s	$\textbf{\color{#35bf28}+9.72\%}$
test_sync_pixels	96.1406ms	94.8738ms	10.5403 Ops/s	10.5653 Ops/s	$\color{#d91a1a}-0.24\%$
test_async_pixels	0.2441s	91.4206ms	10.9385 Ops/s	10.8832 Ops/s	$\color{#35bf28}+0.51\%$
test_simple	0.9400s	0.8743s	1.1438 Ops/s	1.1077 Ops/s	$\color{#35bf28}+3.26\%$
test_transformed	1.1836s	1.1204s	0.8925 Ops/s	0.8606 Ops/s	$\color{#35bf28}+3.71\%$
test_serial	2.5762s	2.4761s	0.4039 Ops/s	0.3952 Ops/s	$\color{#35bf28}+2.19\%$
test_parallel	2.5455s	2.4782s	0.4035 Ops/s	0.4030 Ops/s	$\color{#35bf28}+0.12\%$
test_step_mdp_speed[True-True-True-True-True]	0.3526ms	35.0658μs	28.5178 KOps/s	28.0221 KOps/s	$\color{#35bf28}+1.77\%$
test_step_mdp_speed[True-True-True-True-False]	0.2136ms	20.5628μs	48.6314 KOps/s	47.5833 KOps/s	$\color{#35bf28}+2.20\%$
test_step_mdp_speed[True-True-True-False-True]	55.8310μs	20.2738μs	49.3247 KOps/s	46.7918 KOps/s	$\textbf{\color{#35bf28}+5.41\%}$
test_step_mdp_speed[True-True-True-False-False]	0.2265ms	12.1599μs	82.2377 KOps/s	80.0173 KOps/s	$\color{#35bf28}+2.77\%$
test_step_mdp_speed[True-True-False-True-True]	0.2186ms	36.7112μs	27.2396 KOps/s	26.3280 KOps/s	$\color{#35bf28}+3.46\%$
test_step_mdp_speed[True-True-False-True-False]	53.0210μs	22.5731μs	44.3006 KOps/s	44.0277 KOps/s	$\color{#35bf28}+0.62\%$
test_step_mdp_speed[True-True-False-False-True]	0.1811ms	22.4354μs	44.5725 KOps/s	44.1565 KOps/s	$\color{#35bf28}+0.94\%$
test_step_mdp_speed[True-True-False-False-False]	49.6600μs	14.0727μs	71.0594 KOps/s	69.3704 KOps/s	$\color{#35bf28}+2.43\%$
test_step_mdp_speed[True-False-True-True-True]	69.4210μs	38.8004μs	25.7730 KOps/s	24.8585 KOps/s	$\color{#35bf28}+3.68\%$
test_step_mdp_speed[True-False-True-True-False]	0.2042ms	24.6015μs	40.6479 KOps/s	39.5420 KOps/s	$\color{#35bf28}+2.80\%$
test_step_mdp_speed[True-False-True-False-True]	0.2075ms	22.3171μs	44.8087 KOps/s	42.6228 KOps/s	$\textbf{\color{#35bf28}+5.13\%}$
test_step_mdp_speed[True-False-True-False-False]	51.8610μs	14.1481μs	70.6810 KOps/s	69.8759 KOps/s	$\color{#35bf28}+1.15\%$
test_step_mdp_speed[True-False-False-True-True]	0.2405ms	40.0877μs	24.9453 KOps/s	24.2771 KOps/s	$\color{#35bf28}+2.75\%$
test_step_mdp_speed[True-False-False-True-False]	0.2201ms	26.4826μs	37.7607 KOps/s	38.0738 KOps/s	$\color{#d91a1a}-0.82\%$
test_step_mdp_speed[True-False-False-False-True]	0.2212ms	24.0733μs	41.5398 KOps/s	40.4497 KOps/s	$\color{#35bf28}+2.69\%$
test_step_mdp_speed[True-False-False-False-False]	40.5500μs	15.9072μs	62.8647 KOps/s	61.8626 KOps/s	$\color{#35bf28}+1.62\%$
test_step_mdp_speed[False-True-True-True-True]	0.2241ms	39.0316μs	25.6202 KOps/s	25.1354 KOps/s	$\color{#35bf28}+1.93\%$
test_step_mdp_speed[False-True-True-True-False]	0.2335ms	24.5516μs	40.7306 KOps/s	40.0853 KOps/s	$\color{#35bf28}+1.61\%$
test_step_mdp_speed[False-True-True-False-True]	75.3310μs	27.3024μs	36.6269 KOps/s	36.8471 KOps/s	$\color{#d91a1a}-0.60\%$
test_step_mdp_speed[False-True-True-False-False]	44.1910μs	16.0090μs	62.4650 KOps/s	61.6346 KOps/s	$\color{#35bf28}+1.35\%$
test_step_mdp_speed[False-True-False-True-True]	0.2362ms	40.7604μs	24.5336 KOps/s	24.2233 KOps/s	$\color{#35bf28}+1.28\%$
test_step_mdp_speed[False-True-False-True-False]	56.7510μs	26.3714μs	37.9198 KOps/s	37.3105 KOps/s	$\color{#35bf28}+1.63\%$
test_step_mdp_speed[False-True-False-False-True]	64.8300μs	28.6141μs	34.9479 KOps/s	34.5778 KOps/s	$\color{#35bf28}+1.07\%$
test_step_mdp_speed[False-True-False-False-False]	0.1892ms	17.8097μs	56.1492 KOps/s	54.8225 KOps/s	$\color{#35bf28}+2.42\%$
test_step_mdp_speed[False-False-True-True-True]	0.2386ms	42.7652μs	23.3835 KOps/s	23.0339 KOps/s	$\color{#35bf28}+1.52\%$
test_step_mdp_speed[False-False-True-True-False]	55.5900μs	27.7784μs	35.9992 KOps/s	34.0361 KOps/s	$\textbf{\color{#35bf28}+5.77\%}$
test_step_mdp_speed[False-False-True-False-True]	0.2148ms	28.8509μs	34.6609 KOps/s	34.7637 KOps/s	$\color{#d91a1a}-0.30\%$
test_step_mdp_speed[False-False-True-False-False]	0.1955ms	17.9160μs	55.8159 KOps/s	56.1403 KOps/s	$\color{#d91a1a}-0.58\%$
test_step_mdp_speed[False-False-False-True-True]	0.2571ms	43.4539μs	23.0129 KOps/s	22.7136 KOps/s	$\color{#35bf28}+1.32\%$
test_step_mdp_speed[False-False-False-True-False]	0.1033ms	29.4559μs	33.9491 KOps/s	33.2310 KOps/s	$\color{#35bf28}+2.16\%$
test_step_mdp_speed[False-False-False-False-True]	0.2253ms	29.8521μs	33.4984 KOps/s	33.0126 KOps/s	$\color{#35bf28}+1.47\%$
test_step_mdp_speed[False-False-False-False-False]	0.1784ms	19.4874μs	51.3151 KOps/s	50.5017 KOps/s	$\color{#35bf28}+1.61\%$
test_values[generalized_advantage_estimate-True-True]	25.7322ms	24.8787ms	40.1950 Ops/s	37.2987 Ops/s	$\textbf{\color{#35bf28}+7.77\%}$
test_values[vec_generalized_advantage_estimate-True-True]	86.7903ms	3.2813ms	304.7550 Ops/s	288.1094 Ops/s	$\textbf{\color{#35bf28}+5.78\%}$
test_values[td0_return_estimate-False-False]	0.1198ms	65.8025μs	15.1970 KOps/s	14.8439 KOps/s	$\color{#35bf28}+2.38\%$
test_values[td1_return_estimate-False-False]	55.7890ms	54.7042ms	18.2801 Ops/s	17.3811 Ops/s	$\textbf{\color{#35bf28}+5.17\%}$
test_values[vec_td1_return_estimate-False-False]	1.9953ms	1.7304ms	577.9099 Ops/s	568.2580 Ops/s	$\color{#35bf28}+1.70\%$
test_values[td_lambda_return_estimate-True-False]	89.6442ms	88.5608ms	11.2917 Ops/s	10.9137 Ops/s	$\color{#35bf28}+3.46\%$
test_values[vec_td_lambda_return_estimate-True-False]	2.0242ms	1.7396ms	574.8297 Ops/s	570.1069 Ops/s	$\color{#35bf28}+0.83\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	23.9646ms	23.6952ms	42.2026 Ops/s	38.7856 Ops/s	$\textbf{\color{#35bf28}+8.81\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.8541ms	0.7065ms	1.4155 KOps/s	1.3086 KOps/s	$\textbf{\color{#35bf28}+8.17\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.7346ms	0.6691ms	1.4945 KOps/s	1.4217 KOps/s	$\textbf{\color{#35bf28}+5.12\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.5671ms	1.4720ms	679.3553 Ops/s	671.4915 Ops/s	$\color{#35bf28}+1.17\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.9471ms	0.6896ms	1.4502 KOps/s	1.4503 KOps/s	$-0.01\%$
test_dqn_speed	8.2110ms	1.4772ms	676.9588 Ops/s	672.2787 Ops/s	$\color{#35bf28}+0.70\%$
test_ddpg_speed	4.6844ms	3.2848ms	304.4279 Ops/s	300.0743 Ops/s	$\color{#35bf28}+1.45\%$
test_sac_speed	93.5495ms	9.8887ms	101.1257 Ops/s	108.1565 Ops/s	$\textbf{\color{#d91a1a}-6.50\%}$
test_redq_speed	16.9574ms	16.4748ms	60.6987 Ops/s	59.8460 Ops/s	$\color{#35bf28}+1.42\%$
test_redq_deprec_speed	13.9407ms	12.8568ms	77.7796 Ops/s	77.5732 Ops/s	$\color{#35bf28}+0.27\%$
test_td3_speed	19.1161ms	9.4222ms	106.1318 Ops/s	105.4106 Ops/s	$\color{#35bf28}+0.68\%$
test_cql_speed	33.5290ms	31.3677ms	31.8799 Ops/s	31.7268 Ops/s	$\color{#35bf28}+0.48\%$
test_a2c_speed	8.6396ms	7.1372ms	140.1112 Ops/s	139.7366 Ops/s	$\color{#35bf28}+0.27\%$
test_ppo_speed	8.8173ms	7.4351ms	134.4967 Ops/s	134.3110 Ops/s	$\color{#35bf28}+0.14\%$
test_reinforce_speed	7.6082ms	6.1415ms	162.8254 Ops/s	162.3000 Ops/s	$\color{#35bf28}+0.32\%$
test_iql_speed	28.5847ms	26.8908ms	37.1874 Ops/s	36.9734 Ops/s	$\color{#35bf28}+0.58\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.1198ms	2.5024ms	399.6212 Ops/s	397.6964 Ops/s	$\color{#35bf28}+0.48\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.6920ms	2.6619ms	375.6709 Ops/s	333.0814 Ops/s	$\textbf{\color{#35bf28}+12.79\%}$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	3.7993ms	2.6585ms	376.1553 Ops/s	373.1723 Ops/s	$\color{#35bf28}+0.80\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.1221ms	2.4802ms	403.1941 Ops/s	399.0316 Ops/s	$\color{#35bf28}+1.04\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3.9959ms	2.6658ms	375.1229 Ops/s	371.0064 Ops/s	$\color{#35bf28}+1.11\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3.9025ms	2.6541ms	376.7731 Ops/s	373.7699 Ops/s	$\color{#35bf28}+0.80\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.1344ms	2.4732ms	404.3354 Ops/s	400.4657 Ops/s	$\color{#35bf28}+0.97\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	3.7261ms	2.6561ms	376.4962 Ops/s	372.2895 Ops/s	$\color{#35bf28}+1.13\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	3.8876ms	2.6659ms	375.1009 Ops/s	373.0233 Ops/s	$\color{#35bf28}+0.56\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.9055ms	2.4940ms	400.9594 Ops/s	400.0948 Ops/s	$\color{#35bf28}+0.22\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.9733ms	2.6682ms	374.7857 Ops/s	371.2844 Ops/s	$\color{#35bf28}+0.94\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	3.9331ms	2.6687ms	374.7177 Ops/s	374.7793 Ops/s	$\color{#d91a1a}-0.02\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.7200ms	2.5026ms	399.5775 Ops/s	398.9568 Ops/s	$\color{#35bf28}+0.16\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	4.0899ms	2.6670ms	374.9488 Ops/s	374.1470 Ops/s	$\color{#35bf28}+0.21\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3.7528ms	2.6666ms	375.0074 Ops/s	371.8298 Ops/s	$\color{#35bf28}+0.85\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.0732ms	2.5087ms	398.6055 Ops/s	400.1455 Ops/s	$\color{#d91a1a}-0.38\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	4.2087ms	2.6567ms	376.4097 Ops/s	370.1625 Ops/s	$\color{#35bf28}+1.69\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	3.8634ms	2.6645ms	375.2989 Ops/s	371.2197 Ops/s	$\color{#35bf28}+1.10\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.2083s	19.5992ms	51.0225 Ops/s	50.3940 Ops/s	$\color{#35bf28}+1.25\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	0.1286s	17.8470ms	56.0317 Ops/s	55.3415 Ops/s	$\color{#35bf28}+1.25\%$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	0.1274s	18.0247ms	55.4795 Ops/s	63.5460 Ops/s	$\textbf{\color{#d91a1a}-12.69\%}$
test_populate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1269s	17.9616ms	55.6745 Ops/s	55.5587 Ops/s	$\color{#35bf28}+0.21\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	0.1277s	15.7181ms	63.6208 Ops/s	55.4638 Ops/s	$\textbf{\color{#35bf28}+14.71\%}$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	0.1274s	17.9230ms	55.7941 Ops/s	55.2941 Ops/s	$\color{#35bf28}+0.90\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1288s	17.8888ms	55.9010 Ops/s	55.7224 Ops/s	$\color{#35bf28}+0.32\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	0.1297s	18.0184ms	55.4989 Ops/s	55.3033 Ops/s	$\color{#35bf28}+0.35\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	0.1288s	18.0402ms	55.4316 Ops/s	63.6100 Ops/s	$\textbf{\color{#d91a1a}-12.86\%}$