Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Remove snapshot from CI #701

Merged
merged 1 commit into from
Mar 6, 2024
Merged

[CI] Remove snapshot from CI #701

merged 1 commit into from
Mar 6, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Mar 6, 2024

No description provided.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 6, 2024
Copy link

github-actions bot commented Mar 6, 2024

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 126. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}31$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_plain_set_nested 34.8650μs 16.9245μs 59.0860 KOps/s 61.4705 KOps/s $\color{#d91a1a}-3.88\%$
test_plain_set_stack_nested 42.6490μs 17.4438μs 57.3271 KOps/s 60.5506 KOps/s $\textbf{\color{#d91a1a}-5.32\%}$
test_plain_set_nested_inplace 53.6390μs 19.6957μs 50.7725 KOps/s 53.3270 KOps/s $\color{#d91a1a}-4.79\%$
test_plain_set_stack_nested_inplace 55.1920μs 19.6729μs 50.8312 KOps/s 53.7328 KOps/s $\textbf{\color{#d91a1a}-5.40\%}$
test_items 22.5420μs 2.6245μs 381.0179 KOps/s 400.3725 KOps/s $\color{#d91a1a}-4.83\%$
test_items_nested 0.9626ms 0.2791ms 3.5828 KOps/s 3.6372 KOps/s $\color{#d91a1a}-1.50\%$
test_items_nested_locked 0.4633ms 0.2803ms 3.5677 KOps/s 3.7138 KOps/s $\color{#d91a1a}-3.93\%$
test_items_nested_leaf 0.5966ms 0.1720ms 5.8134 KOps/s 6.0594 KOps/s $\color{#d91a1a}-4.06\%$
test_items_stack_nested 0.5206ms 0.2815ms 3.5525 KOps/s 3.7224 KOps/s $\color{#d91a1a}-4.57\%$
test_items_stack_nested_leaf 0.3014ms 0.1740ms 5.7473 KOps/s 6.1334 KOps/s $\textbf{\color{#d91a1a}-6.30\%}$
test_items_stack_nested_locked 1.0415ms 0.2823ms 3.5420 KOps/s 3.6829 KOps/s $\color{#d91a1a}-3.83\%$
test_keys 27.5310μs 3.9575μs 252.6830 KOps/s 253.1225 KOps/s $\color{#d91a1a}-0.17\%$
test_keys_nested 1.7844ms 0.1466ms 6.8234 KOps/s 6.8178 KOps/s $\color{#35bf28}+0.08\%$
test_keys_nested_locked 0.4147ms 0.1519ms 6.5833 KOps/s 6.6481 KOps/s $\color{#d91a1a}-0.98\%$
test_keys_nested_leaf 39.0920ms 0.1347ms 7.4241 KOps/s 7.8013 KOps/s $\color{#d91a1a}-4.84\%$
test_keys_stack_nested 0.2636ms 0.1462ms 6.8395 KOps/s 6.6111 KOps/s $\color{#35bf28}+3.45\%$
test_keys_stack_nested_leaf 0.2412ms 0.1283ms 7.7932 KOps/s 7.5041 KOps/s $\color{#35bf28}+3.85\%$
test_keys_stack_nested_locked 0.2851ms 0.1522ms 6.5720 KOps/s 6.3714 KOps/s $\color{#35bf28}+3.15\%$
test_values 6.0463μs 1.1941μs 837.4809 KOps/s 808.6413 KOps/s $\color{#35bf28}+3.57\%$
test_values_nested 0.1039ms 51.7086μs 19.3391 KOps/s 18.7056 KOps/s $\color{#35bf28}+3.39\%$
test_values_nested_locked 0.1302ms 51.5493μs 19.3989 KOps/s 19.1890 KOps/s $\color{#35bf28}+1.09\%$
test_values_nested_leaf 0.1039ms 46.0960μs 21.6938 KOps/s 21.6959 KOps/s $-0.01\%$
test_values_stack_nested 0.1476ms 52.1171μs 19.1876 KOps/s 18.7670 KOps/s $\color{#35bf28}+2.24\%$
test_values_stack_nested_leaf 91.3790μs 46.0885μs 21.6974 KOps/s 21.6698 KOps/s $\color{#35bf28}+0.13\%$
test_values_stack_nested_locked 0.1014ms 52.0621μs 19.2078 KOps/s 18.8536 KOps/s $\color{#35bf28}+1.88\%$
test_membership 16.2900μs 1.3330μs 750.1960 KOps/s 769.5654 KOps/s $\color{#d91a1a}-2.52\%$
test_membership_nested 37.3400μs 3.7659μs 265.5412 KOps/s 296.3606 KOps/s $\textbf{\color{#d91a1a}-10.40\%}$
test_membership_nested_leaf 23.6940μs 3.5732μs 279.8593 KOps/s 297.6569 KOps/s $\textbf{\color{#d91a1a}-5.98\%}$
test_membership_stacked_nested 28.1130μs 3.5710μs 280.0370 KOps/s 275.8469 KOps/s $\color{#35bf28}+1.52\%$
test_membership_stacked_nested_leaf 25.9980μs 3.5393μs 282.5403 KOps/s 288.8312 KOps/s $\color{#d91a1a}-2.18\%$
test_membership_nested_last 28.4630μs 4.3614μs 229.2816 KOps/s 234.8589 KOps/s $\color{#d91a1a}-2.37\%$
test_membership_nested_leaf_last 23.8040μs 4.4254μs 225.9698 KOps/s 235.5334 KOps/s $\color{#d91a1a}-4.06\%$
test_membership_stacked_nested_last 23.6630μs 4.3092μs 232.0622 KOps/s 76.0145 KOps/s $\textbf{\color{#35bf28}+205.29\%}$
test_membership_stacked_nested_leaf_last 24.0040μs 4.3664μs 229.0236 KOps/s 75.8753 KOps/s $\textbf{\color{#35bf28}+201.84\%}$
test_nested_getleaf 30.7370μs 10.7352μs 93.1517 KOps/s 95.9834 KOps/s $\color{#d91a1a}-2.95\%$
test_nested_get 29.7450μs 10.1683μs 98.3452 KOps/s 100.7505 KOps/s $\color{#d91a1a}-2.39\%$
test_stacked_getleaf 30.4660μs 10.6150μs 94.2064 KOps/s 97.3275 KOps/s $\color{#d91a1a}-3.21\%$
test_stacked_get 27.9220μs 10.0631μs 99.3729 KOps/s 103.0406 KOps/s $\color{#d91a1a}-3.56\%$
test_nested_getitemleaf 31.3280μs 11.1145μs 89.9723 KOps/s 93.1505 KOps/s $\color{#d91a1a}-3.41\%$
test_nested_getitem 29.7250μs 10.5774μs 94.5412 KOps/s 97.4228 KOps/s $\color{#d91a1a}-2.96\%$
test_stacked_getitemleaf 35.1050μs 11.1658μs 89.5595 KOps/s 93.5130 KOps/s $\color{#d91a1a}-4.23\%$
test_stacked_getitem 32.2490μs 10.4640μs 95.5659 KOps/s 94.9412 KOps/s $\color{#35bf28}+0.66\%$
test_lock_nested 0.7632ms 0.3405ms 2.9369 KOps/s 3.0210 KOps/s $\color{#d91a1a}-2.78\%$
test_lock_stack_nested 0.4456ms 0.3057ms 3.2715 KOps/s 3.4613 KOps/s $\textbf{\color{#d91a1a}-5.48\%}$
test_unlock_nested 81.1406ms 0.4218ms 2.3706 KOps/s 2.3310 KOps/s $\color{#35bf28}+1.70\%$
test_unlock_stack_nested 0.5390ms 0.3139ms 3.1857 KOps/s 3.3568 KOps/s $\textbf{\color{#d91a1a}-5.10\%}$
test_flatten_speed 0.6587ms 0.2799ms 3.5721 KOps/s 3.6357 KOps/s $\color{#d91a1a}-1.75\%$
test_unflatten_speed 0.7130ms 0.4152ms 2.4085 KOps/s 2.4861 KOps/s $\color{#d91a1a}-3.12\%$
test_common_ops 1.2664ms 0.7058ms 1.4168 KOps/s 1.4965 KOps/s $\textbf{\color{#d91a1a}-5.32\%}$
test_creation 63.3070μs 1.9176μs 521.4910 KOps/s 554.6130 KOps/s $\textbf{\color{#d91a1a}-5.97\%}$
test_creation_empty 39.7840μs 10.6947μs 93.5041 KOps/s 112.1565 KOps/s $\textbf{\color{#d91a1a}-16.63\%}$
test_creation_nested_1 0.1045ms 13.5254μs 73.9350 KOps/s 87.2831 KOps/s $\textbf{\color{#d91a1a}-15.29\%}$
test_creation_nested_2 73.2060μs 16.6222μs 60.1605 KOps/s 68.7542 KOps/s $\textbf{\color{#d91a1a}-12.50\%}$
test_clone 63.1570μs 13.1789μs 75.8790 KOps/s 75.7685 KOps/s $\color{#35bf28}+0.15\%$
test_getitem[int] 39.0320μs 11.0261μs 90.6938 KOps/s 92.3570 KOps/s $\color{#d91a1a}-1.80\%$
test_getitem[slice_int] 69.4990μs 22.1785μs 45.0887 KOps/s 46.0319 KOps/s $\color{#d91a1a}-2.05\%$
test_getitem[range] 0.2422ms 42.6397μs 23.4523 KOps/s 24.6562 KOps/s $\color{#d91a1a}-4.88\%$
test_getitem[tuple] 47.5780μs 18.2203μs 54.8838 KOps/s 55.6421 KOps/s $\color{#d91a1a}-1.36\%$
test_getitem[list] 0.2134ms 37.4616μs 26.6940 KOps/s 27.4748 KOps/s $\color{#d91a1a}-2.84\%$
test_setitem_dim[int] 65.0210μs 34.2284μs 29.2155 KOps/s 31.0147 KOps/s $\textbf{\color{#d91a1a}-5.80\%}$
test_setitem_dim[slice_int] 0.1256ms 61.6681μs 16.2158 KOps/s 17.1416 KOps/s $\textbf{\color{#d91a1a}-5.40\%}$
test_setitem_dim[range] 0.1388ms 78.8145μs 12.6880 KOps/s 12.9512 KOps/s $\color{#d91a1a}-2.03\%$
test_setitem_dim[tuple] 94.1450μs 51.4395μs 19.4403 KOps/s 21.1007 KOps/s $\textbf{\color{#d91a1a}-7.87\%}$
test_setitem 80.7500μs 20.2005μs 49.5038 KOps/s 53.2549 KOps/s $\textbf{\color{#d91a1a}-7.04\%}$
test_set 81.4210μs 19.5735μs 51.0895 KOps/s 55.3447 KOps/s $\textbf{\color{#d91a1a}-7.69\%}$
test_set_shared 5.6427ms 0.1411ms 7.0881 KOps/s 7.0227 KOps/s $\color{#35bf28}+0.93\%$
test_update 0.1732ms 22.2023μs 45.0404 KOps/s 48.9399 KOps/s $\textbf{\color{#d91a1a}-7.97\%}$
test_update_nested 0.1482ms 31.3409μs 31.9072 KOps/s 36.2564 KOps/s $\textbf{\color{#d91a1a}-12.00\%}$
test_set_nested 82.1630μs 21.5887μs 46.3205 KOps/s 48.9883 KOps/s $\textbf{\color{#d91a1a}-5.45\%}$
test_set_nested_new 83.1740μs 25.2700μs 39.5725 KOps/s 42.5085 KOps/s $\textbf{\color{#d91a1a}-6.91\%}$
test_select 0.1428ms 40.8692μs 24.4683 KOps/s 26.5008 KOps/s $\textbf{\color{#d91a1a}-7.67\%}$
test_select_nested 0.1224ms 60.5368μs 16.5189 KOps/s 17.2060 KOps/s $\color{#d91a1a}-3.99\%$
test_exclude_nested 0.2000ms 0.1200ms 8.3333 KOps/s 8.5756 KOps/s $\color{#d91a1a}-2.83\%$
test_empty[True] 0.7193ms 0.4199ms 2.3817 KOps/s 2.5252 KOps/s $\textbf{\color{#d91a1a}-5.68\%}$
test_empty[False] 8.8024μs 1.0591μs 944.2287 KOps/s 983.7128 KOps/s $\color{#d91a1a}-4.01\%$
test_unbind_speed 0.3185ms 0.2497ms 4.0046 KOps/s 4.1803 KOps/s $\color{#d91a1a}-4.20\%$
test_unbind_speed_stack0 0.4865ms 0.2478ms 4.0362 KOps/s 4.2928 KOps/s $\textbf{\color{#d91a1a}-5.98\%}$
test_unbind_speed_stack1 0.1223s 0.6800ms 1.4706 KOps/s 1.5222 KOps/s $\color{#d91a1a}-3.39\%$
test_split 0.1181s 1.6502ms 605.9746 Ops/s 620.1865 Ops/s $\color{#d91a1a}-2.29\%$
test_chunk 1.6952ms 1.4594ms 685.2238 Ops/s 679.7650 Ops/s $\color{#35bf28}+0.80\%$
test_creation[device0] 0.1897ms 0.1019ms 9.8118 KOps/s 9.4406 KOps/s $\color{#35bf28}+3.93\%$
test_creation_from_tensor 5.2207ms 81.9643μs 12.2004 KOps/s 12.2641 KOps/s $\color{#d91a1a}-0.52\%$
test_add_one[memmap_tensor0] 0.1334ms 5.6530μs 176.8975 KOps/s 189.8544 KOps/s $\textbf{\color{#d91a1a}-6.82\%}$
test_contiguous[memmap_tensor0] 8.1650μs 0.6246μs 1.6010 MOps/s 1.5681 MOps/s $\color{#35bf28}+2.10\%$
test_stack[memmap_tensor0] 32.1390μs 3.5911μs 278.4697 KOps/s 277.2245 KOps/s $\color{#35bf28}+0.45\%$
test_memmaptd_index 0.9328ms 0.2365ms 4.2291 KOps/s 4.2320 KOps/s $\color{#d91a1a}-0.07\%$
test_memmaptd_index_astensor 1.0722ms 0.3064ms 3.2633 KOps/s 3.3719 KOps/s $\color{#d91a1a}-3.22\%$
test_memmaptd_index_op 0.8702ms 0.6085ms 1.6433 KOps/s 1.7865 KOps/s $\textbf{\color{#d91a1a}-8.01\%}$
test_serialize_model 0.2261s 0.1160s 8.6218 Ops/s 8.3024 Ops/s $\color{#35bf28}+3.85\%$
test_serialize_model_pickle 0.4474s 0.3778s 2.6471 Ops/s 2.6359 Ops/s $\color{#35bf28}+0.43\%$
test_serialize_weights 0.1082s 98.6085ms 10.1411 Ops/s 9.9284 Ops/s $\color{#35bf28}+2.14\%$
test_serialize_weights_returnearly 0.2470s 0.1360s 7.3520 Ops/s 6.9650 Ops/s $\textbf{\color{#35bf28}+5.56\%}$
test_serialize_weights_pickle 0.7501s 0.4862s 2.0568 Ops/s 2.5557 Ops/s $\textbf{\color{#d91a1a}-19.52\%}$
test_serialize_weights_filesystem 0.1011s 91.7631ms 10.8976 Ops/s 10.4973 Ops/s $\color{#35bf28}+3.81\%$
test_serialize_model_filesystem 98.0656ms 92.1014ms 10.8576 Ops/s 10.5884 Ops/s $\color{#35bf28}+2.54\%$
test_reshape_pytree 69.9700μs 21.7589μs 45.9582 KOps/s 47.6161 KOps/s $\color{#d91a1a}-3.48\%$
test_reshape_td 86.1100μs 32.0089μs 31.2413 KOps/s 32.7650 KOps/s $\color{#d91a1a}-4.65\%$
test_view_pytree 52.0570μs 21.2523μs 47.0538 KOps/s 48.6406 KOps/s $\color{#d91a1a}-3.26\%$
test_view_td 0.1177s 62.3108μs 16.0486 KOps/s 16.5513 KOps/s $\color{#d91a1a}-3.04\%$
test_unbind_pytree 55.5530μs 25.0607μs 39.9032 KOps/s 41.2186 KOps/s $\color{#d91a1a}-3.19\%$
test_unbind_td 0.1244ms 36.1212μs 27.6846 KOps/s 27.7309 KOps/s $\color{#d91a1a}-0.17\%$
test_split_pytree 54.9320μs 24.3656μs 41.0414 KOps/s 42.0587 KOps/s $\color{#d91a1a}-2.42\%$
test_split_td 0.6233ms 40.1411μs 24.9121 KOps/s 26.0545 KOps/s $\color{#d91a1a}-4.38\%$
test_add_pytree 80.5990μs 30.3646μs 32.9331 KOps/s 34.2063 KOps/s $\color{#d91a1a}-3.72\%$
test_add_td 0.1794ms 55.5412μs 18.0047 KOps/s 19.5974 KOps/s $\textbf{\color{#d91a1a}-8.13\%}$
test_distributed 0.2661ms 0.1027ms 9.7368 KOps/s 9.8230 KOps/s $\color{#d91a1a}-0.88\%$
test_tdmodule 96.7690μs 17.5807μs 56.8805 KOps/s 62.3454 KOps/s $\textbf{\color{#d91a1a}-8.77\%}$
test_tdmodule_dispatch 50.4840μs 32.9313μs 30.3662 KOps/s 32.3265 KOps/s $\textbf{\color{#d91a1a}-6.06\%}$
test_tdseq 36.0970μs 20.0013μs 49.9966 KOps/s 52.4259 KOps/s $\color{#d91a1a}-4.63\%$
test_tdseq_dispatch 58.5180μs 38.6234μs 25.8911 KOps/s 27.6212 KOps/s $\textbf{\color{#d91a1a}-6.26\%}$
test_instantiation_functorch 1.8685ms 1.3286ms 752.6823 Ops/s 763.1361 Ops/s $\color{#d91a1a}-1.37\%$
test_instantiation_td 1.9572ms 1.0445ms 957.4264 Ops/s 1.0024 KOps/s $\color{#d91a1a}-4.49\%$
test_exec_functorch 0.3744ms 0.1598ms 6.2578 KOps/s 6.3676 KOps/s $\color{#d91a1a}-1.72\%$
test_exec_functional_call 0.3265ms 0.1501ms 6.6637 KOps/s 6.7204 KOps/s $\color{#d91a1a}-0.84\%$
test_exec_td 0.2336ms 0.1424ms 7.0220 KOps/s 6.9242 KOps/s $\color{#35bf28}+1.41\%$
test_exec_td_decorator 0.6256ms 0.1992ms 5.0200 KOps/s 4.9803 KOps/s $\color{#35bf28}+0.80\%$
test_vmap_mlp_speed[True-True] 0.6248ms 0.4691ms 2.1318 KOps/s 2.1094 KOps/s $\color{#35bf28}+1.06\%$
test_vmap_mlp_speed[True-False] 0.7196ms 0.4696ms 2.1294 KOps/s 2.1450 KOps/s $\color{#d91a1a}-0.73\%$
test_vmap_mlp_speed[False-True] 0.6234ms 0.3796ms 2.6343 KOps/s 2.5678 KOps/s $\color{#35bf28}+2.59\%$
test_vmap_mlp_speed[False-False] 0.6176ms 0.3802ms 2.6303 KOps/s 2.5945 KOps/s $\color{#35bf28}+1.38\%$
test_vmap_mlp_speed_decorator[True-True] 1.0386ms 0.4886ms 2.0466 KOps/s 2.0356 KOps/s $\color{#35bf28}+0.54\%$
test_vmap_mlp_speed_decorator[True-False] 0.7792ms 0.4845ms 2.0640 KOps/s 1.9945 KOps/s $\color{#35bf28}+3.49\%$
test_vmap_mlp_speed_decorator[False-True] 0.6013ms 0.3975ms 2.5155 KOps/s 2.4500 KOps/s $\color{#35bf28}+2.68\%$
test_vmap_mlp_speed_decorator[False-False] 0.7730ms 0.3985ms 2.5095 KOps/s 2.5045 KOps/s $\color{#35bf28}+0.20\%$
test_to_module_speed[True] 2.1548ms 1.4080ms 710.2268 Ops/s 736.7110 Ops/s $\color{#d91a1a}-3.59\%$
test_to_module_speed[False] 2.1620ms 1.3919ms 718.4333 Ops/s 747.9422 Ops/s $\color{#d91a1a}-3.95\%$

@vmoens vmoens marked this pull request as ready for review March 6, 2024 11:44
@vmoens vmoens added the CI label Mar 6, 2024
@vmoens vmoens merged commit 2d01ecb into main Mar 6, 2024
46 of 48 checks passed
@vmoens vmoens deleted the deprec_snapshot_tests branch March 6, 2024 11:47
vmoens added a commit that referenced this pull request Mar 24, 2024
vmoens added a commit that referenced this pull request Mar 24, 2024
(cherry picked from commit 2d01ecb)
vmoens added a commit that referenced this pull request Mar 25, 2024
(cherry picked from commit 2d01ecb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants