Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use simdjson for binary json construction for improved performance #10464

Merged
merged 27 commits into from
Oct 21, 2024

Conversation

swalrus1
Copy link
Collaborator

@swalrus1 swalrus1 commented Oct 15, 2024

Use simdjson as a backend for conversion from string to binary json.

benchmarks show 220% performance improvement of binary json construction

-------------------------------------------------------------
Benchmark                   Time             CPU   Iterations
-------------------------------------------------------------
BenchWriteRapidJson      46687 ns        46687 ns       151833

-------------------------------------------------------------
Benchmark                   Time             CPU   Iterations
-------------------------------------------------------------
BenchWriteSimdJson      21247 ns        21247 ns       330340

Copy link

github-actions bot commented Oct 15, 2024

2024-10-15 15:20:45 UTC Pre-commit check linux-x86_64-relwithdebinfo for 17a887f has started.
2024-10-15 15:20:55 UTC Artifacts will be uploaded here
2024-10-15 15:24:41 UTC ya make is running...
🔴 2024-10-15 15:47:54 UTC Build failed, see the logs.

Copy link

github-actions bot commented Oct 15, 2024

2024-10-15 15:21:01 UTC Pre-commit check linux-x86_64-release-asan for 17a887f has started.
2024-10-15 15:21:11 UTC Artifacts will be uploaded here
2024-10-15 15:24:58 UTC ya make is running...
🔴 2024-10-15 15:47:28 UTC Build failed, see the logs.

TMemoryInput input(json.data(), json.size());
TBinaryJsonCallbacks callbacks(/* throwException */ false);
if (!ReadJson(&input, &callbacks)) {
simdjson::dom::parser parser;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, ondemand should be faster than dom

Copy link
Collaborator Author

@swalrus1 swalrus1 Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on benchmark of SerializeToBinaryJson, both performed very closely, so I'm keeping dom for simpler API: https://gist.github.com/swalrus1/b3db37c1574756ff037349a91d9b155a

Copy link

github-actions bot commented Oct 15, 2024

2024-10-15 16:13:01 UTC Pre-commit check linux-x86_64-relwithdebinfo for 9a25bc2 has started.
2024-10-15 16:13:12 UTC Artifacts will be uploaded here
2024-10-15 16:17:00 UTC ya make is running...
🔴 2024-10-15 17:39:41 UTC Too many tests failed, NOT going to retry
🔴 2024-10-15 17:40:57 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
79619 64417 0 828 14267 107

🟢 2024-10-15 17:48:34 UTC Build successful.
🟢 2024-10-15 17:48:58 UTC ydbd size 2.8 GiB changed* by +9.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 3ab7dd7 merge: 9a25bc2 diff diff %
ydbd size 3 010 665 048 Bytes 3 010 674 936 Bytes +9.7 KiB +0.000%
ydbd stripped size 477 552 856 Bytes 477 555 032 Bytes +2.1 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 15, 2024

2024-10-15 16:13:03 UTC Pre-commit check linux-x86_64-release-asan for 9a25bc2 has started.
2024-10-15 16:13:14 UTC Artifacts will be uploaded here
2024-10-15 16:16:58 UTC ya make is running...
🟡 2024-10-15 18:07:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16710 15809 0 318 543 40

🟢 2024-10-15 18:09:25 UTC Build successful.
🟢 2024-10-15 18:10:00 UTC ydbd size 5.7 GiB changed* by +17.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 3ab7dd7 merge: 9a25bc2 diff diff %
ydbd size 6 099 382 840 Bytes 6 099 401 072 Bytes +17.8 KiB +0.000%
ydbd stripped size 1 521 054 992 Bytes 1 521 060 176 Bytes +5.1 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 10:47:27 UTC Pre-commit check linux-x86_64-release-asan for e4cd492 has started.
2024-10-16 10:47:38 UTC Artifacts will be uploaded here
2024-10-16 10:48:32 UTC Check cancelled

@swalrus1 swalrus1 force-pushed the experiment-simdjson-binary branch from 442dc83 to 50347d8 Compare October 16, 2024 10:48
Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 10:51:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4fd7248 has started.
2024-10-16 10:51:38 UTC Artifacts will be uploaded here
2024-10-16 10:54:49 UTC Check cancelled

Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 10:53:15 UTC Pre-commit check linux-x86_64-release-asan for 4fd7248 has started.
2024-10-16 10:53:26 UTC Artifacts will be uploaded here
2024-10-16 10:54:50 UTC Check cancelled

Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 10:57:48 UTC Pre-commit check linux-x86_64-release-asan for aac3696 has started.
2024-10-16 10:58:00 UTC Artifacts will be uploaded here
2024-10-16 11:01:44 UTC ya make is running...
🔴 2024-10-16 11:07:23 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 10:58:01 UTC Pre-commit check linux-x86_64-relwithdebinfo for aac3696 has started.
2024-10-16 10:58:44 UTC Artifacts will be uploaded here
2024-10-16 11:02:59 UTC ya make is running...
🔴 2024-10-16 11:09:20 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 11:31:53 UTC Pre-commit check linux-x86_64-release-asan for d98dfdd has started.
2024-10-16 11:32:27 UTC Artifacts will be uploaded here
2024-10-16 11:36:38 UTC ya make is running...
🔴 2024-10-16 11:43:22 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 11:32:45 UTC Pre-commit check linux-x86_64-relwithdebinfo for d98dfdd has started.
2024-10-16 11:32:57 UTC Artifacts will be uploaded here
2024-10-16 11:36:45 UTC ya make is running...
🔴 2024-10-16 11:40:04 UTC Build failed, see the logs. Also see fail summary

@swalrus1 swalrus1 changed the title use simdjson in SerializeToBinaryJson Use simdjson for binary json construction for improved performance Oct 16, 2024
Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 12:43:11 UTC Pre-commit check linux-x86_64-release-asan for a4cdb4e has started.
2024-10-16 12:43:46 UTC Artifacts will be uploaded here
2024-10-16 12:48:04 UTC ya make is running...
🔴 2024-10-16 12:55:28 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 12:45:10 UTC Pre-commit check linux-x86_64-relwithdebinfo for a4cdb4e has started.
2024-10-16 12:45:21 UTC Artifacts will be uploaded here
2024-10-16 12:49:05 UTC ya make is running...
🔴 2024-10-16 13:09:59 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 13:56:38 UTC Pre-commit check linux-x86_64-relwithdebinfo for e22315f has started.
2024-10-16 13:57:09 UTC Artifacts will be uploaded here
2024-10-16 14:01:24 UTC ya make is running...
2024-10-16 14:42:36 UTC Check cancelled

Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 13:59:58 UTC Pre-commit check linux-x86_64-release-asan for e22315f has started.
2024-10-16 14:00:30 UTC Artifacts will be uploaded here
2024-10-16 14:04:58 UTC ya make is running...
2024-10-16 14:42:47 UTC Check cancelled

@swalrus1 swalrus1 force-pushed the experiment-simdjson-binary branch from ee6b1e3 to e17c42a Compare October 16, 2024 14:42
Copy link

github-actions bot commented Oct 16, 2024

2024-10-16 15:11:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0c83a9a has started.
2024-10-16 15:12:16 UTC Artifacts will be uploaded here
2024-10-16 15:16:37 UTC ya make is running...
🟡 2024-10-16 16:24:45 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
79602 65616 0 12 13871 103

2024-10-16 16:32:46 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-16 16:44:40 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
114 (only retried tests) 12 0 2 1 99

2024-10-16 16:44:48 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-16 16:56:06 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
101 (only retried tests) 1 0 2 0 98

🟢 2024-10-16 16:56:13 UTC Build successful.
🟢 2024-10-16 16:56:35 UTC ydbd size 2.8 GiB changed* by +62.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 8739de4 merge: 0c83a9a diff diff %
ydbd size 3 028 486 472 Bytes 3 028 550 448 Bytes +62.5 KiB +0.002%
ydbd stripped size 480 129 624 Bytes 480 149 080 Bytes +19.0 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Oct 20, 2024
Copy link

github-actions bot commented Oct 20, 2024

2024-10-20 16:18:45 UTC Pre-commit check linux-x86_64-release-asan for 2536799 has started.
2024-10-20 16:18:56 UTC Artifacts will be uploaded here
2024-10-20 16:22:47 UTC ya make is running...
🟡 2024-10-20 18:12:44 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16624 16140 0 152 214 118

🟢 2024-10-20 18:14:08 UTC Build successful.
🟡 2024-10-20 18:14:46 UTC ydbd size 5.7 GiB changed* by +232.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 07f2e60 merge: 2536799 diff diff %
ydbd size 6 108 544 368 Bytes 6 108 782 488 Bytes +232.5 KiB +0.004%
ydbd stripped size 1 522 659 568 Bytes 1 522 703 216 Bytes +42.6 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 20, 2024

2024-10-20 16:18:58 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2536799 has started.
2024-10-20 16:19:10 UTC Artifacts will be uploaded here
2024-10-20 16:23:09 UTC ya make is running...
🟡 2024-10-20 17:26:08 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
79588 65581 0 4 13904 99

2024-10-20 17:33:42 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-20 17:45:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
102 (only retried tests) 8 0 1 0 93

2024-10-20 17:45:23 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-10-20 17:56:38 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
94 (only retried tests) 0 0 1 0 93

🟢 2024-10-20 17:56:45 UTC Build successful.
🟡 2024-10-20 17:57:15 UTC ydbd size 2.8 GiB changed* by +337.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: e0b481c merge: 2536799 diff diff %
ydbd size 3 015 151 392 Bytes 3 015 497 376 Bytes +337.9 KiB +0.011%
ydbd stripped size 477 957 592 Bytes 477 985 368 Bytes +27.1 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 08:01:40 UTC Pre-commit check linux-x86_64-release-asan for e00d7b5 has started.
2024-10-21 08:01:51 UTC Artifacts will be uploaded here
2024-10-21 08:05:42 UTC ya make is running...
🟡 2024-10-21 10:08:08 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16623 16015 0 161 401 46

🟢 2024-10-21 10:09:33 UTC Build successful.
🟡 2024-10-21 10:10:09 UTC ydbd size 5.7 GiB changed* by +251.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 5c4d7fa merge: e00d7b5 diff diff %
ydbd size 6 108 606 984 Bytes 6 108 864 528 Bytes +251.5 KiB +0.004%
ydbd stripped size 1 522 673 392 Bytes 1 522 712 944 Bytes +38.6 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 08:08:30 UTC Pre-commit check linux-x86_64-relwithdebinfo for e00d7b5 has started.
2024-10-21 08:08:41 UTC Artifacts will be uploaded here
2024-10-21 08:12:32 UTC ya make is running...
🟢 2024-10-21 09:18:12 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
79591 65590 0 0 13904 97

🟢 2024-10-21 09:25:37 UTC Build successful.
🟠 2024-10-21 09:25:48 UTC Muted new 1 tests
🟡 2024-10-21 09:26:05 UTC ydbd size 2.8 GiB changed* by +393.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 7e95d38 merge: e00d7b5 diff diff %
ydbd size 3 015 151 392 Bytes 3 015 553 952 Bytes +393.1 KiB +0.013%
ydbd stripped size 477 957 592 Bytes 477 988 888 Bytes +30.6 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 10:13:15 UTC Pre-commit check linux-x86_64-release-asan for d6a0a2a has started.
2024-10-21 10:13:20 UTC Artifacts will be uploaded here
🔴 2024-10-21 10:15:16 UTC Graph compare failed, see the logs.

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 10:14:25 UTC Pre-commit check linux-x86_64-relwithdebinfo for d6a0a2a has started.
2024-10-21 10:14:30 UTC Artifacts will be uploaded here
🔴 2024-10-21 10:16:35 UTC Graph compare failed, see the logs.

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 10:46:38 UTC Pre-commit check linux-x86_64-relwithdebinfo for 68eb16c has started.
2024-10-21 10:46:49 UTC Artifacts will be uploaded here
🔴 2024-10-21 10:48:47 UTC Graph compare failed, see the logs.

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 10:48:03 UTC Pre-commit check linux-x86_64-release-asan for 68eb16c has started.
2024-10-21 10:48:14 UTC Artifacts will be uploaded here
🔴 2024-10-21 10:50:12 UTC Graph compare failed, see the logs.

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 11:01:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for d211940 has started.
2024-10-21 11:02:09 UTC Artifacts will be uploaded here
2024-10-21 11:06:13 UTC ya make is running...
🔴 2024-10-21 11:13:45 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 11:02:27 UTC Pre-commit check linux-x86_64-release-asan for d211940 has started.
2024-10-21 11:02:39 UTC Artifacts will be uploaded here
2024-10-21 11:06:33 UTC ya make is running...
🔴 2024-10-21 11:12:28 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 11:21:52 UTC Pre-commit check linux-x86_64-relwithdebinfo for d23f625 has started.
2024-10-21 11:22:02 UTC Artifacts will be uploaded here
2024-10-21 11:25:54 UTC ya make is running...
🟡 2024-10-21 12:52:39 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
79593 65575 0 4 13917 97

2024-10-21 13:03:50 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-21 13:15:38 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
112 (only retried tests) 19 0 0 0 93

🟢 2024-10-21 13:15:46 UTC Build successful.
🟠 2024-10-21 13:16:09 UTC Muted new 1 tests
🟡 2024-10-21 13:16:29 UTC ydbd size 2.8 GiB changed* by +335.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: e6f71f0 merge: d23f625 diff diff %
ydbd size 3 015 313 344 Bytes 3 015 656 720 Bytes +335.3 KiB +0.011%
ydbd stripped size 477 978 264 Bytes 478 008 216 Bytes +29.2 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 21, 2024

2024-10-21 11:22:00 UTC Pre-commit check linux-x86_64-release-asan for d23f625 has started.
2024-10-21 11:22:12 UTC Artifacts will be uploaded here
2024-10-21 11:26:04 UTC ya make is running...
🟡 2024-10-21 13:19:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16575 16395 0 36 46 98

🟢 2024-10-21 13:20:45 UTC Build successful.
🟡 2024-10-21 13:21:22 UTC ydbd size 5.7 GiB changed* by +299.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: e6f71f0 merge: d23f625 diff diff %
ydbd size 6 108 702 688 Bytes 6 109 009 384 Bytes +299.5 KiB +0.005%
ydbd stripped size 1 522 711 760 Bytes 1 522 759 984 Bytes +47.1 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@swalrus1 swalrus1 merged commit 25311e4 into ydb-platform:main Oct 21, 2024
10 checks passed
@swalrus1 swalrus1 deleted the experiment-simdjson-binary branch October 21, 2024 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use simdjson for binary json construction for improved performance
4 participants