incremental rebuilds of Node.js are slow #47984

kvakil · 2023-05-12T22:02:18Z

What is the problem this feature will solve?

Incremental rebuilds of Node.js are slow. This is annoying for developers, especially those with less powerful machines.

I am fortunate enough to have a very powerful machine, and it's still pretty slow for me. It takes 22 seconds to do an incremental rebuild where I've changed a single Javascript file:

$ TERM=dumb ninja -j 1 -C out/Release node | ts -i '%.s'
0.020594 ninja: Entering directory `out/Release'
1.410648 [1/12] ACTION libnode: node_js2c_1613b577312726d6acf41fb14bd20601
0.001637 [2/12] STAMP obj/libnode.actions_rules_copies.stamp
9.011218 [3/12] CXX obj/gen/libnode.node_javascript.o
0.101600 [4/12] AR obj/libnode.a
0.001407 [5/12] STAMP obj/node_mksnapshot.compile_depends.stamp
0.001513 [6/12] STAMP obj/node_mksnapshot.actions_depends.stamp
2.838031 [7/12] LINK node_mksnapshot
0.001629 [8/12] STAMP obj/node.actions_depends.stamp
0.283595 [9/12] ACTION node: node_mksnapshot_9b7a2d2290b02e76d66661df74749f56
0.001580 [10/12] STAMP obj/node.actions_rules_copies.stamp
5.806897 [11/12] CXX obj/gen/node.node_snapshot.o
2.939010 [12/12] LINK node

What is the feature you are proposing to solve the problem?

Not sure, some ideas:

For gen/node_javascript.cc and gen/node_snapshot.cc, we have a bunch of large generated C++ array literals, which is known to be slow. Currently compiling those two files takes 9s + 6s ~ 15s on my powerful machine. I hacked together a version which makes js2c and node_mksnapshot generate the data via inline assembly, and it made that incremental recompile ~2x as fast (12s):

$ TERM=dumb ninja -j 1 -C out/Release node | ts -i '%.s'
0.020747 ninja: Entering directory `out/Release'
0.213469 [1/12] ACTION libnode: node_js2c_1613b577312726d6acf41fb14bd20601
0.001563 [2/12] STAMP obj/libnode.actions_rules_copies.stamp
3.576886 [3/12] CXX obj/gen/libnode.node_javascript.o
0.102211 [4/12] AR obj/libnode.a
0.002206 [5/12] STAMP obj/node_mksnapshot.compile_depends.stamp
0.001503 [6/12] STAMP obj/node_mksnapshot.actions_depends.stamp
2.811231 [7/12] LINK node_mksnapshot
0.001377 [8/12] STAMP obj/node.actions_depends.stamp
0.191197 [9/12] ACTION node: node_mksnapshot_9b7a2d2290b02e76d66661df74749f56
0.001393 [10/12] STAMP obj/node.actions_rules_copies.stamp
2.070298 [11/12] CXX obj/gen/node.node_snapshot.o
2.889826 [12/12] LINK node

I plan to upstream this change, at least for gcc/Linux. However this is platform-dependent & we'd likely need different inline assembly for each platform (or use an existing tool that abstracts this, like incbin.h).

We could try having postject inject the js2c files/snapshot data. You could also imagine building a single node executable rather than node_mksnapshot/node, and using postject to inject "hydrate" an existing node with the generated snapshot data. This could avoid doing the linking twice. It would also let us centralize on a single path for snapshot deserialization, however it may regress startup performance/memory as we'd need to parse the snapshot into our in-memory format.

And perhaps -fvisibility=hidden would help with link times? Not sure.

What alternatives have you considered?

There are some ways to improve incremental rebuild time for specific sorts of changes (like --node-builtin-modules-path or ccache), but ideally it wouldn't be slow.

The text was updated successfully, but these errors were encountered:

anonrig · 2023-05-20T20:48:03Z

It definitely got worse in the past couple of weeks. Would you be interested in opening a PR to Node @kvakil?

kvakil · 2023-05-21T00:01:50Z

@anonrig I'm waiting for Joyee to land #46997 (since it will let me re-use some code between node_snapshot.cc and js2c, & it logically conflicts with these improvements). Please review that if you can :)

anonrig · 2023-05-21T00:24:33Z

Might be related to #48090. I opened a new issue.

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 9 seconds and 830 MB of memory to compile `gen/node_javascript.cc` with g++. This is the longest step when recompiling a small change to a Javascript file. `gen/node_javascript.cc` contains a lot of large binary literals of our Javascript source code. It is well-known that embedding large binary literals as C/C++ arrays is slow. One workaround is to include the data as string literals instead. This is particularly nice for the Javascript included via js2c, which look better as string literals anyway. Add a new flag `--use-string-literals` to js2c. When this flag is set, we emit string literals instead of array literals, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = R"JS2C1b732aee(...)JS2C1b732aee"; // old: static const uint16_t Y[] = { ... }; static const uint16_t *Y = uR"JS2C1b732aee(...)JS2C1b732aee"; ``` This requires some modest refactoring in order to deal with the flag being on or off, but the new code itself is actually shorter. I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation by 5.5s (9.0s -> 3.5s). On my Mac system with clang, it speeds up compilation by 2.2s (3.7s -> 1.5s). (I don't think this flag will work with MSVC, but it'd probably speed up clang on windows.) The long-term goal here is probably to allow this to occur incrementally per Javascript file & in parallel, to avoid recompiling all of `gen/node_javascript.cc`. Unfortunately the necessary gyp incantations seem impossible (or at least, far beyond me). Anyway, a 60% speedup is a nice enough win. Refs: nodejs#47984

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.6 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new flag `--use-string-literals` to `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. The thing I'm most unsure about in this PR is how to actually thread the argument through. I considered adding it to the general argument parser, but that felt strange, since this flag only makes sense during the build process. So I kind of hacked it in, which also feels weird. Suggestions are very welcome. Refs: nodejs#47984 Refs: nodejs#48160

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.8 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new flag `--use-string-literals` to `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. The thing I'm most unsure about in this PR is how to actually thread the argument through. I considered adding it to the general argument parser, but that felt strange, since this flag only makes sense during the build process. So I kind of hacked it in, which also feels weird. Suggestions are very welcome. Refs: nodejs#47984 Refs: nodejs#48160

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.8 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new build define `NODE_MKSNAPSHOT_USE_STRING_LITERALS` which is used by `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. Refs: nodejs#47984 Refs: nodejs#48160

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 9 seconds and 830 MB of memory to compile `gen/node_javascript.cc` with g++. This is the longest step when recompiling a small change to a Javascript file. `gen/node_javascript.cc` contains a lot of large binary literals of our Javascript source code. It is well-known that embedding large binary literals as C/C++ arrays is slow. One workaround is to include the data as string literals instead. This is particularly nice for the Javascript included via js2c, which look better as string literals anyway. Add a build flag `NODE_JS2C_USE_STRING_LITERALS` to js2c. When this flag is set, we emit string literals instead of array literals, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = R"JS2C1b732aee(...)JS2C1b732aee"; // old: static const uint16_t Y[] = { ... }; static const uint16_t *Y = uR"JS2C1b732aee(...)JS2C1b732aee"; ``` This requires some modest refactoring in order to deal with the flag being on or off, but the new code itself is actually shorter. I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation by 5.5s (9.0s -> 3.5s). On my Mac system with clang, it speeds up compilation by 2.2s (3.7s -> 1.5s). (I don't think this flag will work with MSVC, but it'd probably speed up clang on windows.) The long-term goal here is probably to allow this to occur incrementally per Javascript file & in parallel, to avoid recompiling all of `gen/node_javascript.cc`. Unfortunately the necessary gyp incantations seem impossible (or at least, far beyond me). Anyway, a 60% speedup is a nice enough win. Refs: nodejs#47984

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.8 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new build define `NODE_MKSNAPSHOT_USE_STRING_LITERALS` which is used by `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. Refs: #47984 Refs: #48160 PR-URL: #48162 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.8 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new build define `NODE_MKSNAPSHOT_USE_STRING_LITERALS` which is used by `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. Refs: nodejs#47984 Refs: nodejs#48160 PR-URL: nodejs#48162 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 9 seconds and 830 MB of memory to compile `gen/node_javascript.cc` with g++. This is the longest step when recompiling a small change to a Javascript file. `gen/node_javascript.cc` contains a lot of large binary literals of our Javascript source code. It is well-known that embedding large binary literals as C/C++ arrays is slow. One workaround is to include the data as string literals instead. This is particularly nice for the Javascript included via js2c, which look better as string literals anyway. Add a build flag `NODE_JS2C_USE_STRING_LITERALS` to js2c. When this flag is set, we emit string literals instead of array literals, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = R"JS2C1b732aee(...)JS2C1b732aee"; // old: static const uint16_t Y[] = { ... }; static const uint16_t *Y = uR"JS2C1b732aee(...)JS2C1b732aee"; ``` This requires some modest refactoring in order to deal with the flag being on or off, but the new code itself is actually shorter. I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation by 5.5s (9.0s -> 3.5s). On my Mac system with clang, it speeds up compilation by 2.2s (3.7s -> 1.5s). (I don't think this flag will work with MSVC, but it'd probably speed up clang on windows.) The long-term goal here is probably to allow this to occur incrementally per Javascript file & in parallel, to avoid recompiling all of `gen/node_javascript.cc`. Unfortunately the necessary gyp incantations seem impossible (or at least, far beyond me). Anyway, a 60% speedup is a nice enough win. Refs: #47984 PR-URL: #48160 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.8 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new build define `NODE_MKSNAPSHOT_USE_STRING_LITERALS` which is used by `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. Refs: nodejs#47984 Refs: nodejs#48160 PR-URL: nodejs#48162 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 9 seconds and 830 MB of memory to compile `gen/node_javascript.cc` with g++. This is the longest step when recompiling a small change to a Javascript file. `gen/node_javascript.cc` contains a lot of large binary literals of our Javascript source code. It is well-known that embedding large binary literals as C/C++ arrays is slow. One workaround is to include the data as string literals instead. This is particularly nice for the Javascript included via js2c, which look better as string literals anyway. Add a build flag `NODE_JS2C_USE_STRING_LITERALS` to js2c. When this flag is set, we emit string literals instead of array literals, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = R"JS2C1b732aee(...)JS2C1b732aee"; // old: static const uint16_t Y[] = { ... }; static const uint16_t *Y = uR"JS2C1b732aee(...)JS2C1b732aee"; ``` This requires some modest refactoring in order to deal with the flag being on or off, but the new code itself is actually shorter. I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation by 5.5s (9.0s -> 3.5s). On my Mac system with clang, it speeds up compilation by 2.2s (3.7s -> 1.5s). (I don't think this flag will work with MSVC, but it'd probably speed up clang on windows.) The long-term goal here is probably to allow this to occur incrementally per Javascript file & in parallel, to avoid recompiling all of `gen/node_javascript.cc`. Unfortunately the necessary gyp incantations seem impossible (or at least, far beyond me). Anyway, a 60% speedup is a nice enough win. Refs: nodejs#47984 PR-URL: nodejs#48160 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.8 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new build define `NODE_MKSNAPSHOT_USE_STRING_LITERALS` which is used by `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. Refs: nodejs#47984 Refs: nodejs#48160 PR-URL: nodejs#48162 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 9 seconds and 830 MB of memory to compile `gen/node_javascript.cc` with g++. This is the longest step when recompiling a small change to a Javascript file. `gen/node_javascript.cc` contains a lot of large binary literals of our Javascript source code. It is well-known that embedding large binary literals as C/C++ arrays is slow. One workaround is to include the data as string literals instead. This is particularly nice for the Javascript included via js2c, which look better as string literals anyway. Add a build flag `NODE_JS2C_USE_STRING_LITERALS` to js2c. When this flag is set, we emit string literals instead of array literals, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = R"JS2C1b732aee(...)JS2C1b732aee"; // old: static const uint16_t Y[] = { ... }; static const uint16_t *Y = uR"JS2C1b732aee(...)JS2C1b732aee"; ``` This requires some modest refactoring in order to deal with the flag being on or off, but the new code itself is actually shorter. I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation by 5.5s (9.0s -> 3.5s). On my Mac system with clang, it speeds up compilation by 2.2s (3.7s -> 1.5s). (I don't think this flag will work with MSVC, but it'd probably speed up clang on windows.) The long-term goal here is probably to allow this to occur incrementally per Javascript file & in parallel, to avoid recompiling all of `gen/node_javascript.cc`. Unfortunately the necessary gyp incantations seem impossible (or at least, far beyond me). Anyway, a 60% speedup is a nice enough win. Refs: nodejs#47984 PR-URL: nodejs#48160 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

github-actions · 2023-11-17T01:25:15Z

There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.

For more information on how the project manages feature requests, please consult the feature request management document.

github-actions · 2023-12-17T01:27:34Z

There has been no activity on this feature request and it is being closed. If you feel closing this issue is not the right thing to do, please leave a comment.

For more information on how the project manages feature requests, please consult the feature request management document.

kvakil added build Issues and PRs related to build files or the CI. feature request Issues that request new features to be added to Node.js. snapshot Issues and PRs related to the startup snapshot labels May 12, 2023

kvakil mentioned this issue May 25, 2023

tools: speedup compilation of js2c output #48160

Merged

kvakil mentioned this issue May 25, 2023

build: speedup compilation of mksnapshot output #48162

Merged

github-actions bot added the stale label Nov 17, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

incremental rebuilds of Node.js are slow #47984

incremental rebuilds of Node.js are slow #47984

kvakil commented May 12, 2023

anonrig commented May 20, 2023

kvakil commented May 21, 2023

anonrig commented May 21, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Dec 17, 2023

incremental rebuilds of Node.js are slow #47984

incremental rebuilds of Node.js are slow #47984

Comments

kvakil commented May 12, 2023

What is the problem this feature will solve?

What is the feature you are proposing to solve the problem?

What alternatives have you considered?

anonrig commented May 20, 2023

kvakil commented May 21, 2023

anonrig commented May 21, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Dec 17, 2023