Lower TLS memory footprint #1529

indutny · 2015-04-26T12:49:22Z

indutny · 2015-04-26T12:50:12Z

CI: https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/603/

ChALkeR · 2015-04-26T13:11:07Z

Saves ~30% rss in my testcase (that consists of repeating a https request many times).
That's short-term, i haven't measured how this changes the time when full gc kicks in yet.

ChALkeR · 2015-04-26T13:41:15Z

Looks like not all ssl buffers are freed, see #1522 (comment)

ChALkeR · 2015-04-27T06:53:09Z

I guess that's in the SecureContext.
Not 100% sure, though.

ChALkeR · 2015-04-27T07:18:32Z

Yes, it looks like SecureContext objects are created per every request and not deleted (until a full gc run).

indutny · 2015-04-27T07:27:59Z

It is, and I don't think that there is something that we could do about it with enabling the keepAlive. I mean, that's the purpose of it!

ChALkeR · 2015-04-27T07:33:11Z

No, when the keepAlive is not enabled and the context isn't going to be reused, isn't it best to dispose of it immediately without waiting for the full gc run? As you did with the TLSWrap data here.

Results:
http://oserv.org/bugs/iojs/memory0/pr1529.t1.3000iter.without.massif.out.12103 — without this pr
http://oserv.org/bugs/iojs/memory0/pr1529.t1.3000iter.massif.out.12080 — with this pr
http://oserv.org/bugs/iojs/memory0/pr1529.t1.3000iter.contextkill.massif.out.31454 — with this pr and destroying the context with

  this.on('close', function() {
    context.context.close();
  });

indutny · 2015-04-27T07:35:19Z

Oh, good idea...

indutny · 2015-04-27T07:40:28Z

@ChALkeR pushed, thank you!

ChALkeR · 2015-04-27T07:57:43Z

Does not work somewhy. Will check in several minutes.
This commit didn't change anything.

indutny · 2015-04-27T08:00:09Z

Because I made a typo, sorry!

indutny · 2015-04-27T08:03:08Z

Force pushed...

shigeki · 2015-04-27T08:35:12Z

src/node_crypto.h

@@ -221,6 +231,8 @@ class SSLWrap {
  static void SSLGetter(v8::Local<v8::String> property,
                        const v8::PropertyCallbackInfo<v8::Value>& info);

+  void DestroySSL();


@indutny This causes a link error with gcc-4.8.4 on my Ubuntu but works fine with gcc-4.9.2 on Ubuntu and with clang on Mac though I've not checked yet whether it comes from a gcc bug or not.

/home/ohtsu/github/shigeki/io.js/out/Release/obj.target/iojs/src/tls_wrap.o: In function `node::crypto::SSLWrap<node::TLSWrap>::~SSLWrap()': tls_wrap.cc:(.text._ZN4node6crypto7SSLWrapINS_7TLSWrapEED0Ev[_ZN4node6crypto7SSLWrapINS_7TLSWrapEED0Ev]+0x14): undefined reference to `node::crypto::SSLWrap<node::TLSWrap>::DestroySSL()' /home/ohtsu/github/shigeki/io.js/out/Release/obj.target/iojs/src/tls_wrap.o: In function `node::TLSWrap::~TLSWrap()': tls_wrap.cc:(.text._ZN4node7TLSWrapD2Ev+0x1fe): undefined reference to `node::crypto::SSLWrap<node::TLSWrap>::DestroySSL()' collect2: error: ld returned 1 exit status iojs.target.mk:198: recipe for target '/home/ohtsu/github/shigeki/io.js/out/Release/iojs' failed make[1]: *** [/home/ohtsu/github/shigeki/io.js/out/Release/iojs] Error 1 make[1]: Leaving directory '/home/ohtsu/github/shigeki/io.js/out' Makefile:35: recipe for target 'iojs' failed make: *** [iojs] Error 2 ohtsu@u1504:~/github/shigeki/io.js$ g++ --version g++ (Ubuntu 4.8.4-1ubuntu15) 4.8.4

ChALkeR · 2015-04-27T08:35:44Z

Does not work.
options.keepAlive is undefined by default on the line where you compare it to false, but by default keep-alive is disabled AFAIK.

indutny · 2015-04-27T08:44:40Z

Oh, right! I had keepAlive: false in my test. Fixed everything!

ChALkeR · 2015-04-27T08:49:22Z

http://oserv.org/bugs/iojs/memory0/pr1529.t2.3000iter.massif.out.1609 — massif.out for the updated pull request.
Works for me, the total memory usage after 3000 requests (without keep-alive) went from 225 MiB down to 8 MiB (in first version of this pr — 120 MiB).
Full gc was not triggered in either of the three versions (without the patch, first version, current version).

Hope this doesn't break anything ;-).

ChALkeR · 2015-04-27T08:59:45Z

I will run upload the results of a longer test soon.

rvagg · 2015-04-27T09:08:46Z

I'd love a comparison of this to 0.12 and 0.10 if anyone can manage it, are we actually improving on what was there before or just on the TLSWrap changes, does OpenSSL 1.0.2 buy us anything here? I've heard some complaints about Node's memory footprint with tls in general in the past and it'd be great if this is attacking that.

ChALkeR · 2015-04-27T09:13:22Z

I'd love a comparison of this to 0.12 and 0.10 if anyone can manage it, are we actually improving on what was there before or just on the TLSWrap changes,

Afaik 0.12 behaves like iojs without this pull request, it just runs full gc earlier due to an older version of v8, so the problem wasn't so noticable. I will upload 0.12 test results with the same parameters a bit later.

ChALkeR · 2015-04-27T09:25:05Z

With the current version of the pr full gc was triggered after 8720 iterations, memory usage before full gc run looks like this: http://oserv.org/bugs/iojs/memory0/pr1529.t2.8720iter.massif.out.4212
Memory usage for 8720 iterations is 17 MiB.

ChALkeR · 2015-04-27T09:30:30Z

The two parts that grow are node::TLSWrap::Wrap() (3,3 MiB), that's the TLSWrap objects themselves and node::NodeBIO::TryAllocateForWrite() (8,5 MiB), from TLSWrap::DoWrite().
The base (non-growing part) is ~4.8 MiB.

That's about 1.4 KiB per request (was ~75 KiB per request before this pr).
node::NodeBIO::TryAllocateForWrite() is ~1KiB per request.
node::TLSWrap::Wrap() is ~0.4KiB per request.

Just in case: is there something that could be done with node::NodeBIO::TryAllocateForWrite()?

Do not keep SSL structure in memory once socket is closed. This should lower the memory usage in many cases. Fix: nodejs#1522

Ensure that GC kicks in at the right times and the RSS does not blow up. Fix: nodejs#1522

When connecting to server with `keepAlive` turned off - make sure that the read/write buffers won't be kept in a single use SSL_CTX instance after the socket will be destroyed. Fix: nodejs#1522

Destroy singleUse context right after it is going out of use. Fix: nodejs#1522

@othiym23

PR-URL: #1532 Notable Changes: * crypto: significantly reduced memory usage for TLS (Fedor Indutny & Сковорода Никита Андреевич) #1529 * net: socket.connect() now accepts a 'lookup' option for a custom DNS resolution mechanism, defaults to dns.lookup() (Evan Lucas) #1505 * npm: Upgrade npm to 2.9.0. See the v2.8.4 and v2.9.0 release notes for details. Notable items: - Add support for default author field to make npm init -y work without user-input (@othiym23) npm/npm/d8eee6cf9d - Include local modules in npm outdated and npm update (@ArnaudRinquin) npm/npm#7426 - The prefix used before the version number on npm version is now configurable via tag-version-prefix (@kkragenbrink) npm/npm#8014 * os: os.tmpdir() is now cross-platform consistent and will no longer returns a path with a trailling slash on any platform (Christian Tellnes) #747 * process: - process.nextTick() performance has been improved by between 2-42% across the benchmark suite, notable because this is heavily used across core (Brian White) #1548 - New process.geteuid(), process.seteuid(id), process.getegid() and process.setegid(id) methods allow you to get and set effective UID and GID of the process (Evan Lucas) #1536 * repl: - REPL history can be persisted across sessions if the NODE_REPL_HISTORY_FILE environment variable is set to a user accessible file, NODE_REPL_HISTORY_SIZE can set the maximum history size and defaults to 1000 (Chris Dickinson) #1513 - The REPL can be placed in to one of three modes using the NODE_REPL_MODE environment variable: sloppy, strict or magic (default); the new magic mode will automatically run "strict mode only" statements in strict mode (Chris Dickinson) #1513 * smalloc: the 'smalloc' module has been deprecated due to changes coming in V8 4.4 that will render it unusable * util: add Promise, Map and Set inspection support (Christopher Monsanto) #1471 * V8: upgrade to 4.2.77.18, see the ChangeLog for full details. Notable items: - Classes have moved out of staging; the class keyword is now usable in strict mode without flags - Object literal enhancements have moved out of staging; shorthand method and property syntax is now usable ({ method() { }, property }) - Rest parameters (function(...args) {}) are implemented in staging behind the --harmony-rest-parameters flag - Computed property names ({['foo'+'bar']:'bam'}) are implemented in staging behind the --harmony-computed-property-names flag - Unicode escapes ('\u{xxxx}') are implemented in staging behind the --harmony_unicode flag and the --harmony_unicode_regexps flag for use in regular expressions * Windows: - Random process termination on Windows fixed (Fedor Indutny) #1512 / #1563 - The delay-load hook introduced to fix issues with process naming (iojs.exe / node.exe) has been made opt-out for native add-ons. Native add-ons should include 'win_delay_load_hook': 'false' in their binding.gyp to disable this feature if they experience problems . (Bert Belder) #1433 * Governance: - Rod Vagg (@rvagg) was added to the Technical Committee (TC) - Jeremiah Senkpiel (@Fishrock123) was added to the Technical Committee (TC)

Maintenance release Notable Changes: * crypto: significantly reduced memory usage for TLS (Fedor Indutny & Сковорода Никита Андреевич) #1529 * npm: Upgrade npm to 2.9.0. See the v2.8.4 and v2.9.0 release notes for details.

Maintenance release Notable Changes: * crypto: significantly reduced memory usage for TLS (Fedor Indutny & Сковорода Никита Андреевич) nodejs#1529 * npm: Upgrade npm to 2.9.0. See the v2.8.4 and v2.9.0 release notes for details.

This function was introduced in 2684c90 as an internal helper function. The C++ implementation became a no-op in a57e2f2 when building against OpenSSL 1.1.0 (instead of OpenSSL 1.0.2), and eventually became a no-op in all supported OpenSSL versions in 970ce14. Finally, eb20447 removed the only call site of setFreeListLength (which was already a no-op at that point). Refs: nodejs#1529 Refs: nodejs#10859 Refs: nodejs#19794 Refs: nodejs#38116

This function was introduced in 2684c90 as an internal helper function. The C++ implementation became a no-op in a57e2f2 when building against OpenSSL 1.1.0 (instead of OpenSSL 1.0.2), and eventually became a no-op in all supported OpenSSL versions in 970ce14. Finally, eb20447 removed the only call site of setFreeListLength (which was already a no-op at that point). Refs: #1529 Refs: #10859 Refs: #19794 Refs: #38116 PR-URL: #44300 Reviewed-By: Feng Yu <F3n67u@outlook.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

This function was introduced in 2684c90 as an internal helper function. The C++ implementation became a no-op in a57e2f2 when building against OpenSSL 1.1.0 (instead of OpenSSL 1.0.2), and eventually became a no-op in all supported OpenSSL versions in 970ce14. Finally, eb20447 removed the only call site of setFreeListLength (which was already a no-op at that point). Refs: nodejs#1529 Refs: nodejs#10859 Refs: nodejs#19794 Refs: nodejs#38116 PR-URL: nodejs#44300 Reviewed-By: Feng Yu <F3n67u@outlook.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

This function was introduced in 2684c90 as an internal helper function. The C++ implementation became a no-op in a57e2f2 when building against OpenSSL 1.1.0 (instead of OpenSSL 1.0.2), and eventually became a no-op in all supported OpenSSL versions in 970ce14. Finally, eb20447 removed the only call site of setFreeListLength (which was already a no-op at that point). Refs: #1529 Refs: #10859 Refs: #19794 Refs: #38116 PR-URL: #44300 Reviewed-By: Feng Yu <F3n67u@outlook.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

This function was introduced in 2684c90 as an internal helper function. The C++ implementation became a no-op in a57e2f2 when building against OpenSSL 1.1.0 (instead of OpenSSL 1.0.2), and eventually became a no-op in all supported OpenSSL versions in 970ce14. Finally, eb20447 removed the only call site of setFreeListLength (which was already a no-op at that point). Refs: nodejs/node#1529 Refs: nodejs/node#10859 Refs: nodejs/node#19794 Refs: nodejs/node#38116 PR-URL: nodejs/node#44300 Reviewed-By: Feng Yu <F3n67u@outlook.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

indutny mentioned this pull request Apr 26, 2015

high tls memory usage (rss) #1522

Closed

mscdex added the tls Issues and PRs related to the tls subsystem. label Apr 26, 2015

indutny force-pushed the fix/gh-1522 branch from 3058ad0 to 9eed89b Compare April 27, 2015 08:03

shigeki reviewed Apr 27, 2015
View reviewed changes

indutny force-pushed the fix/gh-1522 branch 2 times, most recently from b217b38 to 7470588 Compare April 27, 2015 08:39

indutny added 4 commits April 27, 2015 11:40

tls: destroy SSL once it is out of use

4e066c6

Do not keep SSL structure in memory once socket is closed. This should lower the memory usage in many cases. Fix: nodejs#1522

crypto: track external memory for SSL structures

6c9620a

Ensure that GC kicks in at the right times and the RSS does not blow up. Fix: nodejs#1522

tls: zero SSL_CTX freelist for a singleUse socket

dbac061

When connecting to server with `keepAlive` turned off - make sure that the read/write buffers won't be kept in a single use SSL_CTX instance after the socket will be destroyed. Fix: nodejs#1522

tls: destroy singleUse context immediately

1357735

Destroy singleUse context right after it is going out of use. Fix: nodejs#1522

Fishrock123 mentioned this pull request May 14, 2015

Release proposal: v1.8.2 #1703

Closed

diagramatics mentioned this pull request May 25, 2015

Create weekly-update.2015-05-22.md nodejs/evangelism#83

Merged

mscdex mentioned this pull request Jun 1, 2015

TLS Crypto Memory Leak ( New Relic ) #1866

Closed

ChALkeR mentioned this pull request Jun 12, 2015

Node.js memory usage issue nodejs/node-v0.x-archive#25519

Closed

ChALkeR added the memory Issues and PRs related to the memory management or memory footprint. label Aug 17, 2015

ChALkeR mentioned this pull request Sep 11, 2015

Large number of C++ objects unclaimed in node v0.12.7 #2813

Closed

ChALkeR mentioned this pull request Oct 12, 2015

tls: catch race condition to prevent memory leak nodejs/node-v0.x-archive#9064

Closed

snyk-bot mentioned this pull request Jul 28, 2021

[Snyk] Upgrade commander from 2.11.0 to 8.0.0 MaxMood96/node#4

Open

tniessen mentioned this pull request Aug 20, 2022

tls: remove SecureContext setFreeListLength #44300

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lower TLS memory footprint #1529

Lower TLS memory footprint #1529

indutny commented Apr 26, 2015

indutny commented Apr 26, 2015

ChALkeR commented Apr 26, 2015

ChALkeR commented Apr 26, 2015

ChALkeR commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

indutny commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

indutny commented Apr 27, 2015

indutny commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

indutny commented Apr 27, 2015

indutny commented Apr 27, 2015

shigeki Apr 27, 2015

ChALkeR commented Apr 27, 2015

indutny commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

rvagg commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

Lower TLS memory footprint #1529

Lower TLS memory footprint #1529

Conversation

indutny commented Apr 26, 2015

indutny commented Apr 26, 2015

ChALkeR commented Apr 26, 2015

ChALkeR commented Apr 26, 2015

ChALkeR commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

indutny commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

indutny commented Apr 27, 2015

indutny commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

indutny commented Apr 27, 2015

indutny commented Apr 27, 2015

shigeki Apr 27, 2015

Choose a reason for hiding this comment

ChALkeR commented Apr 27, 2015

indutny commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

rvagg commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

ChALkeR commented Apr 27, 2015

ChALkeR commented Apr 27, 2015