[v18.x] backport various patches of internal refactorings for snapshot and realms #45007

joyeecheung · 2022-10-14T20:00:39Z

This backports various refactoring patches for snapshot and shadow realms support to smooth out the backports of other patches. AFAICT only the shadow realms support patch changes the behavior a bit (but only if users uses the --experimental-shadow-realms flag and we don't have any compatibility guarantee for that anyway).

src: add initial shadow realm support

Add initial shadow realm support behind an off-by-default flag
--experimental-shadow-realm.

PR-URL: #42869
Refs: #42528
Reviewed-By: Antoine du Hamel duhamelantoine1995@gmail.com
Reviewed-By: Darshan Sen raisinten@gmail.com

src: per-environment time origin value

According to https://html.spec.whatwg.org/#environment-settings-object,
the timeOrigin is a per-environment value. Worker's timeOrigin is the
time when the worker is created.

PR-URL: #43781
Reviewed-By: Matteo Collina matteo.collina@gmail.com
Reviewed-By: Anna Henningsen anna@addaleax.net
Reviewed-By: Paolo Insogna paolo@cowtech.it
Reviewed-By: Joyee Cheung joyeec9h3@gmail.com

src: fix cppgc incompatibility in v8

This patch updates the layout of the BaseObjects to make sure
that the first embedder field of them is a "type" pointer, the
first 16 bits of which are the Node.js embedder ID, so that
cppgc will always skip over them. In addition we now use this
field to determine if the native object should be interpreted
as a Node.js embedder object in the serialization and deserialization
callbacks for the startup snapshot to improve the reliability.

Co-authored-by: Joyee Cheung joyeec9h3@gmail.com
PR-URL: #43521
Reviewed-By: James M Snell jasnell@gmail.com
Reviewed-By: Darshan Sen raisinten@gmail.com

bootstrap: check more metadata when loading the snapshot

This patch stores the metadata about the Node.js binary
into the SnapshotData and adds fields denoting how the
snapshot was generated, on what platform it was
generated as well as the V8 cached data version flag.
Instead of simply crashing when the metadata doesn't
match, Node.js now prints an error message and exit with
1 for the customized snapshot, or ignore the snapshot
and start from scratch if it's the default one.

PR-URL: #44132
Reviewed-By: Anna Henningsen anna@addaleax.net
Reviewed-By: Chengzhong Wu legendecas@gmail.com

src: iterate over base objects to prepare for snapshot

Instead of iterating over the bindings, iterate over the base
objects that are snapshottable. This allows us to snapshot
base objects that are not bindings. In addition this refactors
the InternalFieldInfo class to eliminate potential undefined
behaviors, and renames it to InternalFieldInfoBase.
The {de}serialize callbacks now expect a InternalFieldInfo struct
nested in Snapshotable classes that can be used to carry
serialization data around. This allows us to create structs
inheriting from InternalFieldInfo for Snapshotable objects
that need custom fields.

PR-URL: #44192
Refs: #37476
Reviewed-By: Anna Henningsen anna@addaleax.net
Reviewed-By: Chengzhong Wu legendecas@gmail.com

src: support WeakReference in snapshot

Move util::WeakReference to a separate header and implement
{de}serialization for it to be snapshotable.

PR-URL: #44193
Refs: #44014
Refs: #37476
Reviewed-By: Matteo Collina matteo.collina@gmail.com
Reviewed-By: Chengzhong Wu legendecas@gmail.com

src: introduce node::Realm

To distinguish per-context values from the node::Environment, split
those values to a new node::Realm structure and consolidate
bootstrapping methods with it.

PR-URL: #44179
Refs: #42528
Reviewed-By: Joyee Cheung joyeec9h3@gmail.com
Reviewed-By: James M Snell jasnell@gmail.com

src: create BaseObject with node::Realm

BaseObject is a wrapper around JS objects. These objects should be
created in a node::Realm and destroyed when their associated realm is
cleaning up.

PR-URL: #44348
Refs: #42528
Reviewed-By: Joyee Cheung joyeec9h3@gmail.com

src: refactor BaseObject methods

Wrap the initialization of the kSlot and kEmbedderType fields
into a BaseObject::SetInternalFields() method.
Move the tagging of kEmbedderType field into
BaseObject::TagNodeObject()
Add a variant of BaseObject::MakeLazilyInitializedJSTemplate()
that only needs IsolateData.
This makes it easier to create BaseObject subclasses.

PR-URL: #44796
Reviewed-By: Chengzhong Wu legendecas@gmail.com
Reviewed-By: James M Snell jasnell@gmail.com

benchmark: add vm context global proxy benchmark

PR-URL: #44796
Reviewed-By: Chengzhong Wu legendecas@gmail.com
Reviewed-By: James M Snell jasnell@gmail.com

vm: make ContextifyContext a BaseObject

Instead of adding a reference to the ContextifyContext by using
a v8::External, we make ContextifyContext a weak BaseObject that
whose wrapper is referenced by the sandbox via a private symbol.
This makes it easier to snapshot the contexts, in addition to
reusing the BaseObject lifetime management for ContextifyContexts.

PR-URL: #44796
Reviewed-By: Chengzhong Wu legendecas@gmail.com
Reviewed-By: James M Snell jasnell@gmail.com

Add initial shadow realm support behind an off-by-default flag `--experimental-shadow-realm`. PR-URL: nodejs#42869 Refs: nodejs#42528 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com>

According to https://html.spec.whatwg.org/#environment-settings-object, the timeOrigin is a per-environment value. Worker's timeOrigin is the time when the worker is created. PR-URL: nodejs#43781 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Paolo Insogna <paolo@cowtech.it> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

This patch updates the layout of the BaseObjects to make sure that the first embedder field of them is a "type" pointer, the first 16 bits of which are the Node.js embedder ID, so that cppgc will always skip over them. In addition we now use this field to determine if the native object should be interpreted as a Node.js embedder object in the serialization and deserialization callbacks for the startup snapshot to improve the reliability. Co-authored-by: Joyee Cheung <joyeec9h3@gmail.com> PR-URL: nodejs#43521 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com>

This patch stores the metadata about the Node.js binary into the SnapshotData and adds fields denoting how the snapshot was generated, on what platform it was generated as well as the V8 cached data version flag. Instead of simply crashing when the metadata doesn't match, Node.js now prints an error message and exit with 1 for the customized snapshot, or ignore the snapshot and start from scratch if it's the default one. PR-URL: nodejs#44132 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Instead of iterating over the bindings, iterate over the base objects that are snapshottable. This allows us to snapshot base objects that are not bindings. In addition this refactors the InternalFieldInfo class to eliminate potential undefined behaviors, and renames it to InternalFieldInfoBase. The {de}serialize callbacks now expect a InternalFieldInfo struct nested in Snapshotable classes that can be used to carry serialization data around. This allows us to create structs inheriting from InternalFieldInfo for Snapshotable objects that need custom fields. PR-URL: nodejs#44192 Refs: nodejs#37476 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Move util::WeakReference to a separate header and implement {de}serialization for it to be snapshotable. PR-URL: nodejs#44193 Refs: nodejs#44014 Refs: nodejs#37476 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

To distinguish per-context values from the node::Environment, split those values to a new node::Realm structure and consolidate bootstrapping methods with it. PR-URL: nodejs#44179 Refs: nodejs#42528 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

nodejs-github-bot · 2022-10-14T20:00:44Z

Review requested:

@nodejs/gyp
@nodejs/http2
@nodejs/net
@nodejs/startup

BaseObject is a wrapper around JS objects. These objects should be created in a node::Realm and destroyed when their associated realm is cleaning up. PR-URL: nodejs#44348 Refs: nodejs#42528 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

- Wrap the initialization of the kSlot and kEmbedderType fields into a BaseObject::SetInternalFields() method. - Move the tagging of kEmbedderType field into BaseObject::TagNodeObject() - Add a variant of BaseObject::MakeLazilyInitializedJSTemplate() that only needs IsolateData. This makes it easier to create BaseObject subclasses. PR-URL: nodejs#44796 Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

PR-URL: nodejs#44796 Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Instead of adding a reference to the ContextifyContext by using a v8::External, we make ContextifyContext a weak BaseObject that whose wrapper is referenced by the sandbox via a private symbol. This makes it easier to snapshot the contexts, in addition to reusing the BaseObject lifetime management for ContextifyContexts. PR-URL: nodejs#44796 Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

nodejs-github-bot · 2022-10-18T14:58:27Z

CI: https://ci.nodejs.org/job/node-test-pull-request/47324/

joyeecheung · 2022-10-18T14:58:33Z

cc @legendecas @nodejs/releasers

legendecas

Thanks for doing the backports!

joyeecheung · 2022-10-26T09:09:25Z

Added back some changes to test/pummel/test-heapdump-env.js that were left out

nodejs-github-bot · 2022-10-26T09:09:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/47493/

mcollina

lgtm

nodejs-github-bot · 2022-10-31T14:27:06Z

CI: https://ci.nodejs.org/job/node-test-pull-request/47592/

nodejs-github-bot · 2022-11-01T15:56:54Z

CI: https://ci.nodejs.org/job/node-test-pull-request/47619/

joyeecheung · 2022-11-01T20:04:43Z

Landed in 0774b64...c37dc7c

danielleadams · 2023-01-03T15:55:21Z

#43781 was dropped from v18.13.0 (#46025) because it is semver-major change - if its decided that this is a bugfix for v18.x, we can open another backport and drop the semver-major label.

legendecas and others added 7 commits October 14, 2022 16:08

src: add initial shadow realm support

c13a4bc

Add initial shadow realm support behind an off-by-default flag `--experimental-shadow-realm`. PR-URL: nodejs#42869 Refs: nodejs#42528 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com>

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. v18.x Issues that can be reproduced on v18.x or PRs targeting the v18.x-staging branch. labels Oct 14, 2022

legendecas and others added 4 commits October 17, 2022 14:31

src: create BaseObject with node::Realm

95d5daf

BaseObject is a wrapper around JS objects. These objects should be created in a node::Realm and destroyed when their associated realm is cleaning up. PR-URL: nodejs#44348 Refs: nodejs#42528 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

benchmark: add vm context global proxy benchmark

d2192e6

PR-URL: nodejs#44796 Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

gengjiawen approved these changes Oct 18, 2022

View reviewed changes

legendecas approved these changes Oct 18, 2022

View reviewed changes

fixup! src: introduce node::Realm

9f90e65

mcollina approved these changes Oct 26, 2022

View reviewed changes

RafaelGSS approved these changes Oct 26, 2022

View reviewed changes

github-actions bot mentioned this pull request Nov 1, 2022

CI Reliability 2022-11-01 nodejs/reliability#417

Open

17 tasks

joyeecheung closed this Nov 1, 2022

github-actions bot mentioned this pull request Nov 2, 2022

CI Reliability 2022-11-02 nodejs/reliability#418

Open

15 tasks

joyeecheung mentioned this pull request Nov 2, 2022

Node.js Technical Steering Committee (TSC) Meeting 2022-11-02 nodejs/TSC#1302

Closed

danielleadams mentioned this pull request Jan 2, 2023

v18.13.0 release proposal #46025

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v18.x] backport various patches of internal refactorings for snapshot and realms #45007

[v18.x] backport various patches of internal refactorings for snapshot and realms #45007

joyeecheung commented Oct 14, 2022 •

edited

Loading

nodejs-github-bot commented Oct 14, 2022

nodejs-github-bot commented Oct 18, 2022

joyeecheung commented Oct 18, 2022 •

edited

Loading

legendecas left a comment •

edited

Loading

joyeecheung commented Oct 26, 2022

nodejs-github-bot commented Oct 26, 2022

mcollina left a comment

nodejs-github-bot commented Oct 31, 2022

nodejs-github-bot commented Nov 1, 2022

joyeecheung commented Nov 1, 2022 •

edited

Loading

danielleadams commented Jan 3, 2023 •

edited

Loading

[v18.x] backport various patches of internal refactorings for snapshot and realms #45007

[v18.x] backport various patches of internal refactorings for snapshot and realms #45007

Conversation

joyeecheung commented Oct 14, 2022 • edited Loading

src: add initial shadow realm support

src: per-environment time origin value

src: fix cppgc incompatibility in v8

bootstrap: check more metadata when loading the snapshot

src: iterate over base objects to prepare for snapshot

src: support WeakReference in snapshot

src: introduce node::Realm

src: create BaseObject with node::Realm

src: refactor BaseObject methods

benchmark: add vm context global proxy benchmark

vm: make ContextifyContext a BaseObject

nodejs-github-bot commented Oct 14, 2022

nodejs-github-bot commented Oct 18, 2022

joyeecheung commented Oct 18, 2022 • edited Loading

legendecas left a comment • edited Loading

Choose a reason for hiding this comment

joyeecheung commented Oct 26, 2022

nodejs-github-bot commented Oct 26, 2022

mcollina left a comment

Choose a reason for hiding this comment

nodejs-github-bot commented Oct 31, 2022

nodejs-github-bot commented Nov 1, 2022

joyeecheung commented Nov 1, 2022 • edited Loading

danielleadams commented Jan 3, 2023 • edited Loading

joyeecheung commented Oct 14, 2022 •

edited

Loading

joyeecheung commented Oct 18, 2022 •

edited

Loading

legendecas left a comment •

edited

Loading

joyeecheung commented Nov 1, 2022 •

edited

Loading

danielleadams commented Jan 3, 2023 •

edited

Loading