Merge vetted shims #1705

packages/compartment-mapper/test/snapshots/test-error-handling.js.md

@@ -95,14 +95,14 @@ Generated by [AVA](https://avajs.dev).
     `Error: Cannot find external module "missing" in package file://.../compartment-mapper/test/fixtures-error-handling/node_modules/cjs/␊
         at Object.execute (file://.../compartment-mapper/src/import-archive.js:…)␊
         at execute (file://.../ses/src/module-instance.js:…)␊
-        at compartmentImportNow (file://.../ses/src/compartment-shim.js:…)␊
-        at Compartment.importNow (file://.../ses/src/compartment-shim.js:…)␊
+        at compartmentImportNow (file://.../ses/src/compartment.js:…)␊
+        at Compartment.importNow (file://.../ses/src/compartment.js:…)␊
         at require (file://.../compartment-mapper/src/parse-cjs-shared-export-wrapper.js:…)␊
         at eval (eval at <anonymous> (eval at makeEvaluate (file://.../ses/src/make-evaluate.js:…)), <anonymous>:…)␊
         at Object.execute (file://.../compartment-mapper/src/parse-pre-cjs.js:…)␊
         at execute (file://.../ses/src/module-instance.js:…)␊
-        at compartmentImportNow (file://.../ses/src/compartment-shim.js:…)␊
-        at file://.../ses/src/compartment-shim.js:…`
+        at compartmentImportNow (file://.../ses/src/compartment.js:…)␊
+        at file://.../ses/src/compartment.js:…`
 
 ## fixtures-error-handling / cjs / makeArchive / parseArchive with a prefix
 
@@ -111,14 +111,14 @@ Generated by [AVA](https://avajs.dev).
     `Error: Cannot find external module "missing" in package file://.../compartment-mapper/test/fixtures-error-handling/node_modules/cjs/␊
         at Object.execute (file://.../compartment-mapper/src/import-archive.js:…)␊
         at execute (file://.../ses/src/module-instance.js:…)␊
-        at compartmentImportNow (file://.../ses/src/compartment-shim.js:…)␊
-        at Compartment.importNow (file://.../ses/src/compartment-shim.js:…)␊
+        at compartmentImportNow (file://.../ses/src/compartment.js:…)␊
+        at Compartment.importNow (file://.../ses/src/compartment.js:…)␊
         at require (file://.../compartment-mapper/src/parse-cjs-shared-export-wrapper.js:…)␊
         at eval (eval at <anonymous> (eval at makeEvaluate (file://.../ses/src/make-evaluate.js:…)), <anonymous>:…)␊
         at Object.execute (file://.../compartment-mapper/src/parse-pre-cjs.js:…)␊
         at execute (file://.../ses/src/module-instance.js:…)␊
-        at compartmentImportNow (file://.../ses/src/compartment-shim.js:…)␊
-        at file://.../ses/src/compartment-shim.js:…`
+        at compartmentImportNow (file://.../ses/src/compartment.js:…)␊
+        at file://.../ses/src/compartment.js:…`
 
 ## fixtures-error-handling / cjs / writeArchive / loadArchive
 
@@ -127,14 +127,14 @@ Generated by [AVA](https://avajs.dev).
     `Error: Cannot find external module "missing" in package file://.../compartment-mapper/test/fixtures-error-handling/node_modules/cjs/␊
         at Object.execute (file://.../compartment-mapper/src/import-archive.js:…)␊
         at execute (file://.../ses/src/module-instance.js:…)␊
-        at compartmentImportNow (file://.../ses/src/compartment-shim.js:…)␊
-        at Compartment.importNow (file://.../ses/src/compartment-shim.js:…)␊
+        at compartmentImportNow (file://.../ses/src/compartment.js:…)␊
+        at Compartment.importNow (file://.../ses/src/compartment.js:…)␊
         at require (file://.../compartment-mapper/src/parse-cjs-shared-export-wrapper.js:…)␊
         at eval (eval at <anonymous> (eval at makeEvaluate (file://.../ses/src/make-evaluate.js:…)), <anonymous>:…)␊
         at Object.execute (file://.../compartment-mapper/src/parse-pre-cjs.js:…)␊
         at execute (file://.../ses/src/module-instance.js:…)␊
-        at compartmentImportNow (file://.../ses/src/compartment-shim.js:…)␊
-        at file://.../ses/src/compartment-shim.js:…`
+        at compartmentImportNow (file://.../ses/src/compartment.js:…)␊
+        at file://.../ses/src/compartment.js:…`
 
 ## fixtures-error-handling / cjs / writeArchive / importArchive
 
@@ -143,71 +143,11 @@ Generated by [AVA](https://avajs.dev).
     `Error: Cannot find external module "missing" in package file://.../compartment-mapper/test/fixtures-error-handling/node_modules/cjs/␊
         at Object.execute (file://.../compartment-mapper/src/import-archive.js:…)␊
         at execute (file://.../ses/src/module-instance.js:…)␊
-        at compartmentImportNow (file://.../ses/src/compartment-shim.js:…)␊
-        at Compartment.importNow (file://.../ses/src/compartment-shim.js:…)␊
+        at compartmentImportNow (file://.../ses/src/compartment.js:…)␊
+        at Compartment.importNow (file://.../ses/src/compartment.js:…)␊
         at require (file://.../compartment-mapper/src/parse-cjs-shared-export-wrapper.js:…)␊
         at eval (eval at <anonymous> (eval at makeEvaluate (file://.../ses/src/make-evaluate.js:…)), <anonymous>:…)␊
         at Object.execute (file://.../compartment-mapper/src/parse-pre-cjs.js:…)␊
         at execute (file://.../ses/src/module-instance.js:…)␊
-        at compartmentImportNow (file://.../ses/src/compartment-shim.js:…)␊
-        at file://.../ses/src/compartment-shim.js:…`
-
-## fixtures-error-handling / both / loadLocation
-
-> Snapshot 1
-
-    `TypeError: Failed to load module "./main.js" in package "file://.../compartment-mapper/test/fixtures-error-handling/node_modules/both/" (1 underlying failures: Cannot find external module "missing" in package file://.../compartment-mapper/test/fixtures-error-handling/node_modules/esm/␊
-        at load (file://.../ses/src/module-load.js:…)␊
-        at async file://.../compartment-mapper/test/scaffold.js:…␊
-        at async file://.../compartment-mapper/test/scaffold.js:…`
-
-
-## fixtures-error-handling / both / makeArchive / parseArchive
-
-> Snapshot 1
-
-    `TypeError: Failed to load module "./main.js" in package "file://.../compartment-mapper/test/fixtures-error-handling/node_modules/both/" (1 underlying failures: Cannot find external module "missing" in package file://.../compartment-mapper/test/fixtures-error-handling/node_modules/esm/␊
-        at load (file://.../ses/src/module-load.js:…)␊
-        at async digestLocation (file://.../compartment-mapper/src/archive.js:…)␊
-        at async makeAndHashArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async makeArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async file://.../compartment-mapper/test/scaffold.js:…␊
-        at async file://.../compartment-mapper/test/scaffold.js:…`
-
-## fixtures-error-handling / both / makeArchive / parseArchive with a prefix
-
-> Snapshot 1
-
-    `TypeError: Failed to load module "./main.js" in package "file://.../compartment-mapper/test/fixtures-error-handling/node_modules/both/" (1 underlying failures: Cannot find external module "missing" in package file://.../compartment-mapper/test/fixtures-error-handling/node_modules/esm/␊
-        at load (file://.../ses/src/module-load.js:…)␊
-        at async digestLocation (file://.../compartment-mapper/src/archive.js:…)␊
-        at async makeAndHashArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async makeArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async file://.../compartment-mapper/test/scaffold.js:…␊
-        at async file://.../compartment-mapper/test/scaffold.js:…`
-
-## fixtures-error-handling / both / writeArchive / loadArchive
-
-> Snapshot 1
-
-    `TypeError: Failed to load module "./main.js" in package "file://.../compartment-mapper/test/fixtures-error-handling/node_modules/both/" (1 underlying failures: Cannot find external module "missing" in package file://.../compartment-mapper/test/fixtures-error-handling/node_modules/esm/␊
-        at load (file://.../ses/src/module-load.js:…)␊
-        at async digestLocation (file://.../compartment-mapper/src/archive.js:…)␊
-        at async makeAndHashArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async makeArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async writeArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async file://.../compartment-mapper/test/scaffold.js:…␊
-        at async file://.../compartment-mapper/test/scaffold.js:…`
-
-## fixtures-error-handling / both / writeArchive / importArchive
-
-> Snapshot 1
-
-    `TypeError: Failed to load module "./main.js" in package "file://.../compartment-mapper/test/fixtures-error-handling/node_modules/both/" (1 underlying failures: Cannot find external module "missing" in package file://.../compartment-mapper/test/fixtures-error-handling/node_modules/esm/␊
-        at load (file://.../ses/src/module-load.js:…)␊
-        at async digestLocation (file://.../compartment-mapper/src/archive.js:…)␊
-        at async makeAndHashArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async makeArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async writeArchive (file://.../compartment-mapper/src/archive.js:…)␊
-        at async file://.../compartment-mapper/test/scaffold.js:…␊
-        at async file://.../compartment-mapper/test/scaffold.js:…`
+        at compartmentImportNow (file://.../ses/src/compartment.js:…)␊
+        at file://.../ses/src/compartment.js:…`

packages/ses/NEWS.md

@@ -2,21 +2,28 @@ User-visible changes in SES:
 
 # Next release
 
+- Extracts `repairIntrinsics(options)` and `hardenIntrinsics()` from the
+  behavior of `lockdown(options)` so vetted shims can run between these
+  calls.
+  Any modifications to shared intrinsics survive if applied after
+  `repairIntrinsics()`.
 - In the SES-shim implementation of HardenedJS, all constructed compartments
-get the same safe `Date`
-constructor, that does not provide the ability to measure duration. It used
-to do this by having `Date.now()` return `NaN`, and to have calls on the
-constructor that would normally have returned an indication of the current date,
-instead return the corresponding invalid date indication. Now, all of these
-throw a `TypeError` whose message begins with `'secure mode'`. This aligns with
-the XS implementation of HardenedJS.
-- Similarly, In the SES-shim implementation of HardenedJS,
-all constructed compartments get the same safe `Math` namespace
-object that does not provide a working `random()` function. It used to do that
-by omitting the `random` property from the safe `Math` namespace object. Now,
-the safe shared `Math` namespace object has a `Math.random()` function that
-throws a `TypeError whose message begins with `'secure mode'`. This again
-aligns with the XS implementation of HardenedJS.
+  get the same safe `Date` constructor, that does not provide the ability to
+  measure duration.
+  It used to do this by having `Date.now()` return `NaN`, and to have calls on
+  the constructor that would normally have returned an indication of the
+  current date, instead return the corresponding invalid date indication.
+  Now, all of these throw a `TypeError` whose message begins with `'secure
+  mode'`.
+  This aligns with the XS implementation of HardenedJS.
+- Similarly, In the SES-shim implementation of HardenedJS, all constructed
+  compartments get the same safe `Math` namespace object that does not provide
+  a working `random()` function.
+  It used to do that by omitting the `random` property from the safe `Math`
+  namespace object.
+  Now, the safe shared `Math` namespace object has a `Math.random()` function
+  that throws a `TypeError whose message begins with `'secure mode'`.
+  This again aligns with the XS implementation of HardenedJS.
 
 # v0.18.6 (2023-08-07)
 

packages/ses/docs/guide.md

@@ -28,7 +28,7 @@ Hardened JavaScript:
 - Removes non-determinism by modifying a few built-in objects.
 - Adds functionality to freeze and make immutable both built-in JavaScript
   objects and program created objects and make them immutable.
-- Is (as SES) is a proposed extension to the JavaScript standard.
+- Is (tentatively named SES) a proposed extension to the JavaScript standard.
 
 Hardened JavaScript consists of three parts:
 - Lockdown is a function that irreversibly repairs and hardens an existing
@@ -39,6 +39,12 @@ Hardened JavaScript consists of three parts:
   globals and modules, but shared hardened primordials and limited access to
   other powerful objects in global scope.
 
+Lockdown consists of separable Repair Intrinsics and Harden Intrinsics phases,
+so that shims (other programs that alter JavaScript) may run between them.
+These shims are obliged to maintain the object capability safety invariants
+provided by Lockdown and must be carefully reviewed.
+We call these "vetted shims".
+
 ## What is SES?
 
 SES is an old umbrella term for the Hardened JavaScript effort, and while we
@@ -143,6 +149,53 @@ To use SES as a script on the web, use the UMD build.
 <script src="node_modules/ses/dist/ses.umd.min.js">
 ```
 
+## Using Hardened JavaScript with vetted shims
+
+Some modules depend on language features that may not be present in the
+underlying platform.
+Some of these shims will compose poorly with `lockdown()`.
+Lockdown will remove any property it finds on a shared intrinsic (like
+`Array.prototype`) that it does not recognize and then freeze all the shared
+intrinsics and all the objects transitively reachable through own properties
+and prototypes.
+
+So, if a shim adds `Array.prototype.collate`, running that shim before calling
+`lockdown()` will have no net effect and running this shim after calling
+`lockdown()` will throw an exception when attempting to assign or define that
+property.
+
+Lockdown consists of two phases: Repair Intrinsics and Harden Intrinsics.
+A shim can run between these phases and its effects will persist.
+The following programs are equivalent:
+
+```js
+lockdown(options);
+```
+
+And,
+
+```js
+repairIntrinsics(options);
+hardenIntrinsics();
+```
+
+And, an application that choses to call these also has the option of running
+shims between the two phases.
+
+```js
+import './non-ses-code-before-lockdown.js';
+import './ses-repair-intrinsics.js'; // calls repairIntrinsics.
+import './vetted-shim.js';
+import './ses-harden-intrinsics.js'; // calls hardenIntrinsics.
+import './ses-code-after-lockdown.js';
+```
+
+However, any such shim must preserve the qualities of Lockdown:
+all reachable objects in an empty compartment must be hardened and must not
+provide a way for isolated parties to communicate.
+So, application authors are responsible for ensuring these shims maintain their
+application’s integrity invariants.
+
 ## What Lockdown does to JavaScript
 
 Hardened JavaScript does not include any I/O objects providing "unsafe" [*ambient authority*](https://en.wikipedia.org/wiki/Ambient_authority).
@@ -204,7 +257,7 @@ Most Node.js-specific [global objects](https://nodejs.org/dist/latest-v14.x/docs
 * `WebAssembly`
 * `TextEncoder` and `TextDecoder`
 * `global`
-  * Use `globalThis` instead (and remember it is frozen).
+  * Use `globalThis` instead.
 * `process`
   * No `process.env` to access the process's environment variables.
   * No `process.argv` for the argument array.
@@ -244,15 +297,31 @@ makes those global objects available.
   and defer the construction of error objects and computed messages until an
   assertion fails.
 
-- `lockdown()` and `harden()` both freeze an object’s API surface (enumerable data properties).
-  A hardened object’s properties cannot be changed, only read, so the only way to interact with a
-  hardened object is through its methods. `harden()` is similar to `Object.freeze()` but more
-  thorough. See the individual [`lockdown()`](#lockdown) and [`harden()`](#harden) sections
-  below.
+- `repairIntrinsics` adds, removes, and replaces various properties of the
+  global environment and shared intrinsics.
+  Introduces `hardenIntrinsics`.
+
+- `hardenIntrinsics` freezes the transitive own properties and prototypes of
+  the shared intrinsics.
+  Introduces `harden`.
 
-- [`Compartment`](https://github.com/endojs/endo/tree/SES-v0.8.0/packages/ses#compartment) is
-  a global. Code runs inside a `Compartment` and can create sub-compartments to host other
-  code (with different globals or transforms). Note that these child compartments get `harden()` and `Compartment`.
+- [`harden()`](#harden) provides a shorthand for reliably freezing the
+  transitive properties and prototypes of other objects, such that the API
+  surface of these objects are tamper-proof when shared between otherwise
+  isolated programs.
+
+- [`lockdown()`](#lockdown) is a shorthand for `repairIntrinsics` and `hardenIntrinsics`.
+
+- [`Compartment`](https://github.com/endojs/endo/tree/SES-v0.8.0/packages/ses#compartment)
+  Code runs inside a `Compartment` and can create sub-compartments to host
+  other code (with different globals or code transforms).
+  The globals in a child compartment include the shared intrinsics including
+  `harden` and a batch of evaluators that run programs that will also be
+  confined to the compartment including `eval`, `Function`, and `Compartment`
+  itself.
+  Compartments can be created with support for loading modules.
+  Comaprtments constructed after `repairIntrinsics()` and `hardenIntrinsics()`
+  also confine the evaluation of modules.
 
 ## Realms
 
@@ -417,6 +486,21 @@ some covert communication channels).
 For a full explanation of `lockdown()` and its options, please click
 [here](./lockdown.md).
 
+## `repairIntrinsics()`
+
+Performs the first part of Lockdown: adding, removing, and replacing certain
+JavaScript intrinsics so that some intrinsics can be safely shared between
+confined programs.
+Running `repairIntrinsics()` introduces `hardenIntrinsics()`.
+
+## `hardenIntrinsics()`
+
+Performs the last part of Lockdown: hardening the shared intrinsics so they can
+be safely shared between confined programs.
+Running `hardenIntrinsics()` reveals the `harden()` function.
+(Harden is not useful until after `hardenIntrinsics()` and could interfere with
+the execution of repairs or shims if it were revealed earlier.)
+
 ## `harden()`
 
 `harden()` is automatically provided by `lockdown()`. Any code that will run inside a vat or a

packages/ses/docs/reference.md

@@ -11,27 +11,50 @@ comprehensive [Endo and Hardened JavaScript Programming Guide](./guide.md).
 The SES shim transforms ordinary JavaScript environments into Hardened JavaScript environments.
 
 On Node.js you can import or require `ses` in either CommonJS or ECMAScript modules, then call `lockdown()`. This is a *shim*. It mutates the environment in place so any code running after the shim can assume it’s running in a Hardened JavaScript environment. This includes the globals `lockdown()`, `harden()`, `Compartment`, and so on. For example:
+
 ```js
 require("ses");
 lockdown();
 ```
+
 Or:
+
 ```js
 import 'ses';
 lockdown();
 ```
 
 To ensure a module runs in a Hardened JavaScript environment, wrap the above code in a `ses-lockdown.js` module and import it:
+
 ```js
 import './non-ses-code-before-lockdown.js';
 import './ses-lockdown.js'; // calls lockdown.
 import './ses-code-after-lockdown.js';
 ```
+
 To use SES as a script on the web, use the UMD build.
+
 ```js
 <script src="node_modules/ses/dist/ses.umd.min.js">
 ```
 
+To run shims after `ses` repairs the intrinsics but before `ses` hardens the
+intrinsics, calling `lockdown(options)` is equivalent to running
+`repairIntrinsics(options)` then `hardenIntrinsics()` and vetted shims can run
+in between.
+
+```js
+import './non-ses-code-before-lockdown.js';
+import './ses-repair-intrinsics.js'; // calls repairIntrinsics.
+import './vetted-shim.js';
+import './ses-harden-intrinsics.js'; // calls hardenIntrinsics.
+import './ses-code-after-lockdown.js';
+```
+
+SES is vulnerable to any code that runs before hardening intrinsics.
+All such code, including vetted shims, must receive careful review to ensure it
+preserves the invariants of the OCap security model.
+
 ## Removed by Hardened JavaScript summary
 
 The following are missing or unusable under Hardened JavaScript: