feat(gatsby): add DANGEROUSLY_DISABLE_OOM (#13066)

* feat(gatsby): add DANGEROUSLY_DISABLE_OOM

* add telemetry to DANGEROUSLY_DISABLE_OOM

* remove test pages

* Revert "add telemetry to DANGEROUSLY_DISABLE_OOM"

This reverts commit 002b722.

* chore: rename and add a doc

* chore: rename

* chore: add some links

* chore: revert the name change

* Update scaling-issues.md

* Update scaling-issues.md

* Update scaling-issues.md

* Update scaling-issues.md

docs/docs/scaling-issues.md

@@ -0,0 +1,92 @@
+---
+title: Scaling Issues
+---
+
+In certain circumstances, your application may hit some scaling issues that necessitate workarounds. These workarounds should be treated as _temporary_ and should be revisited in the future as Gatsby scales to support larger applications with hundreds of thousands of pages.
+
+However -- until we get to that point, some workarounds are useful to consider if they unblock your team from deploying, developing, etc.
+
+> Just looking for possible solutions? [Jump ahead](#possible-solutions-to-scaling-issues)
+
+## What is a scaling issue?
+
+A scaling issue is evident if your application is unable to build due to an error _or_ if it is extremely slow in some lifecycle, e.g. `develop` or `build`. For example:
+
+- An "Out of Memory" occurs when building in CI
+- The `develop` lifecycle takes 10x as long as `build`
+
+and more. An example of a scaling error thrown to the console may look something like:
+
+```shell
+success open and validate gatsby-configs — 0.005 s
+success load plugins — 0.131 s
+success onPreInit — 0.283 s
+success initialize cache — 0.026 s
+success copy gatsby files — 0.071 s
+success onPreBootstrap — 0.009 s
+success source and transform nodes — 5.291 s
+success building schema — 0.623 s
+success createPages — 0.608 s
+success createPagesStatefully — 0.025 s
+success onPreExtractQueries — 0.000 s
+success update schema — 0.478 s
+success extract queries from components — 0.127 s
+⠄ run graphql queries — 310/728 2.51 queries/second
+<--- Last few GCs --->
+
+[10208:0000029380BC1810]   139036 ms: Scavenge 1374.2 (1423.1) -> 1373.3 (1424.1) MB, 2.2 / 0.0 ms  (average mu = 0.317, current mu = 0.262) allocation failure
+[10208:0000029380BC1810]   139039 ms: Scavenge 1374.3 (1424.1) -> 1373.5 (1424.6) MB, 2.0 / 0.0 ms  (average mu = 0.317, current mu = 0.262) allocation failure
+[10208:0000029380BC1810]   139043 ms: Scavenge 1374.4 (1424.6) -> 1373.6 (1425.1) MB, 2.1 / 0.0 ms  (average mu = 0.317, current mu = 0.262) allocation failure
+```
+
+## When can a scaling issue arise?
+
+A scaling issue _can_ arise in varied cases, but typically something like:
+
+- A Gatsby application with ~100K+ pages
+  - See [this issue](https://github.com/gatsbyjs/gatsby/issues/12343) for an example
+- Extremely large `json` files sourced with `gatsby-transformer-json`
+- Extremely large GraphQL queries, which are stored in memory in the `develop` lifecycle
+  - See [this issue](https://github.com/gatsbyjs/gatsby/issues/12566) for an example
+
+If you are seeing errors or slowness **and** your Gatsby app matches one of the above use-cases, it's very likely you are hitting some scaling issues.
+
+## Possible solutions to scaling issues
+
+It's difficult to pin down exactly _how_ to fix a scaling issue. We have some recommendations and workarounds that _may_ work for your application.
+
+Note: the application of these techniques should be considered analogous to a applying a bandage. A bandage solves the underlying issue, but at some indeterminate point in the future the underlying issue may be healed! In much the same way--treat these techniques as temporary and re-visit in the future if underlying scaling issues in Gatsby have since been resolved.
+
+### `DANGEROUSLY_DISABLE_OOM`
+
+Gatsby's caching mechanism persists a (possibly!) large `json` file to disk _in memory._ To opt-out of this, consider something like:
+
+```json
+{
+  "devDependencies": {
+    "cross-env": "^5.2.0"
+  },
+  "scripts": {
+    "build": "cross-env DANGEROUSLY_DISABLE_OOM=true gatsby build"
+  }
+}
+```
+
+This will prevent caching, so it's recommended to only use this alongside the `gatsby build` command.
+
+### `GATSBY_DB_NODES`
+
+In preparation for future versions of Gatsby, we've enabled **experimental** support for a different mechanism for the persistence of nodes: [Loki](https://www.npmjs.com/package/loki). It's challenging to assess whether this could lead to unforeseen issues without breaking changes, so we've exposed it behind a flag while we continue to assess the impact to Gatsby applications.
+
+Loki allows us to opt-in to possibly more performant internal operations and it _may_ resolve your scaling issues. If it does--please let us know! To opt-in to this experimental feature:
+
+```json
+{
+  "devDependencies": {
+    "cross-env": "^5.2.0"
+  },
+  "scripts": {
+    "build": "cross-env GATSBY_DB_NODES=loki gatsby build"
+  }
+}
+```

packages/gatsby/src/redux/__tests__/index.js

@@ -0,0 +1,47 @@
+const path = require(`path`)
+const fs = require(`fs-extra`)
+const { saveState, store } = require(`../index`)
+const {
+  actions: { createPage },
+} = require(`../actions`)
+
+jest.mock(`fs-extra`)
+
+describe(`redux db`, () => {
+  beforeEach(() => {
+    store.dispatch(
+      createPage(
+        {
+          path: `/my-sweet-new-page/`,
+          component: path.resolve(`./src/templates/my-sweet-new-page.js`),
+          // The context is passed as props to the component as well
+          // as into the component's GraphQL query.
+          context: {
+            id: `123456`,
+          },
+        },
+        { name: `default-site-plugin` }
+      )
+    )
+
+    fs.writeFile.mockClear()
+  })
+  it(`should write cache to disk`, async () => {
+    await saveState()
+
+    expect(fs.writeFile).toBeCalledWith(
+      expect.stringContaining(`.cache/redux-state.json`),
+      expect.stringContaining(`my-sweet-new-page.js`)
+    )
+  })
+
+  it(`does not write to the cache when DANGEROUSLY_DISABLE_OOM is set`, async () => {
+    process.env.DANGEROUSLY_DISABLE_OOM = true
+
+    await saveState()
+
+    expect(fs.writeFile).not.toBeCalled()
+
+    delete process.env.DANGEROUSLY_DISABLE_OOM
+  })
+})

packages/gatsby/src/redux/index.js

@@ -73,6 +73,10 @@ const store = Redux.createStore(
 
 // Persist state.
 function saveState() {
+  if (process.env.DANGEROUSLY_DISABLE_OOM) {
+    return Promise.resolve()
+  }
+
   const state = store.getState()
   const pickedState = _.pick(state, [
     `nodes`,