refactor(gatsby): load config and plugins in worker #31773

pieh · 2021-06-04T15:13:01Z

Description

This PR allows worker process to load gatsby-config and therefore also execute node APIs

[ch32027]

pieh · 2021-06-05T08:54:27Z

packages/gatsby/src/bootstrap/load-plugins/load.ts

@@ -71,9 +71,9 @@ export function resolvePlugin(
  rootDir = (!_.isString(plugin) && plugin.parentDir) || rootDir

  // Only find plugins when we're not given an absolute path
-  if (!existsSync(pluginName)) {
+  if (!existsSync(pluginName) && rootDir) {


to appease TS as rootDir can be undefined - this need some verification if it will do what is expected tho before this PR is ready

From my testing it seems correct, but will leave this unresolved for now

pieh · 2021-06-08T08:39:20Z

packages/gatsby-cli/src/reporter/prepare-stack-trace.ts

+  if (frame.getFileName().includes(`webpack:`)) {
+    // if source-map-register is initiated, stack traces would already be converted
+    return {
+      column: frame.getColumnNumber() - 1,
+      line: frame.getLineNumber(),
+      source: frame
+        .getFileName()
+        .substr(frame.getFileName().indexOf(`webpack:`))
+        .replace(/webpack:\/+/g, `webpack://`),
+      name: null,
+    }
+  }
+


this is interesting - once we import (directly or indirectly) gatsby-telemetry in worker, stack traces are different (due to @turist/fetch installing source map handler that map stack traces to source files). This just adjust stack traces produced this way so they are still compatible with our codeframe printer for html generation errors

…ading config and plugins

… worker

pieh · 2021-06-09T16:15:43Z

integration-tests/gatsby-cli/__tests__/build.js

+    logs.should.contain(
+      `success open and validate gatsby-configs, load plugins`
+    )


I merged 2 activities together as loading config and plugins really seem like single step (and it was just cleaner code-wise after putting it in single function to be executed by worker) - can backtrack from it to go back to current 2 steps (or maybe just figure out better label than currently used one for merged activity)

LekoArts

LGTM

LekoArts · 2021-06-10T06:12:21Z

packages/gatsby/src/bootstrap/load-config-and-plugins.ts

+  siteDirectory: string
+  processFlags: boolean
+}): Promise<{
+  config: any


I guess we can't type config at the moment?

Maybe we could, but requires more effort that is "tangential" to this PR (IMO) - it's any currently in master branch too ( configModule type in

gatsby/packages/gatsby/src/bootstrap/get-config-file.ts

Lines 17 to 24 in db946a5

export async function getConfigFile(

rootDir: string,

configName: string,

distance: number = 3

): Promise<{

configModule: any

configFilePath: string

}> {

)

We do have config type for what we store in redux -

gatsby/packages/gatsby/src/redux/types.ts

Lines 60 to 82 in db946a5

export interface IGatsbyConfig {

plugins?: Array<{

// This is the name of the plugin like `gatsby-plugin-manifest

resolve: string

options: {

[key: string]: unknown

}

}>

siteMetadata?: {

title?: string

author?: string

description?: string

siteUrl?: string

// siteMetadata is free form

[key: string]: unknown

}

// @deprecated

polyfill?: boolean

developMiddleware?: any

proxy?: any

pathPrefix?: string

mapping?: Record<string, string>

}

but this doesn't contain for example flags (I did look to have it type briefly, but saw there is lot of unrelated things to handle, so decided to not pursue it for this PR)

* refactor(gatsby): single function to load config and plugins * expose loadConfigAndPlugins function in worker * don't rely on cwd when loading plugins * add test veryfing config can be loaded and APIs executed in worker * adjust cli integration tests to cover for activity change (merging loading config and plugins * fix functions (flag need to be handled before plugins loading) * fix build error stack traces when ... gatsby-telemetry is imported in worker * satisfy ts * ensure double slashes (?)

gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Jun 4, 2021

pieh added status: blocked This issue/PR can't be solved at the moment and shouldn't be closed/merged and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Jun 4, 2021

pieh force-pushed the pieh/load-config-in-worker branch from 1fa55f5 to 6f1b0df Compare June 4, 2021 15:21

pieh changed the title ~~refactor(gatsby): single function to load config and plugins~~ refactor(gatsby): load config and plugins in worker Jun 5, 2021

pieh commented Jun 5, 2021

View reviewed changes

pieh force-pushed the pieh/load-config-in-worker branch 2 times, most recently from 416b86c to 86eda16 Compare June 8, 2021 08:20

pieh changed the base branch from pieh/worker-tests to pieh/reporter-in-worker-doesnt-crash June 8, 2021 08:20

pieh force-pushed the pieh/reporter-in-worker-doesnt-crash branch from 13a4be3 to c699a8f Compare June 8, 2021 08:36

pieh force-pushed the pieh/load-config-in-worker branch from 86eda16 to 091592f Compare June 8, 2021 08:36

pieh commented Jun 8, 2021

View reviewed changes

pieh force-pushed the pieh/reporter-in-worker-doesnt-crash branch from c699a8f to 52157ae Compare June 8, 2021 14:23

pieh force-pushed the pieh/load-config-in-worker branch from 091592f to ef9c739 Compare June 8, 2021 14:24

Base automatically changed from pieh/reporter-in-worker-doesnt-crash to master June 8, 2021 15:07

pieh added 9 commits June 8, 2021 17:09

refactor(gatsby): single function to load config and plugins

469d272

expose loadConfigAndPlugins function in worker

0730a2a

don't rely on cwd when loading plugins

b51938c

add test veryfing config can be loaded and APIs executed in worker

910df18

adjust cli integration tests to cover for activity change (merging lo…

ea508ad

…ading config and plugins

fix functions (flag need to be handled before plugins loading)

05b222b

fix build error stack traces when ... gatsby-telemetry is imported in…

fd46f43

… worker

satisfy ts

7501211

ensure double slashes (?)

ce916eb

pieh force-pushed the pieh/load-config-in-worker branch from ef9c739 to ce916eb Compare June 8, 2021 15:09

pieh removed the status: blocked This issue/PR can't be solved at the moment and shouldn't be closed/merged label Jun 8, 2021

pieh marked this pull request as ready for review June 8, 2021 15:10

pieh commented Jun 9, 2021

View reviewed changes

LekoArts approved these changes Jun 10, 2021

View reviewed changes

LekoArts mentioned this pull request Jun 10, 2021

feat(gatsby): PQR workers can access page & static queries #31852

Merged

pieh merged commit 81458a0 into master Jun 10, 2021

pieh deleted the pieh/load-config-in-worker branch June 10, 2021 08:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(gatsby): load config and plugins in worker #31773

refactor(gatsby): load config and plugins in worker #31773

pieh commented Jun 4, 2021 •

edited

Loading

pieh Jun 5, 2021

pieh Jun 8, 2021

pieh Jun 8, 2021

pieh Jun 9, 2021

LekoArts left a comment

LekoArts Jun 10, 2021

pieh Jun 10, 2021

	export async function getConfigFile(
	rootDir: string,
	configName: string,
	distance: number = 3
	): Promise<{
	configModule: any
	configFilePath: string
	}> {

	export interface IGatsbyConfig {
	plugins?: Array<{
	// This is the name of the plugin like `gatsby-plugin-manifest
	resolve: string
	options: {
	[key: string]: unknown
	}
	}>
	siteMetadata?: {
	title?: string
	author?: string
	description?: string
	siteUrl?: string
	// siteMetadata is free form
	[key: string]: unknown
	}
	// @deprecated
	polyfill?: boolean
	developMiddleware?: any
	proxy?: any
	pathPrefix?: string
	mapping?: Record<string, string>
	}

refactor(gatsby): load config and plugins in worker #31773

refactor(gatsby): load config and plugins in worker #31773

Conversation

pieh commented Jun 4, 2021 • edited Loading

Description

pieh Jun 5, 2021

Choose a reason for hiding this comment

pieh Jun 8, 2021

Choose a reason for hiding this comment

pieh Jun 8, 2021

Choose a reason for hiding this comment

pieh Jun 9, 2021

Choose a reason for hiding this comment

LekoArts left a comment

Choose a reason for hiding this comment

LekoArts Jun 10, 2021

Choose a reason for hiding this comment

pieh Jun 10, 2021

Choose a reason for hiding this comment

pieh commented Jun 4, 2021 •

edited

Loading