refactor(gatsby): use GatsbyIterable + extract common tools for querying #32172
Conversation
gatsbot
bot
added
the
status: triage needed
vladar
removed
the
status: triage needed
vladar
force-pushed
the
vladar/common-query-utils
branch
from
d6345ce to
c56a0e0
Compare
| export type FilterValueNullable = // TODO: merge with DbComparatorValue | ||
| | string | ||
| | number | ||
| | boolean | ||
| | null | ||
| | undefined | ||
| | RegExp // Only valid for $regex | ||
| | Array<string | number | boolean | null | undefined> | ||
|
|
||
| // This is filter value in most cases | ||
| export type FilterValue = | ||
| | string | ||
| | number | ||
| | boolean | ||
| | RegExp // Only valid for $regex | ||
| | Array<string | number | boolean> | ||
|
|
||
| // The value is an object with arbitrary keys that are either filter values or, | ||
| // recursively, an object with the same struct. Ie. `{a: {a: {a: 2}}}` | ||
| export interface IInputQuery { | ||
| [key: string]: FilterValueNullable | IInputQuery | ||
| } | ||
| // Similar to IInputQuery except the comparator leaf nodes will have their | ||
| // key prefixed with `$` and their value, in some cases, normalized. | ||
| export interface IPreparedQueryArg { | ||
| [key: string]: FilterValueNullable | IPreparedQueryArg | ||
| } | ||
|
|
There was a problem hiding this comment.
Thos types are moved here from datastore/in-memory/indexing.ts and datastore/in-memory/run-fast-filters.ts
| /** | ||
| * Transforms filters coming from input GraphQL query to mongodb-compatible format | ||
| * (by prefixing comparators with "$"). | ||
| * | ||
| * Example: | ||
| * { foo: { eq: 5 } } -> { foo: { $eq: 5 }} | ||
| */ | ||
| export function prepareQueryArgs( | ||
| filterFields: Array<IInputQuery> | IInputQuery = {} | ||
| ): IPreparedQueryArg { | ||
| const filters = {} | ||
| Object.keys(filterFields).forEach(key => { | ||
| const value = filterFields[key] | ||
| if (_.isPlainObject(value)) { | ||
| filters[key === `elemMatch` ? `$elemMatch` : key] = prepareQueryArgs( | ||
| value as IInputQuery | ||
| ) | ||
| } else { | ||
| switch (key) { | ||
| case `regex`: | ||
| if (typeof value !== `string`) { | ||
| throw new Error( | ||
| `The $regex comparator is expecting the regex as a string, not an actual regex or anything else` | ||
| ) | ||
| } | ||
| filters[`$regex`] = prepareRegex(value) | ||
| break | ||
| case `glob`: | ||
| filters[`$regex`] = makeRe(value) | ||
| break | ||
| default: | ||
| filters[`$${key}`] = value | ||
| } | ||
| } | ||
| }) | ||
| return filters | ||
| } | ||
|
|
There was a problem hiding this comment.
Also moved from datastore/in-memory/run-fast-filters.ts
| @@ -297,7 +297,7 @@ class LocalNodeModel { | |||
| runQueryActivity.start() | |||
| } | |||
|
|
|||
| const { entries, totalCount } = runFastFiltersAndSort({ | |||
| const { entries, totalCount } = await getDataStore().runQuery({ | |||
There was a problem hiding this comment.
Now querying through datastore - the basis for future lmdb runQuery implementation
There was a problem hiding this comment.
Just checking if I'm reading things correctly and understand the plan - in this PR both "redux" and lmdb have same implementation (returning Promise.resolve(runFastFiltersAndSort(args))). In next PR(s) lmdb datastore will use its own (potentially falling back to same thing if something is impossible or just not implemented yet)?
| return { | ||
| gqlType, | ||
| entries: entries.map(node => { | ||
| // With GatsbyIterable it happens lazily as we iterate | ||
| this.trackInlineObjectsInRootNode(node) | ||
| return node | ||
| }), | ||
| totalCount, |
There was a problem hiding this comment.
Behavior change: switching from eager to lazy tracking of inline objects during iteration.
Now entries is an instance of GatsbyIterable, so we can apply this lazily. This makes a difference for lmdb because each iteration is essentially a separate query (reset cursor, re-apply filters, etc, which can be slow).
vladar
added
the
bot: merge on green
Description
This PR mostly affects various utilities related to querying:
lmdbrunQueryimplementation.GatsbyIterableGatsbyIterableforrunQueryand various node iteration methodsThere are several minor behavior changes, I'll comment on them inline.