docgen: Support ES2019, approved ES2020 syntax

[aduth]

Previously: #18045 (comment)
Related: #19931 (comment), #19831

Docgen uses a different JavaScript parser (Espree) than that which is used in the plugin build process (Babel). This hasn't been much of an issue thus far, but with especially new syntaxes, it can cause errors to be thrown when running npm run docs:build and with the corresponding pre-commit step.

Examples:

• ES2019 optional catch binding
  • Was intended to be used in URL: Conform to URL Living Standard definition of valid URL #19871
• ES2020 optional chaining
  • Was intended to be used in Block Editor: LinkControl: Prevent focus loss in edit mode toggle #19931

Originally, I thought it might just be a matter of keeping the espree dependency up to date, assuming that it's been updated for newer versions of the specification.

This is unfortunately not as simple as it might seem, since one of the breaking changes introduced in espree@5.0.0 is to remove the attachComment feature, which is pretty critical to how docgen works (or at least is currently implemented). Thus, upgrading to the latest version would require a pretty significant refactor to the code, one where we'd likely emulate how comment attachment worked previously, using the comment (and perhaps tokens) option(s) to infer where a comment occurs in relation to an exported member. I started down this path on Friday, but it seems significant enough to warrant whether it's worth the effort, considering if the alternative to Doctrine would bring with it a different parser implementation as well (e.g. TypeScript).

One thing I noted in the process of this is that we configure the ECMAScript version here:

gutenberg/packages/docgen/src/engine.js

Line 15 in 1fb3c38

ecmaVersion: 2018,

This is particularly relevant for my previous comment where I mention being unable to use ES2019 features. It's pretty clear by this code why this wouldn't be expected to work 😄 I haven't tested yet, but it's possible that Espree versions prior to the breaking 5.0 version might have support for newer versions of ECMAScript, so that we could still use those language features without bumping to the latest Espree version.

Task: Allow new syntaxes to be parsed without error by docgen.

Proposed solution:

One of:

1. Replace Doctrine and Espree with TypeScript (docgen: remove doctrine #18045).
2. Or: Bump Espree to the latest version, including necessary refactoring to docgen to support removal of attachComment option.
3. Or: Optionally bump Espree to the last version including attachComment option, and check to see whether it can support ES2019 and ES2020 as the configured ecmaVersion option.

Follow-up tasks:

Revisit the "intended to be used" examples from above, to introduce the desired syntax.

• https://github.com/WordPress/gutenberg/pull/19871/files#r372503756
• Block Editor: LinkControl: Prevent focus loss in edit mode toggle #19931 (comment)

[sainthkh]

I'm currently working on this. I refactored getExportTokens, getExportEntries and getJsdocFromToken. And currently working on getIntermediateRepresentation.

[youknowriad]

It seems like I'm blocked by this in #21021

@sainthkh Do you know when you'll be able to push something?

[aduth]

I checked to see what we could do with Espree < 5.0.0 (since its this upgrade which is the hardest to accommodate):

Both the current version used (4.0.0) and the latest version in the 4.x (4.1.0) do support ES2019, so we could at least bump it to that without any other changes. However, this doesn't help in cases where we'd plan to use ES2020 (like optional chaining).

https://github.com/eslint/espree/tree/v4.0.0#what-ecmascript-2019-features-do-you-support

[ZebulanStanphill]

I'm also blocked by this; I'm working on a PR to replace all usage of lodash.has with vanilla JS, and the changes use a lot of optional chaining (and a bit of nullish coalescing). Trying to commit my changes causes a pre-commit hook error.

[sainthkh]

I'm not sure. But I'm planning to do this next week. I refactored all IR functions and should fix index.js and check if it works correctly.

[sainthkh]

I've finished replacing espree and doctrine.

I'm now writing custom type-to-string function for this project because:

• Object is treated as any.
• if typescript cannot find the definition of an interface, it's any. So, File becomes any.
• it ignores jsdoc nullable types: ?string is simple string.

Currently, I listed types for test like below:

/**
 * A function with many params.
 *
 * @param p undocumented type
 * @param {any} p any
 * @param {*} p all types
 * @param {?} p unknown
 * @param {unknown} p unknown TS
 * @param {string} p string
 * @param {number} p number
 * @param {bigint} p bigint
 * @param {boolean} p boolean
 * @param {symbol} p symbol
 * @param {undefined} p undefined
 * @param {null} p null
 * @param {never} p never
 * @param {Object} p Object
 * @param {Object.<string, number>} p jsdoc record type
 * @param {File} p Random Type name
 * @param {'string literal'} p string literal
 * @param {42} p number literal
 * @param {typeof J} p type query
 * @param {number[]} p number array
 * @param {Array<WPElements>} p TypeScript array
 * @param {Array.<WPElements>} p jsdoc style array
 * @param {[string, number]} p simple tuple
 * @param {[TypeChecker, SourceFile, string?]} p tuple with optionalType
 * @param {string | number} p union type
 * @param {X & Y} p intersection type
 * @param {?string} p jsdoc nullable 1
 * @param {string?} p jsdoc nullable 2
 * @param {number=} p jsdoc optional type
 * @param {number} [p] jsdoc optional type 2
 * @param {number} [p=42] jsdoc optional with default
 * @param {(string | number)} [p] parenthesized type
 * @param {!string} p jsdoc non-nullable type
 * @param {string!} p jsdoc non-nullable type
 * @param {[string, ...X]} p rest type
 * @param {...number[]} p jsdoc variadic type
 * @param {(x: number, y: Test) => number} p function type
 * @param {(a: string, b: string) => {x: string, y: string}} p function + type literal
 * @param {(k: () => number) => React.FC} p function arg
 * @param {() => void} p functino void return type
 * @param {new () => T} p constructor type
 * @param {{x: number, y: XY}} p type literal
 * @param {{[setting:string]:any}} p indexable interface
 * @param {{j: (a: string) => number, k: number}} p function as type literal property type
 * @param {Object} p jsdoc type literal
 * @param {string} p.x0 property 0
 * @param {XXX} [p.x1] property 1
 * @param {number} [p.x2=11] property 2
 * @param {function(b): c} p jsdoc function type
 * @param {keyof X} p jsdoc type operator 1: keyof
 * @param {readonly Y} p jsdoc type operator 2: readonly
 * @param {unique symbol} p jsdoc type operator 3: unique
 * @param {T['key']} p indexed access type
 * @param {{readonly [P in keyof T]: T[P]}} p mapped type
 * @param {T extends U ? X : Y} p conditional type
 * @param {T extends (...args: any[]) => infer R ? R : any} p infer type
 * @param {import('typescript').Statement} p import type
 * @param {asdf} p identifier
 *
 */

And I need to write a function that generate correct string from these types.

[aduth]

Hi @sainthkh , this is great work, and from my own experience trying to refactor it, I know it's not straight-forward either.

Maybe it can wait for your pull request, but I'm interested to learn a bit more about what your specific plan is for this. Based on your message, is it safe to assume you're planning to use TypeScript to do most of the work for parsing the modules, and the challenges you're encountering currently have more to do with the outputs of that parsing not always being directly usable?

[sainthkh]

Hi @aduth

Your assumption is correct. I'm taking the path of solution 1: replace espree and doctrine with typescript. In my local branch, docgen doesn't depend of espree and doctrine any more. And it only uses typescript to do all the things.

It solves problems like showing (timestamp:number)=>void, {[setting: string]: string} correctly. But it creates a long list of things to discuss. I'll list that in the PR.

You're also right. As I said in the comment above, default typechecker doesn't generate type string we want. I don't write it here, but it was full of anys. Currently, I'm writing code to generate the correct/useful type strings.

Maybe after finishing that generator, I can publish the PR. If something happens after that, I'll write about them here.

[sainthkh]

Coding is done. You can read the code in my local branch.

But I cannot open a PR now because this PR needs a lot of discussions and explanations of my decisions of implementation. Examples are like:

• Object vs. object
• Array vs. []
• etc. etc.

And there are some ideas that I couldn't add here because they make big PR even bigger. Examples are:

• When and where should we show warnings?
• Should we turn off some eslint errors if that package is covered by tsc?
• etc. etc.

I'm now organizing and writing those PR comments.

@youknowriad and everyone, I'm sorry. I failed to upload this PR this week.

[youknowriad]

Don't worry @sainthkh I unblocked myself already. And thanks a lot for your efforts. Everyone has its own schedule and that's totally understandable.