Multiple environments support

[inikulin]

Problem 1: Using parse5 in browsers, ReactNative, etc.

According to recent reports (#225, #220, #234, #226, #202, #213) people try to use parse5 in various environments that are distinct from Node.js. parse5 core functionality doesn't use any platform-specific API, however some advanced features like streaming support depend on Node.js API which renders parse5 unusable in other environments.

Problem 2: TypeScript typings

The other problem is the immaturity of TypeScript type dependencies distribution system, starting from vague semi-official recommendations from development team of how such dependencies should be managed and ending with the mutually exclusive issues that this suggested approach causes: #212, #232.

The Big Plan

• Transform parse5 into set of packages. Core parsing functionality will remain in parse5 package. Any advanced functionality will be distributed in separate packages (e.g. parse5-sax, parse5-parse-stream) with a peer dependency on core package.
• Each package will have it's own TypeScript typings.
• Move Node.js typings dependency to devDependencies and document that in order to use parse5 typing one need to install Node.js typings.
• Raise issue regarding types distribution in TypeScript repo

Ad-hoc solution for now

Unfortunately, there are several blockers that prevent us from implementing The Big Plan at the moment:

• It will obviously require some significant time to implement: we'll need to adopt something like lerna to manage multiple packages. Documentation and build system will require significant rework.
• There is a huge pending change set in https://github.com/HTMLParseErrorWG/parse5 fork which we'll need to merge once the job will be done there (hopefully in the near future). Going multi-package path now will make it almost impossible to merge this change set automatically.

So suggested solution for now is:

• Make API that depends on Node.js API lazy-loadable
• Remove Node.js typings dependency and document that it should be installed manually
• Remove types field from package.json, but still distribute typings within the package but suggest approach recommended by @fictitious in Move @types/node from runtime deps to dev deps #226 (comment) to enable them.

If everyone involved in aforementioned conversations is fine with the proposed solutions then I can roll out ad-hoc solution somewhere on weekend.

Milestones

• Merge recent changes from https://github.com/HTMLParseErrorWG/parse5. Close issues fixed in that fork, update docs about breaking changes accordingly.
• Update repo tooling and codebase:
  • Get rid of gulp in favour of npm commands
  • Use prettier
  • Switch to ES6 syntax
  • Update file naming in repo to use - instead of -.
• Move streaming APIs and htmlparser2 tree-adapter to separate packages
• Adopt https://github.com/lerna/lerna for publishing
• Refactor tests and move them to separate packages
• Move TS typings to @types repo
• Update docs with recent changes
  • Use treeAdapter to attach location information #189
  • Source code location: startCol, endCol, startLine, endLine
  • Implement scriptingEnabled flag #192
  • Use token objects as SAXParser event arguments #247
  • Changelog
• Decide what to do with the documentation

[domenic]

I am happy that jsdom consumers won't get these typing dependencies installed by default. I'd be even happier if they got no types at all, just JavaScript, but I guess that would require using DefinitelyTyped, which for some reason some people believe to be "legacy".

[fictitious]

Just a wild side note about TypeScript typings. They are distributed by nature - there is no central authority that recompiles everything on every submission and enforces consistency. And until no one tried to use parse5 and ReactNative together in one project, the system was effectively partitioned.

According to CAP theorem, in the presence of a network partition, one has to choose between consistency and availability.

But with typings, inconsistency makes them completely unusable for everyone because it stops the builds, making system effectively unavailable.

The outtakes from this for those who use typings, 🤔💭 I think are

• apparent success of TypeScript is a big paradox, partially fueled by illusion of consistency it provides
• typings distribution will always be unsolved problem (partition-intolerant, or "immature", if you like)
• one has always to be prepared to deploy local hacks to unblock the builds

[inikulin]

Hi everyone.

I've just released v4.0.0 that implements proposed ad-hoc solution to npm. Now you should be able to bundle parse5 for platforms other than Node.js. Please let me know if you encounter any additional problems.

[felixfbecker]

Is there a reason why the Node typings dependency can't simply use exactly the range of Node versions this module supports?

[fictitious]

@felixfbecker Not everyone who uses parse5 needs node typings as dependency, and having it as non-dev dependency is not harmless either - it was getting installed whenever parse5 was referenced indirectly (as dependency of some other dependency) in environments where installing node typings breaks TypeScript compilation. See #226 and https://stackoverflow.com/questions/48135164/typescript-duplicate-require-error-in-types-node

[RReverser]

in environments where installing node typings breaks TypeScript compilation

That was fixed separately when parse5 was split into core versions and one that requires Node.js. After that I don't see either why the typings had to be removed, as they're not conflicting with RN anymore.

[fictitious]

That was fixed separately when parse5 was split into core versions and one that requires Node.js

@RReverser I don't think this has happened yet, v4 that was published on npm does lazy loading of node-dependent parts but they are still in the same package.

Or was a split version published under a new name? If so, could someone post a link here?

As I understand, splitting into separate packages is still in the future, when it happens the package(s) that unconditionally depend on node will of course include node typings.

[RReverser]

v4 that was published on npm does lazy loading of node-dependent parts but they are still in the same package.

Yeah, my bad, I misunderstood what happened.

[felixfbecker]

I see why you would want to not have environmental typings like node as a dependency (more like a peerDependency). But why the weird requirement to map the typings in tsconfig instead of automatic resolution?

[fictitious]

But why the weird requirement to map the typings in tsconfig instead of automatic resolution?

It's not a requirement, strictly speaking. But with automatic dependency on node typings removed, does it really make sense to have parse5 typings to be resolved automatically to something that has hard-coded node typings reference?

Anyway, it was fine with me either way -types in package.json could probably have stayed to simplify life for TypeScript developers who are targeting node. TypeScript devs who are targeting browser and ReactNative would have to use different typings - without node typings reference - and would have to override automatic type resolution anyway. Now everyone has to do that. If this is a big problem - I don't know, maybe you guys should have spoken up a few days earlier.

[felixfbecker]

Well, it is a requirement for everyone who uses TypeScript and wants to update to the latest version.

@types/node defines a lot of globals so you should only really have one in your project that matches the Node version you are actually using, in a way it is a loose peerDependency (unlike a library you may depend on).

If the desire is to support the browser usage cleanly then imo it should be split into multiple packages where the browser package does not reference Node typings or Node API in any way.

I wasn't following this thread or project, I just got a PR from Greenkeeper to update to the latest version and the build was broken, so I had to research what changed and how to fix it. And as said the required tsconfig change seems very odd to me, I've never seen a package require that before.

[fictitious]

If the desire is to support the browser usage cleanly then imo it should be split into multiple packages where the browser package does not reference Node typings or Node API in any way.

Yes this is the agreed long-term solution (see The Big Plan bit above), it will just take some time to implement. The requirement to have weird thing in tsconfig.json is a temporary workaround.

[mhegazy]

Sorry if I late to the party. Our recommendation is to only include a .d.ts file in a package, iff it is auto generated from the source of the package (e.g. the package is written in typescript, and the declaration file is generated as part of the build). For all other uses, we recommend hosting the declaration file on DefinitelyTyped. The DefinitelyTyped process now allows for a change to a declaration file to make it to @types package within an hour or so. The @types package also already have a mechanism for handling dependencies, version updates, TS compiler version changes, etc..

@inikulin I do not believe the solution outlined in this list is the best path forward. It is a breaking change, but still does not allow the declaration file to be hosted in DT. seems like this is the worst of both worlds. if you are willing to take a breaking change, i would say remove the file altogether, and host it on DT instead.

[fictitious]

@mhegazy your recommendation is different from what TypeScript handbook says about publishing type declarations:

Now that you have authored a declaration file following the steps of this guide, it is time to publish it to npm. There are two main ways you can publish your declaration files to npm:

1 bundling with your npm package, or
2 publishing to the @types organization on npm.

If you control the npm package you are publishing declarations for, then the first approach is favored. That way, your declarations and JavaScript always travel together.

As I understand this project has followed the suggestion in the handbook. Is the handbook going to be updated?

[mhegazy]

Thanks for the pointer, should be updated now (microsoft/TypeScript-Handbook@884e4e8).