New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mini-parse refactors #60

Open

stefnotch wants to merge 63 commits into main from feat/mini-parse-rework

+7,879 −1,997

Contributor

stefnotch commented Jan 30, 2025 •

edited

Loading

While working with the mini-parse library, I found a few bits that I wanted to tweak. This PR does that, and tries to merge it into the import-rework branch. With these changes then in place, we can merge import-rework into the main branch.

This PR does the following

Replaces the mini-parse tokenizing and lexing with a new Stream abstraction. To avoid having to rewrite all the tests, the existing tokenizing and lexing implementations now simply call the Stream stuff under the hood.
Replaces the WESL tokenizing with a bespoke tokenizer.
- CommentsGrammar.ts got removed
Adds an assertUnreachable function. Super cool for saying that a branch of an if statement should never be reachable, and getting the Typescript compiler to check it
Adds types for expression elements, but does not use them yet.

Fix #46

stefnotch added 14 commits

January 30, 2025 14:22


          Add new abstraction for tokenizers

9a9a499


          Hook up new parser

484b3f9


          Remove unused code

f84769d


          Remove preparsing

3a372e8


          Add predeclared enumerants

cf3f96c


          Tweak import printing

dbeb4b7


          Fix template token parsing

6fa3338

I forgot about whitespaces and comments


          Test fixed template parser

f2ba245


          Fix import tests

f8dd239


          More tokenizer tests

ff1fed3


          Add span to old token type

083a505


          Prep for span tracking

d95c057


          Prep for component collection

3b964f0


          Disable failing tests

78e93c3

stefnotch marked this pull request as ready for review

February 1, 2025 19:39

mighdoll reviewed

View reviewed changes

linker/packages/linker/src/parse/WeslStream.ts Outdated Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/mini-parse/src/ParserCombinator.ts Outdated Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/linker/src/WESLGrammar.ts Outdated Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/mini-parse/src/Stream.ts Outdated Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/mini-parse/src/Stream.ts Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/mini-parse/src/Stream.ts Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/mini-parse/src/Stream.ts Outdated Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/mini-parse/src/Stream.ts Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/mini-parse/src/Stream.ts Outdated Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/mini-parse/src/stream/PeekStream.ts Outdated Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/linker/src/BindIdents.ts Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/mini-parse/src/stream/IgnoringStream.ts Outdated Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/linker/src/test/TransformBindingStructs.test.ts Outdated Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/linker/src/AbstractElems.ts Show resolved Hide resolved

mighdoll reviewed

View reviewed changes

linker/packages/linker/src/CommentsGrammar.ts Show resolved Hide resolved

stefnotch added 26 commits

February 5, 2025 11:22


          Remove context from extended results

a34f11e


          Add more efficient map function

527e407


          Remove unused prop

a26b5cf


          Split runParser into two

d588b48


          Simplify simpleParser

1b038e1


          Only reset stream where necessary

ea03f27


          Add custom sep implementation

14f5d67


          Name the anonymous functions

1f56b31


          Add specialized combinator

48f40b2


          Fix opt combinator type

41fa77d


          Make wesl comparison whitespace insensitive

2a14bc3


          Add a test file to benchmark import statement parsing

6c6aad6


          Move collect array into its own combinator

2a745a0


          Update imports parser to latest spec

76a367e


          Add all imports at once

74d8156


          Set vscode to no longer suggest wrong APIs

a0baf69


          Merge branch 'chore/bump-deps-0' into feat/mini-parse-rework

5027e3b


          Fix merging bug

1d3925c


          Optimize text cover

f951f2c


          Tweaks

f985a60


          Fix opt parser

e638c82


          Fix text cover

dd48a91


          Undo yes api tweak

4927a1c


          Add tests case from https://github.com/gfx-rs/wgpu/pull/6898/files

6bf4208


          Fix enum parsing

0e858ff


          Remove todo from toParser

Contributor Author

stefnotch commented Feb 8, 2025

I think this is ready to be merged. Time to get M1 done!

stefnotch mentioned this pull request

Handle new import spec #41

Closed

stefnotch added 2 commits

February 8, 2025 16:17


          Merge remote-tracking branch 'origin/main' into feat/mini-parse-rework

07c6c25


          Address some todos

b27c96d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet