Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add hook in tokenizer to recognize comments for speedier parsing #46

Open
mighdoll opened this issue Jan 9, 2025 · 1 comment · May be fixed by #60
Open

add hook in tokenizer to recognize comments for speedier parsing #46

mighdoll opened this issue Jan 9, 2025 · 1 comment · May be fixed by #60
Assignees
Labels
good first issue Good for newcomers

Comments

@mighdoll
Copy link
Contributor

mighdoll commented Jan 9, 2025

the wesl grammar currently recognizes line and block comments with preParse(). It's nice to recognize comments with a grammar. And having a grammar was powerful for an earlier version of the language that included semantic information inside comments. But now that the language is simpler, it'd be faster to recognize comments in the tokenizer.

  • Add a hook to the tokenMatcher to allow parsing entire comments into tokens. (Perhaps invent an api to pass a user provided function to the tokenMatcher..)
  • Add those comment tokens into the matchingLexer ignore set
  • profit!
@mighdoll mighdoll added this to wesl-js Jan 9, 2025
@github-project-automation github-project-automation bot moved this to Unprioritized in wesl-js Jan 9, 2025
@mighdoll mighdoll moved this from Unprioritized to Todo in wesl-js Jan 9, 2025
@mighdoll mighdoll added the good first issue Good for newcomers label Jan 14, 2025
@mighdoll
Copy link
Contributor Author

mighdoll commented Feb 2, 2025

@stefnotch has a fix pending for this. Looks to be a nice speedup

@mighdoll mighdoll moved this from Todo to In Progress in wesl-js Feb 2, 2025
@stefnotch stefnotch linked a pull request Feb 5, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

2 participants