Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lexer is a mess #3

Open
michaelgwelch opened this issue May 16, 2012 · 0 comments
Open

Lexer is a mess #3

michaelgwelch opened this issue May 16, 2012 · 0 comments

Comments

@michaelgwelch
Copy link
Owner

Perhaps I had a good reason 6 years ago, but it looks like the Lexer is trying to do half the job of the parser. All kinds of crazy state is being kept track of.

I recommend taking a new look at this and seeing if I can't identify X different types of tokens and simplify the Lexer to simply worrying about the tokens and not the context.

Issues:

  • Worrying about startOfLine.
  • Blank lines being skipped rather than just returning Token.EndOfLine (what did I think Token.EndOfLine couldn't be handled as an empty statement in Parser?)
  • other weird stuff.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant