Reusable tokens

[mAAdhaTTah]

Suggestion by @LeaVerou (#1479 (comment)):

Btw we should probably have common tokens as re-usable variables or something (E.g. Prism.NUMBER). Currently every language needs to redefine numbers, and in most cases they're the same... We could also have factory helpers, e.g. Prism.keywords("foo", "bar", ...) that generate regexes for common things.
What do you think?

[RunDevelopment]

What are the use cases of this?

Let's take e.g. NUMBER:
What should it be like? Only decimal numbers? If so, are floating point numbers include (with or without exponents)? What about constants like NaN, Infinity or ∞?
And when you think about hexadecimal and binary numbers, and optional underscores within numbers and the whole thing becomes a mess. (And I don't need something like \d+)

I don't think that there are many places where we can get away with 'number': Prism.NUMBER or even 'number': [ number-style 1, 2, 3, 4, simple pattern for edge cases ].

Patterns like COMMENT or STRING look more promising but they also will fall short in many cases.

The cases where you can easily share patterns are between languages with very similar syntax and these are already taken care of by Prism.languages.extend.

So, where will they be used?

[mAAdhaTTah]

'number': Prism.NUMBER

I think places like these is the intention. If the same pattern isn't used in multiple places, then we don't get much and isn't worth it.

[LeaVerou]

Also the helper that generates keyword regexes could be general enough to be able to combine other regexes as alternatives. Then we could have a Prism.NUMBER for the actual numbers, and then we could do something like Prism.or(Prism.NUMBER, 'NaN', 'Infinity') for JS numbers for example. I think something like that has the potential to simplify many grammars, if done well.

[RunDevelopment]

Prism.or(Prism.NUMBER, 'NaN', 'Infinity')

Doesn't this imply that NUMBER will be a rather broad pattern, matching all kinds of numbers?
So in general, are we going to tolerating false positives to increases the applicability of these constants?

It might be ok because we don't valid syntax, but do we really want to be that inclusive?

[LeaVerou]

Doesn't this imply

How does it imply that? 🤔

[RunDevelopment]

How does it imply that?

If NUMBER didn't include stuff like e.g. hexadecimal numbers, it wouldn't work for JS.
Or did I take your example too literal? If so, I'm sorry.

But I would still like to hear the answer to the other questions. 😄