Implement token list API

[ksss]

This is a PR intended for discussion.

I want a token list.

I am implementing a RuboCop extension for RBS. In RuboCop, indentation and spacing are adjusted based on the positions of various tokens such as comments and (.

Problem

If I try to create these features from the results of RBS::Parser, I have to implement complex processing. This includes searching for token positions from RBS::Location objects and finding the position information of end-of-line comments from all locations.

Example 1: Search space in block.

# Search block start char `{`
# I hope there is no literal `“{”`...
lbrace_length = method_type.location.source.index('{')

# Search char before '{'
char_before_lbrace_length = method_type.location.source.rindex(/[^\s]/, lbrace_length)

if char_before_lbrace_length + 2 != lbrace_length
  add_offence(...)
end

Example 2: Search space between any token.

scanner = StringScanner.new(source)
tokens = []
pos = 0
until scanner.eos?
  case
  when scanner.scan('[')
    pos += 1
    tokens << [:pLBRACKET, pos]
  when scanner.scan(']')
    pos += 1
    tokens << [:pRBRACKET, pos]
  when ...

I understand that my use case is unique, so I believe there is little need to modify the existing parsing process.

It would be helpful to have a method to obtain a sequence of tokens as a new API.

Use case of token list in RuboCop

https://github.com/rubocop/rubocop/blob/12fd014e255617a08b7b42aa5df0745e7382af88/lib/rubocop/cop/layout/extra_spacing.rb

Proposal for token list API

Low level

I propose a low-level API called RBS::Parser#_lex, following the example of _parse_signature and similar methods. This low-level API aims to obtain the necessary information for a sequence of tokens using minimal C code.
It is desirable to be able to obtain all tokens, including comments.

High level

I propose a high-level API called RBS::Parser#lex. The name lex is inspired by Prism#lex. This high-level API will wrap the sequence of tokens obtained from _lex, making it more convenient to handle.

[ksss]

If possible, it would be nice to have a line break token as in Prism.

[soutaro]

@ksss Can you fix the steep type check failure? I plan to implement supports for line breaks and comment tokens, on the top of this PR.

[ksss]

Thank you for reviewing. I fixed type checking.

I plan to implement supports for line breaks and comment tokens, on the top of this PR.

GREAT! THANKS!

[soutaro]

🎉