WIP: use dedent for de-indentation in lexer, fix #15184

[a-mr]

When first line in a doc comment is empty, lexer.nim memorizes indentation at it and keeps one space on the following lines:

##
## x

tokenized as comment tok.literal =

 x

(with space before x) which is considered a block quote in RST.

[a-mr]

bummer: there is no dedent in csources, because csources is of version 0.20.0.

It worked so well in my local copy :-(

[timotheecour]

@a-mr
you're hitting #16646, see workaround mentioned there which works:

in strutils, change as follows:

since (1, 3):
  func indentation*(s: string): Natural = ...
  func dedent*(s: string, count: Natural = indentation(s)): string {.rtl,
      extern: "nsuDedent".} = ...

after that you're hitting timotheecour#521
for which there's also a workaound:
add this to compiler/lexer.nim:

proc dedent(a: string): string =
  # pending https://github.com/timotheecour/Nim/issues/521
  let b = a
  let i = indentation(b)
  dedent(a, i)

then this works

[Araq]

Why can't we fix the existing logic instead...

[timotheecour]

To avoid duplicating code. Code reuse is good.

[a-mr]

@Araq , @timotheecour . Bringing into consideration another edge case. Assume we really want to start comment from a quote:

proc f* =
  ##   Quote
  ## Paragraph
  discard

Current logic sets indentation to 0 at "Quote", then it will re-adapt indentation to 0 again at "Paragraph", so we get tok.literal wrongly as:

Quote
Paragraph

To make indentation right in such cases we need to look ahead through the entire string and find non-whitespace-character with minimal indentation. It's what dedent does. So we are back to re-inventing dedent.

I think, we can avoid the workarounds proposed by timotheecours and just copy a (short, few lines long) implementation of dedent into lexer.nim.

Alternative solution

You can specify that zero or one space in any comment correspond to zero indentation, then grows linearly. This solution avoids to do any de-indentation, except the trivial crop that can be done on each line separately #x -> x, # x -> x,

#  x

-> " x", etc.

initial indentation, spaces after ##	resulting indentation
0	0
1	0
2	1
..	..
n	n-1

The side effect of such decision is that there will appear a block quote if e.g. 2 spaces are present after "#"

const pi = 3.14 ##  I made a mistake and put 2 spaces accidentally and would get a block quote all of a sudden :-)

[Araq]

To avoid duplicating code. Code reuse is good.

So do it in a way that doesn't slow down things. Also: I avoid helpers from the stdlib nowadays because you never know if it gets "fixed" in an incompatible manner because it's "inconsistent with Python/Unix/some other proc somewhere else".