[CS2] Comments

[GeoffreyBooth]

This PR aims to reimplement how we compile comments—both block (### ### or /* */) and end-of-line (# or //). This would close #4290 per the discussion in #4541. It would also help provide better support for plugins, as discussed in #4540, #4529 and #4563. Once we get this working with comments, the same techniques could apply to embedded JavaScript, which will also help plugins or pre-/postprocessors.

The basic idea is that when the lexer consumes a comment, instead of making a token for it, the comment data is attached to a nearby token as a property like our current locationData. The comments property hitches a ride on that token through the parser and rewriter, possibly getting moved onto a different token by the rewriter as appropriate; and then in nodes.coffee the comment gets output before or after the token it stowed away on.

Because the comment never becomes a token, the grammar never touches it; and so you can put block comments anywhere: foo = 2 ### yo! ### + 3. The tricky part is that not all tokens make it through the parser; and some that do get jumbled. The rewriter has significant work to do to make sure that the comment makes it across attached to the nearest surviving token, with enough information preserved that the comment can be output sensibly.

So, have you ever wondered what magic awaits in the rewriter, where streams of tokens are mutated while being looped over? Ever glimpsed parser.js and pondered the mysterious yy variable and gargantuan switch statement? Read grammar.coffee and considered how the hell the function at the top somehow gets rendered into JavaScript in parser.js, and what it does there? Well, this PR is for you!

[vendethiel]

PR best viewed with ?w=1..

[jashkenas]

Amazing! Heroic work. Bravo.

This is so much of a better architectural approach to block comments than what we were doing previously — that I think it should definitely be merged once y'all have signed off on the details.

If it works well, should we start passing through line comments and printing them as well?

[GeoffreyBooth]

If it works well, should we start passing through line comments and printing them as well?

Yes, that’s the plan. Line comments are already getting forwarded along to nodes.coffee, I just haven’t yet written the code to output them.

[GeoffreyBooth]

So I’ve gotten this to the point where only one test fails, and it’s because I can’t seem to get the indentation to stay correct across the code fragments. Anyone have any thoughts? It’s the commented-out test in test/comments.coffee.

#3132: Place block-comments nicely
Expected generated JavaScript to be:

/**
 * A dummy class definition
 *
 * @class
 */
var DummyClass;

DummyClass = (function() {
  class DummyClass {

    /**
     * @constructor
     */

    constructor() {}

  };


  /**
   * Singleton reference
   *
   * @type {DummyClass}
   */

  DummyClass.instance = new DummyClass();

  return DummyClass;

})();

but instead it was:

/**
 * A dummy class definition
 *
 * @class
 */
var DummyClass;

DummyClass = (function() {
  class DummyClass {
      /**
       * @constructor
       */
    constructor() {}

  };

  /**
   * Singleton reference
   *
   * @type {DummyClass}
   */
DummyClass.instance = new DummyClass();

  return DummyClass;

})();

I’m okay with declaring that block comments will no longer automatically have blank lines on either side of them; but:

1. the @constructor comment should really be at the same indentation as constructor(); and
2. DummyClass.instance should be indented (the same as return on the following nonblank line).

[GeoffreyBooth]

Okay, I’ve gotten that last test to pass. The handling of indentation for comments is now much improved, though I’ve discovered that inserting @tab before a node is handled in several places around nodes.coffee (like in Code and in Return) so there might be more places left that require adjusting indentation. If I’ve missed any, though, they just get block comments with jagged indentation; there should be no impact on code execution. I’ve also gone through the grammar and expanded the list of tokens that I think don’t survive contact with the parser, and improved the handling of moving comments off such tokens.

This might be all that needs to be done for block comments. The CoffeeScript codebase has very few such ### comments, so it’s not the best corpus to use to ensure that this PR is safe for general projects. If anyone has a big project with lots of block comments in it, please compile your code with this PR and let me know if anything breaks 😉

Next up: line comments.

[GeoffreyBooth]

Good god, this resolved a lot of issues.

[srb-]

Great work everyone. I finally got home and was able to try Flow with Beta4. It was brilliant to be able to compile a function and have Flow spit out a type error!

Now I'm stuck trying to declare a variable of a certain type, due to CS separating var declaration and use. Try this code:

# @flow

myNum ###: number ### = 55     # Flow errors out here

nyNum = "string"               # but I want an error here

Perhaps I'm missing something?

[GeoffreyBooth]

@srb- can you please open a new issue? And please include what JavaScript you’re aiming for. I don’t see anything on the Flow Comment Types page that corresponds with variable declarations, only function parameters and return values.

[connec]

See the docs on variable types: https://flow.org/en/docs/types/variables/

It looks like it's to do with the separation of assignment and declaration - all the examples type the var/let/const statement, and the problem doesn't occur if the var myNum line is typed. That's a more awkward thing to fix I suspect (might be easier to add an enhancement to Flow...).