Sourcemap for JS

[alehander92]

I am adding support for sourcemaps.

You can pass --sourceMap:on and that implies --lineDir:on .
I've fixed some of the linedir for js and I removed some generated comments from system/jssys.nim)

I am basically adding a new pass that works on the generated source and builds source nodes based on the line directives: that seems to be correct (as jsgen doesn't generate other comments, so if one hasn't emitted // line , it should be ok, and you can emit #line in C too).

I'll add some tests: for now it seems to work for some files in my project, but I haven't dogfooded it too much. The implementation is based on mozilla's source-map lib

[alehander92]

I also have locally working tracebacks to Nim source based on those sourcemaps based on their lib, but one can also write a handler that depends only on line directives

[alehander92]

Now, also with support for generation of name nodes so one can hover on variables. I use tokenization of js lines which seems very hacky, but it seems to work for a big file of mine (it seems ok enough for an initial impl, if you can think of a more elegant way which is very probable, please advise me)

[alehander92]

thanks to @treeform for the naming idea, I didn't even realize this is supported by sourcemaps: actually it seems he had the first sourcemaps implementation, but without a PR

[Araq]

Rebase please.

[alehander92]

Ok, rebased. The postprocess raw js output approach seems complicated , but still simpler than the alternative seems to be big refactoring of jsgen itself(this might be a good idea?)

[Araq]

I know it says "optimize me" but this tokenizer is really easy enough to make lazy. The compiler's lexer module shows how a lexer can be written.

[alehander92]

turned to iterator?

[Araq]

I'm sure you find some Context object where this can be put into. Globals make the compiler hard to use as a library.

[alehander92]

TODO: i think i can remove that

[Araq]

but still simpler than the alternative seems to be big refactoring of jsgen itself(this might be a good idea?)

Certainly, but it's also more work. Your decision.

[alehander92]

Just rebasing for now, it still needs to be at least optimized

[alehander92]

Ok, @Araq if you don't have objections, I am going to eventually start a refactoring next week like

Why including the ES6 modules support? it seems to me that it would be easier to add when we are refactoring the whole backend anyway, but I can try to somehow split it in two PR-s as well

AST -> 

normal , default mode: 
  one JS code tree (one module) 

es6 modules mode ( `-d:esmodules`): 
  JS code tree for each module 

JS code tree -> [Sourcemap?] -> Raw code

[Araq]

Fine with me but it will be an interesting discussion of whether to use the existing AST as the "JS code tree" or use a different one. I've spent years on this question and I'm slightly in favour of "use the existing AST" but it is your choice.

[alehander92]

Well honestly I think we don't even need a JS code tree: a list of JS tokens with good enough Nim source location info should be sufficient for generating the sourcemaps. The important thing is just to have this tokens instead of generating text and lex it again which is the hacky part.

[alehander92]

Are there any other benefits from AST->AST here
? e.g. enabling Javascript-specific formatting / optimizing passes ? I doubt it, but I haven't thought about it too much

[Araq]

Are there any other benefits from AST->AST here
? e.g. enabling Javascript-specific formatting / optimizing passes ? I doubt it, but I haven't thought about it too much

Every other transformation is also AST->AST, we have good debugging helpers for these and you can for example run a general optimization pass after "JS code generation", seems it can be useful one day.

[alehander92]

Well, rendering is T -> string, so making codegen AST -> Tokens and rendering Tokens -> string is also sensible.

JS AST can be a more powerful abstraction, but this will be an almost complete rewrite of jsgen, compared to splitting generated text into generated tokens in the current jsgen

e.g.

write "a = b[e + 2]" # today

write "a" "=", "b", "[", "e", "+", "2", "]" # tokens

write jsAsgn(name"a", jsIndex(name"b", jsInfix(op"+", name"e", lit(2)))) # ast

Reusing PNode-s might seem easier(but resulting in similar code). However what do we do for js constructs which have no equivalent in PNode-s?

[Araq]

However what do we do for js constructs which have no equivalent in PNode-s?

Well these need to enumerated first, but I always thought of a nkVerbatim node that is a string that is rendered as it is.

[alehander92]

Ok, this might work. verbatim nodes are usually useful

I mostly feel like PNode construction is very verbose, something like breeze, but maybe even simpler, based on just autogenerated template shortcuts seem much cleaner to me compared to the newTree/new combo

asgn(ident"a", index(ident"b", infix(ident"+", ident"e", 2)))
vs
nkAsgn.newTree(ident"a", nnkIndex.newTree(ident"b", nnkInfix.newTree(ident"+", ident"e")))

but probably the con is that it's easier to clash with local variables ?

[Araq]

The tree construction procs grew into existance without much design and it shows.

[alehander92]

Ok, so the initial AST -> JS code tree -> [Sourcemap?] -> javascript design remains, but with JS code tree based on PNod, I'll push a WIP early to make sure we're on the same track.

The tree construction procs grew into existance without much design and it shows.

So should I add my own set of helpers here, or should I follow the style in the compiler

[krux02]

@alehander42 I personally prefer the newTree syntax, because I don't need to think: "does this node kind have a short template? How does it look like?" It is always the same, and there is not a single tree that can't be created using newTree. But I would really prefer it, if it would be shorter. For example just t:

nkAsgn.t(ident"a", nnkIndex.t(ident"b", nnkInfix.t(ident"+", ident"e")))

[Araq]

So should I add my own set of helpers here, or should I follow the style in the compiler

Your own helpers, if they help.

[alehander92]

@krux02 yeah, the t is also nice, but still you have nk everywhere : my idea was

• on compile time iterate through all the nnkNodeKind (except ident maybe?)
• generate a template helper with name <=> nnk<name as title>.newTree

This way you know that the node kinds have a shorter template and that it works in exactly the same way.

[krux02]

@alehander42 that seems to be a good local solution. But currently I don't think it should go into macros, too many exported function symbols.

[alehander92]

agreed

[alehander92]

@Araq the verbatim node is useful, but it's actually not sufficient: e.g. we generate a ternary if, we need the final repr of the child, but it's still a PNode.

One possible hack might be to reuse one TNodeKind for the "JS-only" constructs and additional field/magic string inside which would be a hint to the renderer.

(probably I can just use if/else for ternary cond, but the point stays for other JS-only constructs)

[Araq]

I think this means the JS-AST should get its own type.

[treeform]

What is stopping this? We all need this.

[alehander92]

@treeform i need to finish the rewrite : i wrote some of the gen -> JavaScript AST part and some of the rendering JavaScript AST part but its a lot of manual work and i just have it in a branch somewhere

I am not convinced we need that for sourcemaps: i feel that we can get them even with simpler generation(even with my first patch which just postprocesses source): Javascript AST sounds good but i am not sure if the team wants to maintain and actually write custom passes for it, maybe it's not beneficial enought after all

[alehander92]

Sourcemaps could basically work with some bugs in electron based on analyzing and lexing the raw output of jsgen, I can try to return to this version

[Araq]

Friendly ping @alehander92. Can we get back to the commit 73073d5

I changed my mind and AST to AST codegen isn't required. The mentioned commit was good enough.

[alehander92]

ok, i'll take a look at it later!

[alehander92]

so:

• sourcemap:on/off: sourcemap:on should turn in options linedir:on as well
• linedir:on now generates full paths so sourcemaps can link to them
• sourcemaps might one day be used on C backend with the same flag
• linedir fixes for jsgen

[timotheecour]

@alehander92

• --sourceMap isn't documented in --fullhelp

linedir:on now generates full paths so sourcemaps can link to them

• line directives don't always produce absolute paths, eg:

if (!(eqStrings(cstrToNimstr(arr_13416016), makeNimstrLit("Hello World!")))) {
// line 55 "tostring.nim"
failed_assert_impl_3627680(makeNimstrLit("tostring.nim(55, 10) `$cstring(unsafeAddr arr) == \"Hello World!\"` "));
}

// line 57 "/Users/timothee/git_clone/nim/Nim_prs/tests/system/tostring.nim"
function takes_13435237(c_13435239) {

=> see test case here timotheecour#110

[alehander92]

ah good notes @timotheecour will take a look maybe next week, sorry!