AST as input

[olivier-work]

I have a JSON representation of an SpiderMonkey AST. Is there a way to use this in Flow?

[mroch]

we don't have a way to do this at the moment. flow does parse into an AST and then runs type inference on that AST, so in theory we could translate from SpiderMonkey to our internal AST and go straight to inference.

the AST wouldn't have any type annotations in it, would it? maybe if it comes from babel or another Flow-aware parser? if not, flow would be able to catch some stuff, but also would be limited enough that this wouldn't be a very general thing for us to support. if you have an AST with type annotations like flow ast outputs, that seems like a different story.

[olivier-work]

To give you some context, i am attempting to run flow on lua code. What i have so far is a lua to SpiderMonkey AST parser via lua2js.

I understand that lua is not something you may officially support, but there might be value on allowing the CLI to take a flow AST as input.

@mroch I do intend to have type annotations.

Feel free to close the issue if it is too off-topic.

[mroch]

i think there could be some other use cases that are substantially similar but sound less outrageous than running lua through flow :) (that sounds cool, make sure you let us know how it goes)

one example I could imagine is if you wanted to put some Babel transforms in front of flow, to transpile experimental/custom language features back to ES6 without having to write it to a JS file and then reparse it again.

i'll leave this open

[rattrayalex]

if you wanted to put some Babel transforms in front of flow, to transpile experimental/custom language features back to ES6 without having to write it to a JS file and then reparse it again.

That's exactly what I want to do!

[rattrayalex]

@mroch I'm planning to add this functionality soon (will submit a PR if you're open to that). Do you have any tips on where I could get started?

[rattrayalex]

@mroch now that Babylon can output an ESTree-compliant AST, this seems potentially more relevant / useful.

Do you have any pointers on how to get started or estimates of what will be required? I'm new to OCaml so fairly unsure of where to start.

EDIT: As far as I can tell, this would need to do the inverse of https://github.com/facebook/flow/blob/master/src/parser/estree_translator.ml – reify OCaml objects from a JSON tree. Not sure how onerous that is without knowledge of OCaml. May be trivial?

[AriaMinaei]

Other transpilers such as ESLisp and livescript-next and even sweet.js are now producing ESTree-compatible AST. If flow had an ergonomic way to use these transpilers as a pre-processor, then it could theoretically type-check code written in these languages too. Some features like autocomplete may not work out of the box, obviously, but afaict, type-checking should.

Putting babel-transforms (as @mroch mentioned) in front of flow could be another use-case. This c uld enable us to try early-stage ES proposals in our code while still enjoying the benefits of flow.

I'd imagine the user would have to do something like this to enable such pre-processors:

;; .flowconfig

; '.file_extension' -> 'cli-command-to-parse-file-and-return-ast-in-json'
module.preprocessor_mapper.extension='.sweetjs' -> 'sjs --ast $filepath '

I can imagine how this could make parsing slower, but it probably wouldn't make it too slow if $ sjs itself was fast enough and didn't have a long startup time.

This would be a significant feature imho. There is still a lot of innovation happening in the land of syntactic transpilers. And I'd be more than happy to go back to them instead of using the ES syntax. (Livescript was an absolute joy to use, for example). The only thing holding me back is that none of these transpilers can integrate with flow. Having a good type-checker still outweighs the innovative features of languages like ESLisp. But the two don't have to be mutually exclusive.

[rattrayalex]

Yes, exactly. My use-case is LightScript which is built on Babel.

[AriaMinaei]

Since it's been a year since this discussion was last active with the contributors, I'd like to ask them again what they think of this idea.

@samwgoldman @gabelevi @avikchaudhuri @mroch. I apologize if my mentioning you directly is annoying :)

[nmn]

Since Babel is doing some major work on the parser for 7.0 we should probably let that shake out before making a plan for this.

[andreiglingeanu]

@nmn Is that somehow related to Babel itself? Is flow's parser somehow hardly hardcoded into the current system?

[calebmer]

Is there a reason you can’t print the AST and then use that?

[rattrayalex]

@calebmer sourcemaps, essentially. You want errors to point to the right places in the original code.

[AriaMinaei]

@calebmer Is there a reason you can’t print the AST and then use that?

That could work, but I'd guess a significant number of cycles will be wasted when the pre-processor compiler emits JS code, only for that code to have to be parsed again by flow. We could eliminate this extraneous parse step in flow, by allowing it to consume the AST encoded as a JSON string from the pre-processor.

[brigand]

I need this for reasons other than alt-js languages. You can do many cool things with a procedural macro system.

For example, you can take a function that executes database queries, and type check it against the database schema. On that same note, you can define the returned shape of a query without relying on a human to match things up manually and keep them in sync.

This is stopping me from introducing flow in the main server project at work which has a ton of calls like this:

await ctx.esFetch2(
  'some_index', // which table/index to query
  ['requesterId'], // return these fields
  { targetId: someId }, // an actual query
);

Schema (simplified)

{
  "requesterId": "id",
  "targetId": "id",
  "created": "epoch"
}

I need to define that given 'some_index', the array of returned fields, and the schema: I generate a type definition for the function call.

await ((ctx.esFetch2: any): (
  (
    'some_index',
    Array<'requesterId' | 'targetId' | 'created'>,
    {|requesterId?: string, targetId?: string, created?: number|},
  ) => Promise<Array<{ _id: string, requesterId: string }>>
))(
  'some_index',
  ['requesterId'],
  { targetId: someId },
);

Wow that does not read well (or probably compile) but hopefully it gets the point across. Note that we're replacing the type annotation for the entire function, not just its return value. Very few people would write code like this directly, but it gives us much stronger type checking in this case. Also, our second and third arguments have types based on the schema which is stored in a separate file. It wouldn't type check if we tried to request fields that don't exist, or query on fields that don't exist.

---

There's some prior work in this area; specifically I've been reading about rust recently, where you have a powerful procedural macro system that could handle the above. To be clear, I'm absolutely not proposing a JS macro system, only some form of code that generates flow type annotations on the fly.

I'm not an expert on flow or type systems, but I don't think flow can support this API currently without a ton of bug-prone boilerplate code, or terrible solutions involving generating the type annotations on disk in a watch task with random ids and injecting those ids into the code you're editing.

---

Another API that I could use this for is what could be called a "prop prefix", where the properties are generated based a prefixed string, followed by a static set of suffixes. This is very convenient in react/redux code, where it encourages passing in loading and error states without requiring extra effort from the consumer of the selector.

Note: it isn't important if this is a good pattern or not, but should be possible to type check either way.

someSelector(id, 'foo')

// has type (roughly)
{ foo: Object, fooLoading: boolean, fooError: ?Object }

I have absolutely no expectation of flow to explicitly support either of these patterns, but by allowing arbitrary type annotation generation, you enable people to keep the freedom in API design while continuing to leverage flow (i.e. not using 'any').

---

For implementation, I have 0 experience with ocaml and similar, but my thought is that you could essentially have a mapping of source code ranges to type overrides. The key would be a file path and a character range within the file, and the value would be a generated type, implicitly imported from a virtual type definition file containing all generated types in the project.

Since the generated types don't actually impact the AST flow sees, you don't break editor completion. Wouldn't this work roughly the same as 5 + 'hi' where flow sees that as `((5 + 'hi): string)' without it breaking anything relying on text ranges?

---

Sorry for the long post, but wanted to give some concrete examples and pretend I know what I'm talking about as far as implementation.