Rust Support via tree-sitter's parser

[lpinilla]

Hi all,

I was reading your docs about how you managed to create the support for Ruby where you explain that you used the output from tree-sitter as an extractor.

we built a tool to read node-types.json and spit out a CodeQL database schema

This extraction process is not only straightforward, it’s also completely language-agnostic. That is, the process is entirely mechanical and works for any tree-sitter grammar.

So, is it possible to use tree-sitter's rust parser to build the database? If so, how could I parse the node-types.json generated by the parser?

[edoardopirovano]

Greetings, thanks for getting in touch with this question! tree-sitter did indeed allow us to build the Ruby extractor very quickly. This code is open source and you're welcome to take a look at it here to see how we did this: https://github.com/github/codeql/tree/main/ruby/extractor. There's still a fair bit of plumbing that needs to be written to convert the output from tree-sitter into TRAP files (which is the format the CodeQL CLI understands how to produce a database from). You could probably reuse a lot of this plumbing to get a basic extractor that turns the syntax of Rust files into a CodeQL database.

Having said that, the TRAP format as well as a number of other aspects of implementing your own extractor aren't particularly well documented, so it would really be quite a challenge to add a new language to CodeQL yourself. Additionally, even once an extractor is working there's still quite a lot of work that needs to be done before the database can be used to produce useful results. For reference, even though tree-sitter allowed us to get the extractor part of Ruby support for CodeQL working fairly quickly, writing the libraries and queries to then get meaningful results still required several engineers working full-time for over a year to get to where we are now.

Additionally, Rust is a rather different language from Ruby and presents some more challenges. In particular, it's likely that to properly analyze Rust code we'll need more information than just the AST of the files in the CodeQL database - we'll also need to see information like flags passed to rustc invocations which will require plumbing to observe the build process and produce additional information in the TRAP files recording this. This will make the extractor look more like our C++ extractor, which is much more complex than the Ruby one (the code for that is closed-source so you'll have to take my word for it) and relies on more internals of CodeQL (in particular, something called the tracer which helps CodeQL observe build processes) that aren't documented externally.

We're hoping to improve the experience of external contributors when developing an extractor at some point in the mid-term future, but unfortunately for now the lack of documentation would make this very tricky, so I would probably caution you against such an endeavour (especially for a fairly complex compiled language like Rust).

[lpinilla]

Thanks for the super detailed answer! I can see that this is a much more complex that requires the development and maintenance of several people to achieve it.

I hope that in the future this process is much more documented so that it can spark future contributions.

Thank you for your time!