feat: add assembly SourceLocation

[vlopes11]

This commit introduces [SourceLocation], a structure linked to a [Token] via [TokenStream].

It will allow the link between a MASM source and a parsed [Operation].

related issue: #857

[bobbinth]

Thank you! I left a couple of test-related comments inline, but I have a more general question:

What is the reason for introducing TokenLines and SourceToken structs? It seems to me that to support mapping of tokens to source locations we just need to replace these lines with some parse_line_tokens() function which would return a list of tokens together with their SourceLocation's. Or does this complicate things for some reason?

[bobbinth]

Thank you! Looks good! I left a couple of small comments inline.

[bitwalker]

Just wanted to throw my two cents in, because I've implemented this kind of infrastructure a few times before, have done so for the compiler already, and @grjte and I discussed extracting that crate out of the miden-ir repo into its own repo and using it with AirScript as well (that way we have a common set of diagnostics infrastructure for all our frontends). It may be worth exploring using it here as well.

For some additional context on how this generally gets used with a frontend, this parsing infrastructure builds on top of that diagnostics crate, and can be used with lalrpop or any hand-written parser. Instantiation of the DiagnosticsHandler isn't shown here, nor are the diagnostics that get emitted shown here, but the result are rustc-like diagnostics emitted to the terminal with all the fancy bits you'd expect.

At a high-level, the diagnostics crate provides a few things:

• It tracks SourceFiles in a CodeMap (de-duplicated) with a unique SourceId, source spans (i.e. SourceSpan) are byte ranges paired with a SourceId that lets us reconstruct the original source span or convert to a filename + line/column number when emitting diagnostics.
• Recording of diagnostics emitted by the program via the DiagnosticsHandler. A Diagnostic has a severity (error/warn/etc), an optional error code, a message, and one or more associated spans with optional notes.
• Customizable emission of diagnostics (i.e. they can be recorded in-memory, printed, or ignored)
• A Span<T> wrapper type, a Spanned trait, and proc macro support for deriving Spanned types

All of the above is oriented primarily towards the frontend of a compiler/interpreter (i.e. used during parsing/compilation), but it can be easily integrated into emission of a more stable debug info format in a variety of different ways depending on what properties you want from the debug info. For example, you could serialize the entire CodeMap to preserve all of the original sources; or you could simply emit a map of SourceId to corresponding filenames, which would be much smaller, but slightly less convenient to work with. You could also convert SourceSpan to a more stable Location equivalent which contains filename/line/column, at the cost of increased memory usage.

Emitting Diagnostics

#[derive(Spanned)]
struct Foo {
    #[span]
    span: SourceSpan,
}
impl Foo {
    fn compile(&self, diagnostics: &DiagnosticsHandler) -> anyhow::Result<()> {
       ...
       match bar() {
           ok @ Ok(_) => ok,
           Err(err) => {
               diagnostics.diagnostic(Severity::Error)
                       .with_message(format!("failed to bar: {}", &err))
                       .with_primary_label(self.span(), "in this foo")
                       .emit()
               Err(err)
        }
    }
}

[bobbinth]

All looks good! Thank you!

[bobbinth]

@bitwalker - this looks cool! It might be too big of a lift to integrate this into the assembler in a single go - but I think we'll need to use something like CodeMap relatively soon. So, there might be a good way to move to this type of a structure in stages.

[vlopes11]

@bitwalker yes, using the SourceId seems to be the right path. I'll proceed merging this PR for these initial steps, but on the long run we should target for a structure like you described.

We should target using a centralized implementation too, but let's see a bit down the road