From ca1aea9acdbf95269a40c0ea83f3f81a206004d8 Mon Sep 17 00:00:00 2001 From: Derek Schuff Date: Mon, 1 May 2017 17:58:31 -0700 Subject: [PATCH 1/9] Add proposal for applying sourcemap spec to wasm --- SourceMaps.md | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 SourceMaps.md diff --git a/SourceMaps.md b/SourceMaps.md new file mode 100644 index 00000000..a092dfc7 --- /dev/null +++ b/SourceMaps.md @@ -0,0 +1,70 @@ +# Source Maps for WebAssembly files on the web + +## Background +[Source maps](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k) +are a debug-information format used on the web, generated by toolchains that target +JavaScript, either from other languages (e.g. TypeScript, Emscripten) or from +JavaScript itself (e.g. Closure, minifiers). A browser that supports source maps +can display the original source in its developer tools instead of the compiled +JavaScript that the VM actually executes. It can support setting breakpoints +specified as source lines, single-stepping through source lines, and showing +source lines in stack traces; however it cannot support getting or setting +variables from the original source code because the format is not "full" debug +info: it only maps locations from the compiled code back to locations in the +source files (where a generated-code location consists of a line and column +number, and a source location consists of file, line, and column). Generated +files use a specially-formatted comment to specify a URL for a source map that +describes the file. + +Source maps can be made to serve the same purpose for WebAssembly binary files, +allowing parity with asm.js. By re-using an existing standard, browsers can +take advantage of their existing source map support without having to make +significant changes to their developer tools. + + +## Alternatives + +An alternative method is to specify location mapping information in some way +other than source maps. For example, it could be encoded directly into the wasm +binary the way function and local names are written in the "name" section. This +is more convenient if the VM itself uses the information (for example to +generate a stack trace). However existing developer tools implementations are +designed around the source map model of passing a URL from the VM to the dev +tools code, which fetches and interprets the information. Therefore it is +epxected that it will be much simpler to use "real" source maps than to design +another format. + +Source maps also have known limitations (e.g. source variables cannot be +described). A complete debug info format will eventually be needed; however the +intention of this specification is to allow for easy integration with existing +developer tools implementations in browsers. + + +## Spec + +With few exceptions, most aspects of the source map +[spec](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/) +may be applied directly to wasm. +Because wasm is a binary format (and thus does not have lines and columns, or +comments per se), some simple reinterpretation of concepts is needed. + +### Locations + +Source maps specify how to map a (zero-based) line and column position in +generated code to a file, line, and column location in the source. For +wasm binary locations, the line number is always 0, and the column number +is interpreted as a byte offset into the wasm binary content (this results +in a more efficient encoding than using the line number instead). +Source locations are interpreted as in the source map spec. + +### Linking generated code to source maps + +When the generated code is JavaScript, it includes a specially-formatted line +at the end, which is the URL of the associated source map. For wasm, a custom +section named `"sourceMappingURL"` contains the URL. +As with source maps, +the URL is defined as in [RFC3986](https://tools.ietf.org/html/rfc3986) (e.g. +it must be percent-encoded if necessary) and it may be a data URI. The URL +is also resolved according to the source map spec, and may also be specified +with the `SourceMap:` HTTP header. + From dbb3f06051e9afa36014ddd494dea8e8a9cf36af Mon Sep 17 00:00:00 2001 From: Derek Schuff Date: Mon, 1 May 2017 18:07:52 -0700 Subject: [PATCH 2/9] move to Web.md --- SourceMaps.md | 70 ------------------------------------------------- Web.md | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 70 deletions(-) delete mode 100644 SourceMaps.md diff --git a/SourceMaps.md b/SourceMaps.md deleted file mode 100644 index a092dfc7..00000000 --- a/SourceMaps.md +++ /dev/null @@ -1,70 +0,0 @@ -# Source Maps for WebAssembly files on the web - -## Background -[Source maps](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k) -are a debug-information format used on the web, generated by toolchains that target -JavaScript, either from other languages (e.g. TypeScript, Emscripten) or from -JavaScript itself (e.g. Closure, minifiers). A browser that supports source maps -can display the original source in its developer tools instead of the compiled -JavaScript that the VM actually executes. It can support setting breakpoints -specified as source lines, single-stepping through source lines, and showing -source lines in stack traces; however it cannot support getting or setting -variables from the original source code because the format is not "full" debug -info: it only maps locations from the compiled code back to locations in the -source files (where a generated-code location consists of a line and column -number, and a source location consists of file, line, and column). Generated -files use a specially-formatted comment to specify a URL for a source map that -describes the file. - -Source maps can be made to serve the same purpose for WebAssembly binary files, -allowing parity with asm.js. By re-using an existing standard, browsers can -take advantage of their existing source map support without having to make -significant changes to their developer tools. - - -## Alternatives - -An alternative method is to specify location mapping information in some way -other than source maps. For example, it could be encoded directly into the wasm -binary the way function and local names are written in the "name" section. This -is more convenient if the VM itself uses the information (for example to -generate a stack trace). However existing developer tools implementations are -designed around the source map model of passing a URL from the VM to the dev -tools code, which fetches and interprets the information. Therefore it is -epxected that it will be much simpler to use "real" source maps than to design -another format. - -Source maps also have known limitations (e.g. source variables cannot be -described). A complete debug info format will eventually be needed; however the -intention of this specification is to allow for easy integration with existing -developer tools implementations in browsers. - - -## Spec - -With few exceptions, most aspects of the source map -[spec](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/) -may be applied directly to wasm. -Because wasm is a binary format (and thus does not have lines and columns, or -comments per se), some simple reinterpretation of concepts is needed. - -### Locations - -Source maps specify how to map a (zero-based) line and column position in -generated code to a file, line, and column location in the source. For -wasm binary locations, the line number is always 0, and the column number -is interpreted as a byte offset into the wasm binary content (this results -in a more efficient encoding than using the line number instead). -Source locations are interpreted as in the source map spec. - -### Linking generated code to source maps - -When the generated code is JavaScript, it includes a specially-formatted line -at the end, which is the URL of the associated source map. For wasm, a custom -section named `"sourceMappingURL"` contains the URL. -As with source maps, -the URL is defined as in [RFC3986](https://tools.ietf.org/html/rfc3986) (e.g. -it must be percent-encoded if necessary) and it may be a data URI. The URL -is also resolved according to the source map spec, and may also be specified -with the `SourceMap:` HTTP header. - diff --git a/Web.md b/Web.md index 469d9a5c..126cfd55 100644 --- a/Web.md +++ b/Web.md @@ -154,6 +154,78 @@ Transcoding failure is detected by `decodeURIComponent`, which may throw `URIError`. If it does, the WebAssembly module will not validate. This validation rule is only mandatory for Web embedding. +## Source Maps for WebAssembly files on the web + +### Background +[Source maps](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k) +are a debug-information format used on the web, generated by toolchains that target +JavaScript, either from other languages (e.g. TypeScript, Emscripten) or from +JavaScript itself (e.g. Closure, minifiers). A browser that supports source maps +can display the original source in its developer tools instead of the compiled +JavaScript that the VM actually executes. It can support setting breakpoints +specified as source lines, single-stepping through source lines, and showing +source lines in stack traces; however it cannot support getting or setting +variables from the original source code because the format is not "full" debug +info: it only maps locations from the compiled code back to locations in the +source files (where a generated-code location consists of a line and column +number, and a source location consists of file, line, and column). Generated +files use a specially-formatted comment to specify a URL for a source map that +describes the file. + +Source maps can be made to serve the same purpose for WebAssembly binary files, +allowing parity with asm.js. By re-using an existing standard, browsers can +take advantage of their existing source map support without having to make +significant changes to their developer tools. + + +### Alternatives + +An alternative method is to specify location mapping information in some way +other than source maps. For example, it could be encoded directly into the wasm +binary the way function and local names are written in the "name" section. This +is more convenient if the VM itself uses the information (for example to +generate a stack trace). However existing developer tools implementations are +designed around the source map model of passing a URL from the VM to the dev +tools code, which fetches and interprets the information. Therefore it is +epxected that it will be much simpler to use "real" source maps than to design +another format. + +Source maps also have known limitations (e.g. source variables cannot be +described). A complete debug info format will eventually be needed; however the +intention of this specification is to allow for easy integration with existing +developer tools implementations in browsers. + + +### Spec + +With few exceptions, most aspects of the source map +[spec](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/) +may be applied directly to wasm. +Because wasm is a binary format (and thus does not have lines and columns, or +comments per se), some simple reinterpretation of concepts is needed. + +#### Locations + +Source maps specify how to map a (zero-based) line and column position in +generated code to a file, line, and column location in the source. For +wasm binary locations, the line number is always 0, and the column number +is interpreted as a byte offset into the wasm binary content (this results +in a more efficient encoding than using the line number instead). +Source locations are interpreted as in the source map spec. + +#### Linking generated code to source maps + +When the generated code is JavaScript, it includes a specially-formatted line +at the end, which is the URL of the associated source map. For wasm, a custom +section named `"sourceMappingURL"` contains the URL. +As with source maps, +the URL is defined as in [RFC3986](https://tools.ietf.org/html/rfc3986) (e.g. +it must be percent-encoded if necessary) and it may be a data URI. The URL +is also resolved according to the source map spec, and may also be specified +with the `SourceMap:` HTTP header. + + + ## Security WebAssembly's [security](Security.md) model should depend on the From cc5a7972e0b4379c8011ca8e57ba8bc2663424a0 Mon Sep 17 00:00:00 2001 From: Derek Schuff Date: Tue, 2 May 2017 17:19:58 -0700 Subject: [PATCH 3/9] fix typo --- Web.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Web.md b/Web.md index 126cfd55..be3aa6b0 100644 --- a/Web.md +++ b/Web.md @@ -187,7 +187,7 @@ is more convenient if the VM itself uses the information (for example to generate a stack trace). However existing developer tools implementations are designed around the source map model of passing a URL from the VM to the dev tools code, which fetches and interprets the information. Therefore it is -epxected that it will be much simpler to use "real" source maps than to design +expected that it will be much simpler to use "real" source maps than to design another format. Source maps also have known limitations (e.g. source variables cannot be From 341f5a79db22a155d757e84bee7e2cd322f829c0 Mon Sep 17 00:00:00 2001 From: Derek Schuff Date: Wed, 10 May 2017 11:58:33 -0700 Subject: [PATCH 4/9] Clarify that SourceMap HTTP header only applies to response-based API, and reference WHATWG URL spec --- Web.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/Web.md b/Web.md index be3aa6b0..fbe33cf4 100644 --- a/Web.md +++ b/Web.md @@ -218,11 +218,13 @@ Source locations are interpreted as in the source map spec. When the generated code is JavaScript, it includes a specially-formatted line at the end, which is the URL of the associated source map. For wasm, a custom section named `"sourceMappingURL"` contains the URL. -As with source maps, -the URL is defined as in [RFC3986](https://tools.ietf.org/html/rfc3986) (e.g. -it must be percent-encoded if necessary) and it may be a data URI. The URL -is also resolved according to the source map spec, and may also be specified -with the `SourceMap:` HTTP header. +The URL is defined as in the the WHATWG +[URL spec](http://url.spec.whatwg.org), and is +resolved according to the source map spec, +For wasm modules with an associated HTTP response (e.g. those using +the response-based compile or instantiation +[APIs](#additional-web-embedding-api)) the URL may also be specified +using the `SourceMap:` HTTP header as with JavaScript source maps. From 4bf5625ee42a2348f025bb1f61a5d9e08d9a6ca8 Mon Sep 17 00:00:00 2001 From: Derek Schuff Date: Wed, 10 May 2017 15:39:36 -0700 Subject: [PATCH 5/9] fix typo and use line 1 instead of 0 --- Web.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Web.md b/Web.md index fbe33cf4..143cb4d4 100644 --- a/Web.md +++ b/Web.md @@ -208,9 +208,9 @@ comments per se), some simple reinterpretation of concepts is needed. Source maps specify how to map a (zero-based) line and column position in generated code to a file, line, and column location in the source. For -wasm binary locations, the line number is always 0, and the column number +wasm binary locations, the line number is always 1, and the column number is interpreted as a byte offset into the wasm binary content (this results -in a more efficient encoding than using the line number instead). +in a more efficient encoding than using the line number field instead). Source locations are interpreted as in the source map spec. #### Linking generated code to source maps @@ -218,7 +218,7 @@ Source locations are interpreted as in the source map spec. When the generated code is JavaScript, it includes a specially-formatted line at the end, which is the URL of the associated source map. For wasm, a custom section named `"sourceMappingURL"` contains the URL. -The URL is defined as in the the WHATWG +The URL is defined as in the WHATWG [URL spec](http://url.spec.whatwg.org), and is resolved according to the source map spec, For wasm modules with an associated HTTP response (e.g. those using From 741f9897e0ae16ec9af686dddfcc2c4a95c144d1 Mon Sep 17 00:00:00 2001 From: Derek Schuff Date: Thu, 24 Aug 2017 10:12:55 -0700 Subject: [PATCH 6/9] s/wasm/WebAssembly --- Web.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Web.md b/Web.md index 143cb4d4..ee4857c5 100644 --- a/Web.md +++ b/Web.md @@ -181,7 +181,7 @@ significant changes to their developer tools. ### Alternatives An alternative method is to specify location mapping information in some way -other than source maps. For example, it could be encoded directly into the wasm +other than source maps. For example, it could be encoded directly into the WebAssembly binary the way function and local names are written in the "name" section. This is more convenient if the VM itself uses the information (for example to generate a stack trace). However existing developer tools implementations are @@ -200,23 +200,23 @@ developer tools implementations in browsers. With few exceptions, most aspects of the source map [spec](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/) -may be applied directly to wasm. -Because wasm is a binary format (and thus does not have lines and columns, or +may be applied directly to WebAssembly. +Because WebAssembly is a binary format (and thus does not have lines and columns, or comments per se), some simple reinterpretation of concepts is needed. #### Locations Source maps specify how to map a (zero-based) line and column position in generated code to a file, line, and column location in the source. For -wasm binary locations, the line number is always 1, and the column number -is interpreted as a byte offset into the wasm binary content (this results -in a more efficient encoding than using the line number field instead). +WebAssembly binary locations, the line number is always 1, and the column number +is interpreted as a byte offset into the WebAssembly binary content (this results +in a more efficient encoding than using the line number instead). Source locations are interpreted as in the source map spec. #### Linking generated code to source maps When the generated code is JavaScript, it includes a specially-formatted line -at the end, which is the URL of the associated source map. For wasm, a custom +at the end, which is the URL of the associated source map. For WebAssembly, a custom section named `"sourceMappingURL"` contains the URL. The URL is defined as in the WHATWG [URL spec](http://url.spec.whatwg.org), and is From 8207bae316a8ed535b0f780c13ac3d80f7842e50 Mon Sep 17 00:00:00 2001 From: Derek Schuff Date: Thu, 24 Aug 2017 10:14:46 -0700 Subject: [PATCH 7/9] Remove source maps from FutureFeatures.md --- FutureFeatures.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/FutureFeatures.md b/FutureFeatures.md index bb09b9eb..45952a02 100644 --- a/FutureFeatures.md +++ b/FutureFeatures.md @@ -192,16 +192,6 @@ instance. However, operators with 32-bit indices and operators with 64-bit indices will be given separate names to leave open the possibility of supporting both in the same instance in the future. -### Source maps integration - -* Add a new source maps [module section type](MVP.md#module-structure). -* Either embed the source maps directly or just a URL from which source maps can - be downloaded. -* Text source maps become intractably large for even moderate-sized compiled - codes, so probably need to define new binary format for source maps. -* Gestate ideas and start discussions at the - [Source Map RFC repository](https://github.com/source-map/source-map-rfc/issues) - ### Coroutines Coroutines will [eventually be part of C++][] and is already popular in other From ed75dfa3c1c5a33cac36ef5e6d67ea5aceba2b83 Mon Sep 17 00:00:00 2001 From: Derek Schuff Date: Thu, 24 Aug 2017 10:32:37 -0700 Subject: [PATCH 8/9] punctuation --- Web.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Web.md b/Web.md index ee4857c5..7a45f31d 100644 --- a/Web.md +++ b/Web.md @@ -219,8 +219,8 @@ When the generated code is JavaScript, it includes a specially-formatted line at the end, which is the URL of the associated source map. For WebAssembly, a custom section named `"sourceMappingURL"` contains the URL. The URL is defined as in the WHATWG -[URL spec](http://url.spec.whatwg.org), and is -resolved according to the source map spec, +[URL spec](https://url.spec.whatwg.org), and is +resolved according to the source map spec. For wasm modules with an associated HTTP response (e.g. those using the response-based compile or instantiation [APIs](#additional-web-embedding-api)) the URL may also be specified From d02d42fd40e3943bf0137927127ac8831952f15b Mon Sep 17 00:00:00 2001 From: Derek Schuff Date: Thu, 24 Aug 2017 11:29:24 -0700 Subject: [PATCH 9/9] clarify source mapping URL resolution --- Web.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/Web.md b/Web.md index aa44f775..2a110536 100644 --- a/Web.md +++ b/Web.md @@ -363,12 +363,18 @@ Source locations are interpreted as in the source map spec. When the generated code is JavaScript, it includes a specially-formatted line at the end, which is the URL of the associated source map. For WebAssembly, a custom section named `"sourceMappingURL"` contains the URL. -The URL is defined as in the WHATWG -[URL spec](https://url.spec.whatwg.org), and is -resolved according to the source map spec. +The URL is represented and encoded as defined in the WHATWG +[URL spec](https://url.spec.whatwg.org). +If the URL is not absolute, then: +* If the wasm module is associated with an HTTP response (e.g. the +compileStreaming or instantiateStreaming +[APIs](#additional-web-embedding-api) are used), then it is resolved relative to +the URL of the response. +* Otherwise it is relative to the page's origin. + For wasm modules with an associated HTTP response (e.g. those using -the response-based compile or instantiation -[APIs](#additional-web-embedding-api)) the URL may also be specified +the compileStreaming or instantiateStreaming +[APIs](#additional-web-embedding-api)) the source map URL may also be specified using the `SourceMap:` HTTP header as with JavaScript source maps.