(docs) Overview of the language-tools (#455)

.github/ISSUE_TEMPLATE/bug_report.md

@@ -35,3 +35,7 @@ If applicable, add screenshots to help explain your problem.
 
 **Additional context**
 Add any other context about the problem here.
+
+<!-- Want to help us out? Read this to get started:
+https://github.com/sveltejs/language-tools#development
+-->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -18,3 +18,7 @@ A clear and concise description of any alternative solutions or features you've
 
 **Additional context**
 Add any other context or screenshots about the feature request here.
+
+<!-- Want to help us out? Read this to get started:
+https://github.com/sveltejs/language-tools#development
+-->

.prettierignore

@@ -1,2 +1,3 @@
 packages/svelte2tsx/*.d.ts
-packages/svelte2tsx/test/**
+packages/svelte2tsx/test/**
+.github/**

README.md

@@ -75,6 +75,8 @@ Converts a .svelte file into a legal TypeScript file. Built from [halfnelson/sve
 
 Pull requests are encouraged and always welcome. [Pick an issue](https://github.com/sveltejs/language-tools/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc) and help us out!
 
+To get an overview of the internals, [read here](./docs/internal/overview.md).
+
 To install and work on these tools locally:
 
 > Make sure to uninstall the extension from the marketplace to not have it clash with the local one.

docs/README.md

@@ -100,3 +100,4 @@ Your default formatter for Svelte files may be wrong. Either it's set to the old
 ## Internals
 
 -   [Notes about deployment](./internal/deployment.md)
+-   [Overview of the language-tools and how things work together](./internal/overview.md)

docs/internal/overview.md

@@ -0,0 +1,109 @@
+# Overview of the language-tools and how things work together
+
+The `language-tools` repository is a monorepo containing several packages which are closely related to each other.
+
+-   `svelte2tsx` - transforms Svelte code into JSX/TSX code
+-   `language-server` - a language server adhering to the [LSP](https://microsoft.github.io/language-server-protocol)
+-   `svelte-check` - a command line tool to get diagnostics for Svelte code
+-   `svelte-vscode` - the [Svelte for VSCode](https://marketplace.visualstudio.com/items?itemName=svelte.svelte-vscode) extension
+
+This is how they are related:
+
+```
+svelte-vscode |
+              |-> language-server -> svelte2tsx
+svelte-check  |
+```
+
+## language-server overview
+
+As briefly touched already, this is a language-server adhering to the [Language Server Protocol (LSP)](https://microsoft.github.io/language-server-protocol).
+The protocol defines the communication between an editor or IDE and a language server that provides language features like auto complete, go to definition, find all references etc.
+
+Our `language-server` can roughly be splitted into [four areas](/packages/language-server/src/plugins):
+
+-   CSS: Provides intellisense for the things inside `<style>`. Internally it mostly forwards stuff to the `vscode-css-languageservice`.
+-   HTML: Provides intellisense for basic HTML tags like `div`, `a` etc. Internally it mostly forwards stuff to the `vscode-html-languageservice`. Svelte-specific template syntax is _NOT_ handled here.
+-   Svelte: Provides the diagnostics of the Svelte compiler. If you use preprocessors, those are invoked beforehand - that's why we need the `svelte.config.js` to know how to preprocess your files. It also does the formatting through `prettier-plugin-svelte` and other cross-cutting concerns like the "Extract Component" refactoring.
+-   TypeScript/JavaScript: Provides intellisense for all JS/TS related stuff. This not only includes code inside `<script>` but also inside the template the moment you use any of the Svelte specifics like bindings or template syntax. `svelte2tsx` is used in here, and only here.
+
+The last area, TS/JS, is where most of the work is done. Svelte is a mix of JavaScript/TypeScript inside `script` and more of it within Svelte's template syntax. To get a holistic view of the file, we need to account for both.
+This is also the reason why preprocessors such as `svelte-preprocess` cannot do type checking because it produces wrong diagnostics. To preprocess the script content, it only gets the content from the script. Think of this situation:
+
+```html
+<script lang="ts">
+    let a: number = 1;
+</script>
+{a}
+```
+
+`svelte-preprocess` only gets `let a = 1` and the diagnostics would output that `a` is never used, which is wrong.
+
+To get that holistic view, we need `svelte2tsx`.
+
+### svelte2tsx
+
+To get a holistic view of Svelte files, we have two options:
+
+1. Write a language service ourselves which is capable of doing things like auto complete, go to definition, rename etc.
+2. Convert the Svelte code to something an existing language service can process which then does auto complete, go to defintion, rename etc for us.
+
+We chose the second option because TypeScript provides a language service which can do all the heavy lifting for us: We give it some files and then invoke methods like `getQuickInfoAtPosition` for hover info. These files need to be in a format the language service can understand: A form of JavaScript or TypeScript. `svelte2tsx` is the package which does this transformation: Pass in Svelte code and get back JSX or TSX code, depending on whether or not you use TS in Svelte. `svelte2tsx` uses some [ambient definitions](/packages/svelte2tsx/svelte-shims.d.ts) which are loaded by the language server, to do some of the transformations. It also provides [JSX definitions](/packages/svelte2tsx/svelte-jsx.d.ts) which are recognized by the TypeScript compiler and define intrinsic elements, attributes and events - so if you ever get an error that a DOM attribute is not assignable to a DOM element, it's likely a missing declaration in there.
+
+The code generated by `svelte2tsx` is not runnable in any way and does not actually exist at runtime when you run your app, it purely exists for the intellisense. The code also returns source mappings so we know which position in the original code corresponds to which generated position.
+
+### Integration of svelte2tsx into the language-server
+
+This example shows how our `language-server` uses `svelte2tsx`:
+
+**Svelte file changes:**
+
+1. Svelte file comes in
+
+```html
+<script>
+    export let world = 'name';
+</script>
+
+<h1>hello {world}</h1>
+```
+
+2. Transform Svelte to TSX/JSX
+
+```tsx
+<></>;
+function render() {
+    // -- the transformed script:
+    let world = 'name';
+    // -- the transformed template
+    <>
+        <h1>hello {world}</h1>
+    </>;
+    // -- obtained props, slots and events,
+    //    to get intellisense for this component in other components
+    return { props: { world }, slots: {}, events: {} };
+}
+// -- generate a class
+export default class _World_ extends createSvelte2TsxComponent(__sveltets_partial(render)) {}
+```
+
+3. Pass that result to the TypeScript language service
+
+**User wants hover info:**
+
+1. User hovers over a component property in the template
+2. Get generated file from original file
+3. Transform the position (line, character) where the user hovers from the original to the generated position
+4. Invoke TypeScript language service's method for hover info with the generated position
+5. Hand back results to user
+
+As you can see, the `language-server` is mostly doing position transformations and then handing the main work off to the TypeScript language service. Sometimes we also need to do post processing of the results and work around quirks.
+
+## svelte-check
+
+As mentioned above, preprocessors cannot do type checking. That's why we added `svelte-check`, a command line tool to get diagnostics for Svelte code.
+The tool is using the `language-server` to do this work. Inside the `language-server` we added an [entry point](/packages/language-server/src/svelte-check.ts) for `svelte-check` which spins up the language server and just executes diagnostics.
+
+## svelte-vscode
+
+This is the [Svelte for VSCode](https://marketplace.visualstudio.com/items?itemName=svelte.svelte-vscode) extension most of you probably use. Syntax highlighting is provided through a [JSON file](/packages/svelte-vscode/syntaxes) which tells VSCode how to tokenize the input. To get all the intellisense capabilities, the extension uses the the `language-server`. VSCode implements the LSP and also provides a node library to do most of the wiring for us. Some things like special refactoring commands ("Extract Component") need some extra glue code, but that's all.