Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollup of 17 pull requests #49051

Merged
merged 47 commits into from
Mar 16, 2018
Merged
Changes from 2 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
2bc3f5b
borrowck-asm: enable on mips
jcowgill Jan 23, 2018
1376bf8
tools/compiletest: add mips64 to ARCH_LIST
jcowgill Jan 25, 2018
6f55819
test: remove useless ignore-mips*el headers
jcowgill Jan 25, 2018
dcc438d
test: ignore mips64 in abi-main-signature-16bit-c-int.rs
jcowgill Jan 25, 2018
59199eb
test: remove duplicate ignore-aarch64 from stack-probes test
jcowgill Mar 9, 2018
bceb94e
test: ignore stack probe tests on mips*
jcowgill Mar 9, 2018
e0863c5
test: ignore mips* in x86_mmx test
jcowgill Mar 9, 2018
54467ae
test: ignore asm tests on mips* which won't work
jcowgill Mar 9, 2018
fb806fd
test: fix repr-transparent-aggregates test on mips64
jcowgill Mar 9, 2018
ff227c4
rustbuild: Remove ThinLTO-related configuration
alexcrichton Mar 9, 2018
a67e072
Dedupe rand
Eijebong Mar 12, 2018
74f5dd0
rustc: Start a custom cabi module for wasm32
alexcrichton Mar 12, 2018
0511077
rustc: Add `sha` to the x86 feature whitelist
alexcrichton Mar 12, 2018
0ca9a4c
Move librustdoc readme to rustc guide
mark-i-m Mar 13, 2018
a48224b
Add a section for using assertions into doc tests.
Songbird0 Mar 12, 2018
decbb83
Add test for issue 48941
ExpHP Mar 13, 2018
dc96467
Fix ICE on malformed plugin attributes
ExpHP Mar 13, 2018
12ac032
Implement import renaming with `_` (RFC 2166)
petrochenkov Mar 10, 2018
b4b7ccb
Add crate name to "main function not found" error message.
ehuss Mar 3, 2018
19b5113
rustc_trans: fix small aggregate returns for big-endian mips64 FFI
Mar 14, 2018
6aa4dcb
Add empty main() to tests where it is missing.
ehuss Mar 12, 2018
b08e6d3
Add suggestion where to add main function.
ehuss Mar 12, 2018
2f1b34c
Add backticks to `main` not found errors.
ehuss Mar 13, 2018
16d424f
Some tweaks to "type parameters from outer function" diagnostic
estebank Mar 14, 2018
5257275
Update E0601 test for new message format.
ehuss Mar 14, 2018
b6ec75f
Remove unnecessary "`" in error message E0307 (invalid self type).
kennytm Mar 15, 2018
6fbdaf4
unstabilize FusedIterator for Flatten since Flatten is unstable
Centril Mar 15, 2018
9e64946
setting ABORTING_MALLOC for asmjs backend
Mar 14, 2018
a45b79d
Rollup merge of #48706 - ehuss:main-not-found-in-crate, r=estebank
kennytm Mar 15, 2018
da88827
Rollup merge of #48875 - jcowgill:mips-test-fixes, r=sanxiyn
kennytm Mar 15, 2018
68a602e
Rollup merge of #48892 - alexcrichton:thinlto-again, r=Mark-Simulacrum
kennytm Mar 15, 2018
e1d19df
Rollup merge of #48922 - petrochenkov:asunder, r=nikomatsakis
kennytm Mar 15, 2018
35b008c
Rollup merge of #48957 - Eijebong:rand, r=michaelwoerister
kennytm Mar 15, 2018
292c6ca
Rollup merge of #48959 - alexcrichton:signext, r=eddyb
kennytm Mar 15, 2018
97b489e
Rollup merge of #48961 - Songbird0:rustdoc_doctests_assertions, r=Gui…
kennytm Mar 15, 2018
bf270bb
Rollup merge of #48965 - alexcrichton:add-sha-feature, r=eddyb
kennytm Mar 15, 2018
5e991e1
Improve `AddrParseError` documentation.
Songbird0 Mar 8, 2018
4d5cd21
Coherence diagnostic tweaks
estebank Mar 15, 2018
5a7aa6c
Rollup merge of #49007 - estebank:follow-up-47574, r=oli-obk
kennytm Mar 15, 2018
77cf02d
Rollup merge of #49024 - draganmladjenovic:mips64_cabi_sret, r=sanxiyn
kennytm Mar 15, 2018
55a0075
Rollup merge of #49042 - kennytm:fix-e0307-typo, r=rkruppe
kennytm Mar 15, 2018
90230ce
Rollup merge of #49050 - snf:fix_oom_asmjs, r=alexcrichton
kennytm Mar 15, 2018
6fbd033
Rollup merge of #48853 - Songbird0:addrparseerror_documentation_impro…
kennytm Mar 15, 2018
64490ff
Rollup merge of #48990 - ExpHP:dont-drop-the-bomb, r=estebank
kennytm Mar 15, 2018
a199fb2
Rollup merge of #49037 - estebank:coherence-tweaks, r=nikomatsakis
kennytm Mar 15, 2018
fb49ae0
Rollup merge of #49049 - Centril:fix/flatten-fusediterator, r=bluss
kennytm Mar 15, 2018
db2f0ae
Rollup merge of #48972 - mark-i-m:rustdoc_readme, r=QuietMisdreavus
kennytm Mar 15, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
173 changes: 2 additions & 171 deletions src/librustdoc/README.md
Original file line number Diff line number Diff line change
@@ -1,172 +1,3 @@
# The walking tour of rustdoc
For more information about how `librustdoc` works, see the [rustc guide].

Rustdoc is implemented entirely within the crate `librustdoc`. After partially compiling a crate to
get its AST (technically the HIR map) from rustc, librustdoc performs two major steps past that to
render a set of documentation:

* "Clean" the AST into a form that's more suited to creating documentation (and slightly more
resistant to churn in the compiler).
* Use this cleaned AST to render a crate's documentation, one page at a time.

Naturally, there's more than just this, and those descriptions simplify out lots of details, but
that's the high-level overview.

(Side note: this is a library crate! The `rustdoc` binary is crated using the project in
`src/tools/rustdoc`. Note that literally all that does is call the `main()` that's in this crate's
`lib.rs`, though.)

## Cheat sheet

* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on
other projects.
* Add `src/libtest` to be able to use `rustdoc --test`.
* If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then
after the previous build command, `cargo +local doc` will Just Work.
* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs.
* The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to
be used as though you would copy out the `doc` folder to a web server, since that's where the
CSS/JS and landing page are.
* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It's in a bunch of
`fmt::Display` implementations and supplementary functions.
* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom
`Clean` trait used to process them out of the rustc HIR.
* The bits specific to using rustdoc as a test harness are in `test.rs`.
* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting
doctests from a given block of Markdown.
* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they're handled by the test
runner of rustbuild and the supplementary script `src/etc/htmldocck.py`.
* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript
files that encode queries on the standard library search index and expected results.

## From crate to clean

In `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter
is where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The
former is a state container used when crawling through a crate to gather its documentation.

The main process of crate crawling is done in `clean/mod.rs` through several implementations of the
`Clean` trait defined within. This is a conversion trait, which defines one method:

```rust
pub trait Clean<T> {
fn clean(&self, cx: &DocContext) -> T;
}
```

`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation
pages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from
rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated
items may have some extra processing in its `Clean` implementation, but for the most part these
impls are straightforward conversions. The "entry point" to this module is the `impl Clean<Crate>
for visit_ast::RustdocVisitor`, which is called by `run_core` above.

You see, I actually lied a little earlier: There's another AST transformation that happens before
the events in `clean/mod.rs`. In `visit_ast.rs` is the type `RustdocVisitor`, which *actually*
crawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This
pass is mainly to get a few intermediate wrappers around the HIR types and to process visibility
and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are
processed, as well as the logic for whether a `pub use` should get the full page or a "Reexport"
line in the module page.

The other major thing that happens in `clean/mod.rs` is the collection of doc comments and
`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets
hand-written documentation. This makes it easier to collect this documentation later in the process.

The primary output of this process is a clean::Crate with a tree of Items which describe the
publicly-documentable items in the target crate.

### Hot potato

Before moving on to the next major step, a few important "passes" occur over the documentation.
These do things like combine the separate "attributes" into a single string and strip leading
whitespace to make the document easier on the markdown parser, or drop items that are not public or
deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one
file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping
private/hidden items can be bypassed by passing `--document-private-items` to rustdoc.

(Strictly speaking, you can fine-tune the passes run and even add your own, but [we're trying to
deprecate that][44136]. If you need finer-grain control over these passes, please let us know!)

[44136]: https://github.com/rust-lang/rust/issues/44136

## From clean to crate

This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/`
folder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up
the `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static
files which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript
that live in `html/static/`), creating the search index, and printing out the source code rendering,
before beginning the process of rendering all the documentation for the crate.

Several functions implemented directly on `Context` take the `clean::Crate` and set up some state
between rendering items or recursing on a module's child items. From here the "page rendering"
begins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML
from the items and documentation occurs within a series of `std::fmt::Display` implementations and
functions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes
out the page body is the `impl<'a> fmt::Display for Item<'a>` in `html/render.rs`, which switches
out to one of several `item_*` functions based on the kind of `Item` being rendered.

Depending on what kind of rendering code you're looking for, you'll probably find it either in
`html/render.rs` for major items like "what sections should I print for a struct page" or
`html/format.rs` for smaller component pieces like "how should I print a where clause as part of
some other item".

Whenever rustdoc comes across an item that should print hand-written documentation alongside, it
calls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a
series of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It
takes special care to enable certain features like footnotes and tables and add syntax highlighting
to Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There's also a
function in here (`find_testable_code`) that specifically scans for Rust code blocks so the
test-runner code can find all the doctests in the crate.

### From soup to nuts

(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video])

[video]: https://www.youtube.com/watch?v=hOLAGYmUQV0

It's important to note that the AST cleaning can ask the compiler for information (crucially,
`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within
`run_core` is passed outside the compiler context before being handed to `html::render::run`. This
means that a lot of the "supplementary data" that isn't immediately available inside an item's
definition, like which trait is the `Deref` trait used by the language, needs to be collected during
cleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering.
This manifests as a bunch of shared state, context variables, and `RefCell`s.

Also of note is that some items that come from "asking the compiler" don't go directly into the
`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait
implementations and generate new `Item`s for the impls based on that information. This goes directly
into the returned `Crate` rather than roundabout through the `DocContext`. This way, these
implementations can be collected alongside the others, right before rendering the HTML.

## Other tricks up its sleeve

All this describes the process for generating HTML documentation from a Rust crate, but there are
couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or
it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight
to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output
HTML.

For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in
`test.rs`, but instead of going through the full clean and render process, it runs a much simpler
crate walk to grab *just* the hand-written documentation. Combined with the aforementioned
"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before
handing them off to the libtest test runner. One notable location in `test.rs` is the function
`make_test`, which is where hand-written doctests get transformed into something that can be
executed.

## Dotting i's and crossing t's

So that's rustdoc's code in a nutshell, but there's more things in the repo that deal with it. Since
we have the full `compiletest` suite at hand, there's a set of tests in `src/test/rustdoc` that make
sure the final HTML is what we expect in various situations. These tests also use a supplementary
script, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation
to get a precise look at the output. The full description of all the commands available to rustdoc
tests is in `htmldocck.py`.

In addition, there are separate tests for the search index and rustdoc's ability to query it. The
files in `src/test/rustdoc-js` each contain a different search query and the expected results,
broken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the
Node.js runtime. These tests don't have as thorough of a writeup, but a broad example that features
results in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY`
with a set of `EXPECTED` results, complete with the full item path of each item.
[rustc guide]: https://rust-lang-nursery.github.io/rustc-guide/rustdoc.html