Document ruff_dev and format_dev (#5648)

## Summary

Document all `ruff_dev` subcommands and document the `format_dev` flags
in the formatter readme.

CC @zanieb please flag everything that isn't clear or missing

## Test Plan

n/a

CONTRIBUTING.md

@@ -550,3 +550,134 @@ cargo instruments -t time --bench linter --profile release-debug -p ruff_benchma
 - You may want to pass an additional filter to run a single test file
 
 Otherwise, follow the instructions from the linux section.
+
+## `cargo dev`
+
+`cargo dev` is a shortcut for `cargo run --package ruff_dev --bin ruff_dev`. You can run some useful
+utils with it:
+
+- `cargo dev print-ast <file>`: Print the AST of a python file using the
+  [RustPython parser](https://github.com/astral-sh/RustPython-Parser/tree/main/parser) that is
+  mainly used in Ruff. For `if True: pass # comment`, you can see the syntax tree, the byte offsets
+  for start and stop of each node and also how the `:` token, the comment and whitespace are not
+  represented anymore:
+
+```text
+[
+    If(
+        StmtIf {
+            range: 0..13,
+            test: Constant(
+                ExprConstant {
+                    range: 3..7,
+                    value: Bool(
+                        true,
+                    ),
+                    kind: None,
+                },
+            ),
+            body: [
+                Pass(
+                    StmtPass {
+                        range: 9..13,
+                    },
+                ),
+            ],
+            orelse: [],
+        },
+    ),
+]
+```
+
+- `cargo dev print-tokens <file>`: Print the tokens that the AST is built upon. Again for
+  `if True: pass # comment`:
+
+```text
+0 If 2
+3 True 7
+7 Colon 8
+9 Pass 13
+14 Comment(
+    "# comment",
+) 23
+23 Newline 24
+```
+
+- `cargo dev print-cst <file>`: Print the CST of a python file using
+  [LibCST](https://github.com/Instagram/LibCST), which is used in addition to the RustPython parser
+  in Ruff. E.g. for `if True: pass # comment` everything including the whitespace is represented:
+
+```text
+Module {
+    body: [
+        Compound(
+            If(
+                If {
+                    test: Name(
+                        Name {
+                            value: "True",
+                            lpar: [],
+                            rpar: [],
+                        },
+                    ),
+                    body: SimpleStatementSuite(
+                        SimpleStatementSuite {
+                            body: [
+                                Pass(
+                                    Pass {
+                                        semicolon: None,
+                                    },
+                                ),
+                            ],
+                            leading_whitespace: SimpleWhitespace(
+                                " ",
+                            ),
+                            trailing_whitespace: TrailingWhitespace {
+                                whitespace: SimpleWhitespace(
+                                    " ",
+                                ),
+                                comment: Some(
+                                    Comment(
+                                        "# comment",
+                                    ),
+                                ),
+                                newline: Newline(
+                                    None,
+                                    Real,
+                                ),
+                            },
+                        },
+                    ),
+                    orelse: None,
+                    leading_lines: [],
+                    whitespace_before_test: SimpleWhitespace(
+                        " ",
+                    ),
+                    whitespace_after_test: SimpleWhitespace(
+                        "",
+                    ),
+                    is_elif: false,
+                },
+            ),
+        ),
+    ],
+    header: [],
+    footer: [],
+    default_indent: "    ",
+    default_newline: "\n",
+    has_trailing_newline: true,
+    encoding: "utf-8",
+}
+```
+
+- `cargo dev generate-all`: Update `ruff.schema.json`, `docs/configuration.md` and `docs/rules`.
+  You can also set `RUFF_UPDATE_SCHEMA=1` to update `ruff.schema.json` during `cargo test`.
+- `cargo dev generate-cli-help`, `cargo dev generate-docs` and `cargo dev generate-json-schema`:
+  Update just `docs/configuration.md`, `docs/rules` and `ruff.schema.json` respectively.
+- `cargo dev generate-options`: Generate a markdown-compatible table of all `pyproject.toml`
+  options. Used for <https://beta.ruff.rs/docs/settings/>
+- `cargo dev generate-rules-table`: Generate a markdown-compatible table of all rules. Used for <https://beta.ruff.rs/docs/rules/>
+- `cargo dev round-trip <python file or jupyter notebook>`: Read a Python file or Jupyter Notebook,
+  parse it, serialize the parsed representation and write it back. Used to check how good our
+  representation is so that fixes don't rewrite irrelevant parts of a file.
+- `cargo dev format_dev`: See ruff_python_formatter README.md

crates/ruff_dev/src/generate_all.rs

@@ -21,7 +21,7 @@ pub(crate) enum Mode {
     /// Don't write to the file, check if the file is up-to-date and error if not.
     Check,
 
-    /// Write the generated help to stdout (rather than to `docs/configuration.md`).
+    /// Write the generated help to stdout.
     DryRun,
 }
 

crates/ruff_dev/src/generate_options.rs

@@ -1,4 +1,6 @@
-//! Generate a Markdown-compatible listing of configuration options.
+//! Generate a Markdown-compatible listing of configuration options for `pyproject.toml`.
+//!
+//! Used for <https://beta.ruff.rs/docs/settings/>.
 use itertools::Itertools;
 
 use ruff::settings::options::Options;

crates/ruff_dev/src/generate_rules_table.rs

@@ -1,4 +1,6 @@
 //! Generate a Markdown-compatible table of supported lint rules.
+//!
+//! Used for <https://beta.ruff.rs/docs/rules/>.
 
 use itertools::Itertools;
 use strum::IntoEnumIterator;

crates/ruff_python_formatter/README.md

@@ -276,6 +276,14 @@ python scripts/check_ecosystem.py --checkouts target/checkouts --projects github
 cargo run --bin ruff_dev -- format-dev --stability-check --multi-project target/checkouts
 ```
 
+Compared to `ruff check`, `cargo run --bin ruff_dev -- format-dev` has 4 additional options:
+
+- `--write`: Format the files and write them back to disk
+- `--stability-check`: Format twice (but don't write to disk) and check for differences and crashes
+- `--multi-project`: Treat every subdirectory as a separate project. Useful for ecosystem checks.
+- `--error-file`: Use together with `--multi-project`, this writes all errors (but not status
+  messages) to a file.
+
 ## The orphan rules and trait structure
 
 For the formatter, we would like to implement `Format` from the rust_formatter crate for all AST