Skip to content

Commit

Permalink
cli: replace clap with lexopt and supporting code
Browse files Browse the repository at this point in the history
ripgrep began it's life with docopt for argument parsing. Then it moved
to Clap and stayed there for a number of years. Clap has served ripgrep
well, and it probably could continue to serve ripgrep well, but I ended
up deciding to move off of it.

Why?

The first time I had the thought of moving off of Clap was during the
2->3->4 transition. I thought the 3.x and 4.x releases were great, but
for me, it ended up moving a little too quickly. Since the release of
4.x was telegraphed around when 3.x came out, I decided to just hold off
and wait to migrate to 4.x instead of doing a 3.x migration followed
shortly by another 4.x migration. Of course, I just never ended up doing
the migration at all. I never got around to it and there just wasn't a
compelling reason for me to upgrade. While I never investigated it, I
saw an upgrade as a non-trivial amount of work in part because I didn't
encapsulate the usage of Clap enough.

The above is just what got me started thinking about it. It wasn't
enough to get me to move off of it on its own. What ended up pushing me
over the edge was a combination of factors:

* As mentioned above, I didn't want to run on the migration treadmill.
This has proven to not be much of an issue, but at the time of the
2->3->4 releases, I didn't know how long Clap 4.x would be out before a
5.x would come out.
* The release of lexopt[1] caught my eye. IMO, that crate demonstrates
exactly how something new can arrive on the scene and just thoroughly
solve a problem minimalistically. It has the docs, the reasoning, the
simple API, the tests and good judgment. It gets all the weird corner
cases right that Clap also gets right (and is part of why I was
originally attracted to Clap).
* I have an overall desire to reduce the size of my dependency tree. In
part because a smaller dependency tree tends to correlate with better
compile times, but also in part because it reduces my reliance and trust
on others. It lets me be the "master" of ripgrep's destiny by reducing
the amount of behavior that is the result of someone else's decision
(whether good or bad).
* I perceived that Clap solves a more general problem than what I
actually need solved. Despite the vast number of flags that ripgrep has,
its requirements are actually pretty simple. We just need simple
switches and flags that support one value. No multi-value flags. No
sub-commands. And probably a lot of other functionality that Clap has
that makes it so flexible for so many different use cases. (I'm being
hand wavy on the last point.)

With all that said, perhaps most importantly, the future of ripgrep
possibly demands a more flexible CLI argument parser. In today's world,
I would really like, for example, flags like `--type` and `--type-not`
to be able to accumulate their repeated values into a single sequence
while respecting the order they appear on the CLI. For example, prior
to this migration, `rg regex-automata -Tlock -ttoml` would not return
results in `Cargo.lock` in this repository because the `-Tlock` always
took priority even though `-ttoml` appeared after it. But with this
migration, `-ttoml` now correctly overrides `-Tlock`. We would like to
do similar things for `-g/--glob` and `--iglob` and potentially even
now introduce a `-G/--glob-not` flag instead of requiring users to use
`!` to negate a glob. (Which I had done originally to work-around this
problem.) And some day, I'd like to add some kind of boolean matching to
ripgrep perhaps similar to how `git grep` does it. (Although I haven't
thought too carefully on a design yet.) In order to do that, I perceive
it would be difficult to implement correctly in Clap.

I believe that this last point is possible to implement correctly in
Clap 2.x, although it is awkward to do so. I have not looked closely
enough at the Clap 4.x API to know whether it's still possible there. In
any case, these were enough reasons to move off of Clap and own more of
the argument parsing process myself.

This did require a few things:

* I had to write my own logic for how arguments are combined into one
single state object. Of course, I wanted this. This was part of the
upside. But it's still code I didn't have to write for Clap.
* I had to write my own shell completion generator.
* I had to write my own `-h/--help` output generator.
* I also had to write my own man page generator. Well, I had to do this
with Clap 2.x too, although my understanding is that Clap 4.x supports
this. With that said, without having tried it, my guess is that I
probably wouldn't have liked the output it generated because I
ultimately had to write most of the roff by hand myself to get the man
page I wanted. (This also had the benefit of dropping the build
dependency on asciidoc/asciidoctor.)

While this is definitely a fair bit of extra work, it overall only cost
me a couple days. IMO, that's a good trade off given that this code is
unlikely to change again in any substantial way. And it should also
allow for more flexible semantics going forward.

Fixes #884, Fixes #1648, Fixes #1701, Fixes #1814, Fixes #1966

[1]: https://docs.rs/lexopt/0.3.0/lexopt/index.html
  • Loading branch information
BurntSushi committed Nov 21, 2023
1 parent c33f623 commit 082245d
Show file tree
Hide file tree
Showing 47 changed files with 12,724 additions and 6,141 deletions.
33 changes: 9 additions & 24 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
include:
- build: pinned
os: ubuntu-latest
rust: 1.72.1
rust: 1.74.0
- build: stable
os: ubuntu-latest
rust: stable
Expand Down Expand Up @@ -94,7 +94,7 @@ jobs:
rust: nightly-x86_64-gnu
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Install packages (Ubuntu)
if: matrix.os == 'ubuntu-latest'
Expand Down Expand Up @@ -131,6 +131,7 @@ jobs:
run: |
echo "cargo command is: ${{ env.CARGO }}"
echo "target flag is: ${{ env.TARGET_FLAGS }}"
echo "target dir is: ${{ env.TARGET_DIR }}"
- name: Build ripgrep and all crates
run: ${{ env.CARGO }} build --verbose --workspace ${{ env.TARGET_FLAGS }}
Expand Down Expand Up @@ -164,26 +165,6 @@ jobs:
if: matrix.target != ''
run: ${{ env.CARGO }} test --verbose --workspace ${{ env.TARGET_FLAGS }}

- name: Test for existence of build artifacts (Windows)
if: matrix.os == 'windows-2022'
shell: bash
run: |
outdir="$(ci/cargo-out-dir "${{ env.TARGET_DIR }}")"
ls "$outdir/_rg.ps1" && file "$outdir/_rg.ps1"
- name: Test for existence of build artifacts (Unix)
if: matrix.os != 'windows-2022'
shell: bash
run: |
outdir="$(ci/cargo-out-dir "${{ env.TARGET_DIR }}")"
# TODO: Check for the man page generation here. For whatever reason,
# it seems to be intermittently failing in CI. No idea why.
# for f in rg.bash rg.fish rg.1; do
for f in rg.bash rg.fish; do
# We could use file -E here, but it isn't supported on macOS.
ls "$outdir/$f" && file "$outdir/$f"
done
- name: Test zsh shell completions (Unix, sans cross)
# We could test this when using Cross, but we'd have to execute the
# 'rg' binary (done in test-complete) with qemu, which is a pain and
Expand All @@ -197,11 +178,15 @@ jobs:
shell: bash
run: ${{ env.CARGO }} test --manifest-path crates/cli/Cargo.toml ${{ env.TARGET_FLAGS }} --lib print_hostname -- --nocapture

- name: Print available short flags
shell: bash
run: ${{ env.CARGO }} test --bin rg ${{ env.TARGET_FLAGS }} flags::defs::tests::available_shorts -- --nocapture

rustfmt:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Install Rust
uses: dtolnay/rust-toolchain@master
with:
Expand All @@ -214,7 +199,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Install Rust
uses: dtolnay/rust-toolchain@master
with:
Expand Down
233 changes: 164 additions & 69 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -1,47 +1,39 @@
# The way this works is the following:
#
# The create-release job runs purely to initialize the GitHub release itself
# and to output upload_url for the following job.
#
# The build-release job runs only once create-release is finished. It gets the
# release upload URL from create-release job outputs, then builds the release
# executables for each supported platform and attaches them as release assets
# to the previously created release.
#
# The key here is that we create the release only once.
#
# Reference:
# https://eugene-babichenko.github.io/blog/2020/05/09/github-actions-cross-platform-auto-releases/

name: release

# Only do the release on x.y.z tags.
on:
push:
# Enable when testing release infrastructure on a branch.
# branches:
# - ag/work
tags:
- "[0-9]+.[0-9]+.[0-9]+"

# We need this to be able to create releases.
permissions:
contents: write

jobs:
# The create-release job runs purely to initialize the GitHub release itself,
# and names the release after the `x.y.z` tag that was pushed. It's separate
# from building the release so that we only create the release once.
create-release:
name: create-release
runs-on: ubuntu-latest
# env:
# Set to force version number, e.g., when no tag exists.
# RG_VERSION: TEST-0.0.0
outputs:
rg_version: ${{ env.RG_VERSION }}
# VERSION: TEST-0.0.0
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get the release version from the tag
shell: bash
if: env.RG_VERSION == ''
if: env.VERSION == ''
run: echo "VERSION=${{ github.ref_name }}" >> $GITHUB_ENV
- name: Show the version
run: |
echo "RG_VERSION=$GITHUB_REF_NAME" >> $GITHUB_ENV
echo "version is: ${{ env.RG_VERSION }}"
echo "version is: $VERSION"
- name: Create GitHub release
env:
GH_TOKEN: ${{ github.token }}
run: gh release create ${{ env.RG_VERSION }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: gh release create $VERSION --draft --verify-tag --title $VERSION
outputs:
version: ${{ env.VERSION }}

build-release:
name: build-release
Expand All @@ -52,25 +44,50 @@ jobs:
# systems.
CARGO: cargo
# When CARGO is set to CROSS, this is set to `--target matrix.target`.
TARGET_FLAGS: ""
TARGET_FLAGS:
# When CARGO is set to CROSS, TARGET_DIR includes matrix.target.
TARGET_DIR: ./target
# Bump this as appropriate. We pin to a version to make sure CI
# continues to work as cross releases in the past have broken things
# in subtle ways.
CROSS_VERSION: v0.2.5
# Emit backtraces on panics.
RUST_BACKTRACE: 1
# Build static releases with PCRE2.
PCRE2_SYS_STATIC: 1
strategy:
fail-fast: false
matrix:
build: [linux, linux-arm, macos, win-msvc, win-gnu, win32-msvc]
include:
- build: linux
os: ubuntu-latest
rust: nightly
target: x86_64-unknown-linux-musl
- build: linux-arm
strip: x86_64-linux-musl-strip
- build: stable-x86
os: ubuntu-latest
rust: nightly
target: arm-unknown-linux-gnueabihf
rust: stable
target: i686-unknown-linux-gnu
strip: x86_64-linux-gnu-strip
qemu: i386
- build: stable-aarch64
os: ubuntu-latest
rust: stable
target: aarch64-unknown-linux-gnu
strip: aarch64-linux-gnu-strip
qemu: qemu-aarch64
- build: stable-powerpc64
os: ubuntu-latest
rust: stable
target: powerpc64-unknown-linux-gnu
strip: powerpc64-linux-gnu-strip
qemu: qemu-ppc64
- build: stable-s390x
os: ubuntu-latest
rust: stable
target: s390x-unknown-linux-gnu
strip: s390x-linux-gnu-strip
qemu: qemu-s390x
- build: macos
os: macos-latest
rust: nightly
Expand All @@ -90,15 +107,17 @@ jobs:

steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Install packages (Ubuntu)
if: matrix.os == 'ubuntu-latest'
shell: bash
run: |
ci/ubuntu-install-packages
- name: Install packages (macOS)
if: matrix.os == 'macos-latest'
shell: bash
run: |
ci/macos-install-packages
Expand All @@ -109,64 +128,140 @@ jobs:
target: ${{ matrix.target }}

- name: Use Cross
if: matrix.os == 'ubuntu-latest' && matrix.target != ''
shell: bash
run: |
cargo install cross
# In the past, new releases of 'cross' have broken CI. So for now, we
# pin it. We also use their pre-compiled binary releases because cross
# has over 100 dependencies and takes a bit to compile.
dir="$RUNNER_TEMP/cross-download"
mkdir "$dir"
echo "$dir" >> $GITHUB_PATH
cd "$dir"
curl -LO "https://github.com/cross-rs/cross/releases/download/$CROSS_VERSION/cross-x86_64-unknown-linux-musl.tar.gz"
tar xf cross-x86_64-unknown-linux-musl.tar.gz
echo "CARGO=cross" >> $GITHUB_ENV
- name: Set target variables
shell: bash
run: |
echo "TARGET_FLAGS=--target ${{ matrix.target }}" >> $GITHUB_ENV
echo "TARGET_DIR=./target/${{ matrix.target }}" >> $GITHUB_ENV
- name: Show command used for Cargo
shell: bash
run: |
echo "cargo command is: ${{ env.CARGO }}"
echo "target flag is: ${{ env.TARGET_FLAGS }}"
echo "target dir is: ${{ env.TARGET_DIR }}"
- name: Build release binary
run: ${{ env.CARGO }} build --verbose --release --features pcre2 ${{ env.TARGET_FLAGS }}
shell: bash
run: |
${{ env.CARGO }} build --verbose --release --features pcre2 ${{ env.TARGET_FLAGS }}
if [ "${{ matrix.os }}" = "windows-latest" ]; then
bin="target/${{ matrix.target }}/release/rg.exe"
else
bin="target/${{ matrix.target }}/release/rg"
fi
echo "BIN=$bin" >> $GITHUB_ENV
- name: Strip release binary (linux, macos and macos-arm)
if: matrix.build == 'linux' || matrix.os == 'macos'
run: strip "target/${{ matrix.target }}/release/rg"
- name: Strip release binary (macos)
if: matrix.os == 'macos'
shell: bash
run: strip "$BIN"

- name: Strip release binary (arm)
if: matrix.build == 'linux-arm'
- name: Strip release binary (cross)
if: env.CARGO == 'cross'
shell: bash
run: |
docker run --rm -v \
"$PWD/target:/target:Z" \
rustembedded/cross:arm-unknown-linux-gnueabihf \
arm-linux-gnueabihf-strip \
/target/arm-unknown-linux-gnueabihf/release/rg
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.strip }}" \
"/target/${{ matrix.target }}/release/rg"
- name: Build archive
- name: Determine archive name
shell: bash
run: |
outdir="$(ci/cargo-out-dir "${{ env.TARGET_DIR }}")"
staging="ripgrep-${{ needs.create-release.outputs.rg_version }}-${{ matrix.target }}"
mkdir -p "$staging"/{complete,doc}
version="${{ needs.create-release.outputs.version }}"
echo "ARCHIVE=ripgrep-$version-${{ matrix.target }}" >> $GITHUB_ENV
cp {README.md,COPYING,UNLICENSE,LICENSE-MIT} "$staging/"
cp {CHANGELOG.md,FAQ.md,GUIDE.md} "$staging/doc/"
cp "$outdir"/{rg.bash,rg.fish,_rg.ps1} "$staging/complete/"
cp complete/_rg "$staging/complete/"
- name: Creating directory for archive
shell: bash
run: |
mkdir -p "$ARCHIVE"/{complete,doc}
cp "$BIN" "$ARCHIVE"/
cp {README.md,COPYING,UNLICENSE,LICENSE-MIT} "$ARCHIVE"/
cp {CHANGELOG.md,FAQ.md,GUIDE.md} "$ARCHIVE"/doc/
if [ "${{ matrix.os }}" = "windows-latest" ]; then
cp "target/${{ matrix.target }}/release/rg.exe" "$staging/"
7z a "$staging.zip" "$staging"
certutil -hashfile "$staging.zip" SHA256 > "$staging.zip.sha256"
echo "ASSET=$staging.zip" >> $GITHUB_ENV
echo "ASSET_SUM=$staging.zip.sha256" >> $GITHUB_ENV
else
# The man page is only generated on Unix systems. ¯\_(ツ)_/¯
cp "$outdir"/rg.1 "$staging/doc/"
cp "target/${{ matrix.target }}/release/rg" "$staging/"
tar czf "$staging.tar.gz" "$staging"
shasum -a 256 "$staging.tar.gz" > "$staging.tar.gz.sha256"
echo "ASSET=$staging.tar.gz" >> $GITHUB_ENV
echo "ASSET_SUM=$staging.tar.gz.sha256" >> $GITHUB_ENV
fi
- name: Generate man page and completions (no emulation)
if: matrix.qemu == ''
shell: bash
run: |
"$BIN" --version
"$BIN" --generate complete-bash > "$ARCHIVE/complete/rg.bash"
"$BIN" --generate complete-fish > "$ARCHIVE/complete/rg.fish"
"$BIN" --generate complete-powershell > "$ARCHIVE/complete/_rg.ps1"
"$BIN" --generate complete-zsh > "$ARCHIVE/complete/_rg"
"$BIN" --generate man > "$ARCHIVE/doc/rg.1"
- name: Generate man page and completions (emulation)
if: matrix.qemu != ''
shell: bash
run: |
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" --version
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate complete-bash > "$ARCHIVE/complete/rg.bash"
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate complete-fish > "$ARCHIVE/complete/rg.fish"
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate complete-powershell > "$ARCHIVE/complete/_rg.ps1"
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate complete-zsh > "$ARCHIVE/complete/_rg"
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate man > "$ARCHIVE/doc/rg.1"
- name: Build archive (Windows)
shell: bash
if: matrix.os == 'windows-latest'
run: |
7z a "$ARCHIVE.zip" "$ARCHIVE"
certutil -hashfile "$ARCHIVE.zip" SHA256 > "$ARCHIVE.zip.sha256"
echo "ASSET=$ARCHIVE.zip" >> $GITHUB_ENV
echo "ASSET_SUM=$ARCHIVE.zip.sha256" >> $GITHUB_ENV
- name: Build archive (Unix)
shell: bash
if: matrix.os != 'windows-latest'
run: |
tar czf "$ARCHIVE.tar.gz" "$ARCHIVE"
shasum -a 256 "$ARCHIVE.tar.gz" > "$ARCHIVE.tar.gz.sha256"
echo "ASSET=$ARCHIVE.tar.gz" >> $GITHUB_ENV
echo "ASSET_SUM=$ARCHIVE.tar.gz.sha256" >> $GITHUB_ENV
- name: Upload release archive
env:
GH_TOKEN: ${{ github.token }}
run: gh release upload ${{ needs.create-release.outputs.rg_version }} ${{ env.ASSET }} ${{ env.ASSET_SUM }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
shell: bash
run: |
version="${{ needs.create-release.outputs.version }}"
gh release upload "$version" ${{ env.ASSET }} ${{ env.ASSET_SUM }}
Loading

0 comments on commit 082245d

Please sign in to comment.