diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 1b8e90f15..987a1fefb 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -6,6 +6,7 @@ repos: hooks: - id: check-added-large-files + args: ['--maxkb=1024'] - id: check-case-conflict - id: check-json - id: check-symlinks diff --git a/README.md b/README.md index 868ea048a..1720c2d74 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # Nosey Parker: Find secrets in textual data +## Overview + Nosey Parker is a command-line tool that finds secrets and sensitive information in textual data. It is useful both for offensive and defensive security testing. **Key features:** @@ -30,7 +32,9 @@ Prebuilt binaries are available for x86_64 Linux and x86_64/aarch64 macOS on the This is a simple way to get started and will give good performance. -### Docker image +### Docker images + +
A prebuilt multiplatform Docker image is available for the latest release for x86_64 and aarch64: ```shell @@ -49,6 +53,8 @@ $ docker pull ghcr.io/praetorian-inc/noseyparker-alpine:edge **Note:** The Docker images run noticeably slower than a native binary, particularly on macOS. +
+ ### Arch Linux package @@ -96,6 +102,7 @@ The scanning and reporting steps are implemented as separate commands because yo ### Getting help + Running the `noseyparker` binary without arguments prints top-level help and exits. You can get abbreviated help for a particular command by running `noseyparker COMMAND -h`. More detailed help is available with the `help` command or long-form `--help` option. @@ -103,6 +110,8 @@ More detailed help is available with the `help` command or long-form `--help` op The prebuilt releases also include manpages that collect the command-line help in one place. These manpages converted into Markdown format are also included in the repository [here](docs/v0.17.0/man/man1). +If you have a question that's not answered by this documentation, feel free to [start a discussion](https://github.com/praetorian-inc/noseyparker/discussions/new/choose). + ### Terminology and data model @@ -143,7 +152,7 @@ This is Nosey Parker's top-level unit of reporting. ## Usage examples -### Docker note +### NOTE: When using Docker... If you are using the Docker image, replace `noseyparker` in the following commands with a Docker invocation that uses a mounted volume: @@ -154,7 +163,10 @@ docker run -v "$PWD":/scan ghcr.io/praetorian-inc/noseyparker:latest The Docker container runs with `/scan` as its working directory, so mounting `$PWD` at `/scan` in the container will make tab completion and relative paths in your command-line invocation work. -### Scan filesystem content for secrets +### Scan inputs for secrets + +#### Filesystem content, including local Git repos +![Screenshot showing Nosey Parker's workflow for scanning the filesystem for secrets](docs/usage-examples/gifs/02-scan-git-history.gif)
@@ -181,7 +193,8 @@ Run the `report` command next to show finding details. ```
-### Scan Git history by URL, GitHub username, or GitHub organization name + +#### Git repos given URL, GitHub username, or GitHub organization name
@@ -204,29 +217,8 @@ Providing an access token gives a higher API rate limit and may make additional See `noseyparker help scan` for more details.
-### Summarize findings - -
- -Nosey Parker prints out a summary of its findings when it finishes scanning. -You can also run this step separately: -``` -$ noseyparker summarize --datastore np.cpython - - Rule Distinct Groups Total Matches -─────────────────────────────────────────────────────────── - PEM-Encoded Private Key 1,076 1,192 - Generic Secret 331 478 - netrc Credentials 42 3,201 - Generic API Key 2 31 - md5crypt Hash 1 2 -``` - -Additional output formats are supported, including JSON and JSON lines, via the `--format=FORMAT` option. -
- -### Report finding details +### Report findings
@@ -277,9 +269,38 @@ Showing 3/29 occurrences: ``` (Note: the findings above are synthetic, invalid secrets.) +Additional output formats are supported, including JSON, JSON lines, and SARIF, via the `--format=FORMAT` option. +
+ +#### Human-readable text format +![Screenshot showing Nosey Parker's workflow for rendering its findings in human-readable format](docs/usage-examples/gifs/03-report-human.gif) + +#### JSON format +![Screenshot showing Nosey Parker's workflow for rendering its findings in JSON format](docs/usage-examples/gifs/04-report-json.gif) + + +### Summarize findings + +
+ +Nosey Parker prints out a summary of its findings when it finishes scanning. +You can also run this step separately: +``` +$ noseyparker summarize --datastore np.cpython + + Rule Distinct Groups Total Matches +─────────────────────────────────────────────────────────── + PEM-Encoded Private Key 1,076 1,192 + Generic Secret 331 478 + netrc Credentials 42 3,201 + Generic API Key 2 31 + md5crypt Hash 1 2 +``` + Additional output formats are supported, including JSON and JSON lines, via the `--format=FORMAT` option.
+ ### Enumerate repositories from GitHub
diff --git a/docs/usage-examples/.gitignore b/docs/usage-examples/.gitignore new file mode 100644 index 000000000..e8ee0106c --- /dev/null +++ b/docs/usage-examples/.gitignore @@ -0,0 +1 @@ +scratch/ diff --git a/docs/usage-examples/README.md b/docs/usage-examples/README.md new file mode 100644 index 000000000..165cb0c70 --- /dev/null +++ b/docs/usage-examples/README.md @@ -0,0 +1,26 @@ +# Usage Examples + +## Overview +This directory contains scripts to produce GIFs of Nosey Parker usage examples. + +These scripts are written using [`vhs`](https://github.com/charmbracelet/vhs), a console script runner and recording tool. +The example script fragments are in the [`examples`](examples) directory. +Each one is a `vhs` fragment, which is cobbled together with some common settings using the [`record-examples.zsh`](record-examples.zsh) script. + +The generated GIFs are stored in the [`gifs`](gifs) directory. + +## Generating GIFs +- Install `vhs` +- Install `noseyparker` +- `./record-examples.zsh` + +## Errata: `vhs` +This tool, though it seems to be very popular, is full of significant bugs. +What I've seen in a day's usage: + +- The `Show` and `Hide` commands seem to have a race condition, not reliably working without adding `Sleep 1s` calls before and after. + Otherwise, you are likely to get the hidden commands included in the rendered output. + +- The `Source` command is unsuitable for its main use case (putting common settings in a separate file); the settings seem to be applied, then partially forgotten. + +- The framerate for GIF output is unreliable and doesn't correspond with `Sleep` durations, especially at higher framerates diff --git a/docs/usage-examples/common-config.tape b/docs/usage-examples/common-config.tape new file mode 100644 index 000000000..d4c9f085c --- /dev/null +++ b/docs/usage-examples/common-config.tape @@ -0,0 +1,23 @@ +Require echo +Require less +Require noseyparker + +# Set WindowBar Colorful +# Set WindowBarSize 0 + +Set Margin 0 + +Set Padding 10 +Set FontSize 20 +Set LineHeight 1.1 +Set Width 1200 +Set Height 750 + +Set Shell "bash" +Set TypingSpeed 25ms + +# Use a lower framerate to get more predictable GIF animation speed. +# See https://github.com/charmbracelet/vhs/issues/88. +Set Framerate 20 + +Output "output.gif" diff --git a/docs/usage-examples/examples/01-getting-help.tape b/docs/usage-examples/examples/01-getting-help.tape new file mode 100644 index 000000000..344b70bc4 --- /dev/null +++ b/docs/usage-examples/examples/01-getting-help.tape @@ -0,0 +1,16 @@ +Hide Type "clear" Enter Show Sleep 1s +Type "# Run without arguments or with `-h` for help" Sleep 500ms Enter +Type "noseyparker" Sleep 1s Enter +Sleep 5s + + +Hide Type "clear" Enter Show Sleep 1s +Type "# Get more detailed help with `--help`" Sleep 500ms Enter +Type "noseyparker --help | less" Sleep 1s Enter +Sleep 5s Type "q" + + +Hide Type "clear" Enter Show Sleep 1s +Type "# Get detailed subcommand help with `CMD --help`" Sleep 500ms Enter +Type "noseyparker scan --help | less" Sleep 1s Enter +Sleep 5s Type "q" diff --git a/docs/usage-examples/examples/02-scan-git-history.tape b/docs/usage-examples/examples/02-scan-git-history.tape new file mode 100644 index 000000000..879468859 --- /dev/null +++ b/docs/usage-examples/examples/02-scan-git-history.tape @@ -0,0 +1,13 @@ +Hide +Sleep 1s +Type "git clone -q 'https://github.com/Plazmaz/leaky-repo' && clear" Enter +Sleep 1s +Show + +Sleep 1s + +Type "# Scan the complete history of a local Git repository with the `scan` command." Sleep 0.5s Enter +Type "# This example assumes you have a local clone of https://github.com/Plazmaz/leaky-repo." Sleep 1s Enter + +Type "noseyparker scan leaky-repo" Sleep 0.5s Enter +Sleep 5s diff --git a/docs/usage-examples/examples/03-report-human.tape b/docs/usage-examples/examples/03-report-human.tape new file mode 100644 index 000000000..85038df5e --- /dev/null +++ b/docs/usage-examples/examples/03-report-human.tape @@ -0,0 +1,13 @@ +Hide +Sleep 1s +Type "noseyparker -q scan --git-url 'https://github.com/Plazmaz/leaky-repo' && clear" Enter +Sleep 1s +Show + +Sleep 1s + +Type "# View findings in human-readable format with the `report` command." Sleep 0.5s Enter +Type "# This example assumes you already have a datastore from scanning at `datastore.np`." Sleep 1s Enter + +Type "noseyparker report --color always | less -RS" Sleep 0.5s Enter +Sleep 5s Type "q" diff --git a/docs/usage-examples/examples/04-report-json.tape b/docs/usage-examples/examples/04-report-json.tape new file mode 100644 index 000000000..74475b8c2 --- /dev/null +++ b/docs/usage-examples/examples/04-report-json.tape @@ -0,0 +1,18 @@ +Hide +Sleep 1s +Type "noseyparker -q scan --git-url 'https://github.com/Plazmaz/leaky-repo' && clear" Enter +Sleep 1s +Show + +Sleep 1s + +Type "# View findings in JSON format with the `report --format json` command." Sleep 0.5s Enter +Type "# You can redirect the output to a file using the `-o FILE` option." Sleep 0.5s Enter +Type "# This example assumes you already have a datastore from scanning at `datastore.np`." Sleep 1s Enter + +Type "noseyparker report --format json -o findings.json" Sleep 0.5s Enter +Sleep 1s + +Type "# You can pretty-print JSON using `jq`." Enter +Type "jq -C . findings.json | less -RS" Enter +Sleep 5 Type "q" diff --git a/docs/usage-examples/gifs/01-getting-help.gif b/docs/usage-examples/gifs/01-getting-help.gif new file mode 100644 index 000000000..3b462fbf0 Binary files /dev/null and b/docs/usage-examples/gifs/01-getting-help.gif differ diff --git a/docs/usage-examples/gifs/02-scan-git-history.gif b/docs/usage-examples/gifs/02-scan-git-history.gif new file mode 100644 index 000000000..c29e46dee Binary files /dev/null and b/docs/usage-examples/gifs/02-scan-git-history.gif differ diff --git a/docs/usage-examples/gifs/03-report-human.gif b/docs/usage-examples/gifs/03-report-human.gif new file mode 100644 index 000000000..ae9cf76f8 Binary files /dev/null and b/docs/usage-examples/gifs/03-report-human.gif differ diff --git a/docs/usage-examples/gifs/04-report-json.gif b/docs/usage-examples/gifs/04-report-json.gif new file mode 100644 index 000000000..75d0ba19e Binary files /dev/null and b/docs/usage-examples/gifs/04-report-json.gif differ diff --git a/docs/usage-examples/record-examples.zsh b/docs/usage-examples/record-examples.zsh new file mode 100755 index 000000000..26ab40059 --- /dev/null +++ b/docs/usage-examples/record-examples.zsh @@ -0,0 +1,24 @@ +#!/bin/zsh + +set -e + +rm -rf scratch gifs && mkdir scratch gifs + +for example in examples/*.tape; do + echo "### $example" + scratch_dir="scratch/$(basename "$example" .tape)" + mkdir "$scratch_dir" + + preprocessed="$scratch_dir/input.tape" + + cat common-config.tape "$example" >"$preprocessed" + + (cd "$scratch_dir" && vhs input.tape && cp output.gif "../../gifs/$(basename "$example" .tape).gif") +done + +# other examples to create: +# - listing rules +# - checking rules +# - reporting in SARIF format +# - reporting in JSONL format +# - dumping the report JSON schema