Skip to content

Commit

Permalink
Deprecate detect and protect. Add git, dir, stdin (#1504)
Browse files Browse the repository at this point in the history
* init backwards compatible cmd

* update readme, todos

* ...

* moar comments, sprucing up root

* readme change
  • Loading branch information
zricethezav authored Sep 14, 2024
1 parent e93a7c0 commit 44ad62e
Show file tree
Hide file tree
Showing 7 changed files with 270 additions and 64 deletions.
94 changes: 45 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
Gitleaks is a SAST tool for **detecting** and **preventing** hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an **easy-to-use, all-in-one solution** for detecting secrets, past or present, in your code.

```
➜ ~/code(master) gitleaks detect --source . -v
➜ ~/code(master) gitleaks git -v
│╲
Expand Down Expand Up @@ -64,11 +64,11 @@ brew install gitleaks

# Docker (DockerHub)
docker pull zricethezav/gitleaks:latest
docker run -v ${path_to_host_folder_to_scan}:/path zricethezav/gitleaks:latest [COMMAND] --source="/path" [OPTIONS]
docker run -v ${path_to_host_folder_to_scan}:/path zricethezav/gitleaks:latest [COMMAND] [OPTIONS] [SOURCE_PATH]

# Docker (ghcr.io)
docker pull ghcr.io/gitleaks/gitleaks:latest
docker run -v ${path_to_host_folder_to_scan}:/path ghcr.io/gitleaks/gitleaks:latest [COMMAND] --source="/path" [OPTIONS]
docker run -v ${path_to_host_folder_to_scan}:/path ghcr.io/gitleaks/gitleaks:latest [COMMAND] [OPTIONS] [SOURCE_PATH]

# From Source (make sure `go` is installed)
git clone https://github.com/gitleaks/gitleaks.git
Expand Down Expand Up @@ -105,7 +105,7 @@ jobs:
```
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.16.1
rev: v8.19.0
hooks:
- id: gitleaks
```
Expand Down Expand Up @@ -137,77 +137,73 @@ Usage:
Available Commands:
completion generate the autocompletion script for the specified shell
detect detect secrets in code
dir scan directories or files for secrets
git scan git repositories for secrets
help Help about any command
protect protect secrets in code
stdin detect secrets from stdin
version display gitleaks version
Flags:
-b, --baseline-path string path to baseline with issues that can be ignored
-c, --config string config file path
order of precedence:
1. --config/-c
2. env var GITLEAKS_CONFIG
3. (--source/-s)/.gitleaks.toml
If none of the three options are used, then gitleaks will use the default config
--exit-code int exit code when leaks have been encountered (default 1)
-h, --help help for gitleaks
-l, --log-level string log level (trace, debug, info, warn, error, fatal) (default "info")
--max-target-megabytes int files larger than this will be skipped
--no-color turn off color for verbose output
--no-banner suppress banner
--redact redact secrets from logs and stdout
-f, --report-format string output format (json, csv, junit, sarif) (default "json")
-r, --report-path string report file
-s, --source string path to source (default ".")
-v, --verbose show verbose output from scan
-b, --baseline-path string path to baseline with issues that can be ignored
-c, --config string config file path
order of precedence:
1. --config/-c
2. env var GITLEAKS_CONFIG
3. (target path)/.gitleaks.toml
If none of the three options are used, then gitleaks will use the default config
--enable-rule strings only enable specific rules by id
--exit-code int exit code when leaks have been encountered (default 1)
-i, --gitleaks-ignore-path string path to .gitleaksignore file or folder containing one (default ".")
-h, --help help for gitleaks
--ignore-gitleaks-allow ignore gitleaks:allow comments
-l, --log-level string log level (trace, debug, info, warn, error, fatal) (default "info")
--max-target-megabytes int files larger than this will be skipped
--no-banner suppress banner
--no-color turn off color for verbose output
--redact uint[=100] redact secrets from logs and stdout. To redact only parts of the secret just apply a percent value from 0..100. For example --redact=20 (default 100%)
-f, --report-format string output format (json, csv, junit, sarif) (default "json")
-r, --report-path string report file
-v, --verbose show verbose output from scan
--version version for gitleaks
Use "gitleaks [command] --help" for more information about a command.
```

### Commands

There are two commands you will use to detect secrets; `detect` and `protect`.
⚠️ v8.19.0 introduced a change that deprecated `detect` and `protect`. Those commands are still available but
are hidden in the `--help` menu. Take a look at this [gist](https://gist.github.com/zricethezav/b325bb93ebf41b9c0b0507acf12810d2) for easy command translations.
If you find v8.19.0 broke an existing command (`detect`/`protect`), please open an issue.

#### Detect
There are three scanning modes: `git`, `dir`, and `stdin`.

The `detect` command is used to scan repos, directories, and files. This command can be used on developer machines and in CI environments.
#### Git
The `git` command lets you scan local git repos. Under the hood, gitleaks uses the `git log -p` command to scan patches.
You can configure the behavior of `git log -p` with the `log-opts` option.
For example, if you wanted to run gitleaks on a range of commits you could use the following
command: `gitleaks git -v --log-opts="--all commitA..commitB" path_to_repo`. See the [git log](https://git-scm.com/docs/git-log) documentation for more information.
If there is no target specified as a positional argument, then gitleaks will attempt to scan the current working directory as a git repo.

When running `detect` on a git repository, gitleaks will parse the output of a `git log -p` command (you can see how this executed
[here](https://github.com/zricethezav/gitleaks/blob/7240e16769b92d2a1b137c17d6bf9d55a8562899/git/git.go#L17-L25)).
[`git log -p` generates patches](https://git-scm.com/docs/git-log#_generating_patch_text_with_p) which gitleaks will use to detect secrets.
You can configure what commits `git log` will range over by using the `--log-opts` flag. `--log-opts` accepts any option for `git log -p`.
For example, if you wanted to run gitleaks on a range of commits you could use the following command: `gitleaks detect --source . --log-opts="--all commitA..commitB"`.
See the `git log` [documentation](https://git-scm.com/docs/git-log) for more information.
#### Dir
The `dir` (aliases include `files`, `directory`) command lets you scan directories and files. Example: `gitleaks dir -v path_to_directory_or_file`.
If there is no target specified as a positional argument, then gitleaks will scan the current working directory.

You can scan files and directories by using the `--no-git` option.

If you want to run only specific rules you can do so by using the `--enable-rule` option (with a rule ID as a parameter), this flag can be used multiple times. For example: `--enable-rule=atlassian-api-token` will only apply that rule. You can find a list of rules [here](config/gitleaks.toml).

#### Protect

The `protect` command is used to scan uncommitted changes in a git repo. This command should be used on developer machines in accordance with
[shifting left on security](https://cloud.google.com/architecture/devops/devops-tech-shifting-left-on-security).
When running `protect` on a git repository, gitleaks will parse the output of a `git diff` command (you can see how this executed
[here](https://github.com/zricethezav/gitleaks/blob/7240e16769b92d2a1b137c17d6bf9d55a8562899/git/git.go#L48-L49)). You can set the
`--staged` flag to check for changes in commits that have been `git add`ed. The `--staged` flag should be used when running Gitleaks
as a pre-commit.

**NOTE**: the `protect` command can only be used on git repos, running `protect` on files or directories will result in an error message.
#### Stdin
You can also stream data to gitleaks with the `stdin` command. Example: `cat some_file | gitleaks -v stdin`

### Creating a baseline

When scanning large repositories or repositories with a long history, it can be convenient to use a baseline. When using a baseline,
gitleaks will ignore any old findings that are present in the baseline. A baseline can be any gitleaks report. To create a gitleaks report, run gitleaks with the `--report-path` parameter.

```
gitleaks detect --report-path gitleaks-report.json # This will save the report in a file called gitleaks-report.json
gitleaks git --report-path gitleaks-report.json # This will save the report in a file called gitleaks-report.json
```

Once as baseline is created it can be applied when running the detect command again:

```
gitleaks detect --baseline-path gitleaks-report.json --report-path findings.json
gitleaks git --baseline-path gitleaks-report.json --report-path findings.json
```

After running the detect command with the --baseline-path parameter, report output (findings.json) will only contain new issues.
Expand Down
33 changes: 30 additions & 3 deletions cmd/detect.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
// The `detect` and `protect` command is now deprecated. Here are some equivalent commands
// to help guide you.

// OLD CMD: gitleaks detect --source={repo}
// NEW CMD: gitleaks git {repo}

// OLD CMD: gitleaks protect --source={repo}
// NEW CMD: gitleaks git --pre-commit {repo}

// OLD CMD: gitleaks protect --staged --source={repo}
// NEW CMD: gitleaks git --pre-commit --staged {repo}

// OLD CMD: gitleaks detect --no-git --source={repo}
// NEW CMD: gitleaks directory {directory/file}

// OLD CMD: gitleaks detect --no-git --pipe
// NEW CMD: gitleaks stdin

package cmd

import (
Expand All @@ -15,12 +33,16 @@ func init() {
rootCmd.AddCommand(detectCmd)
detectCmd.Flags().Bool("no-git", false, "treat git repo as a regular directory and scan those files, --log-opts has no effect on the scan when --no-git is set")
detectCmd.Flags().Bool("pipe", false, "scan input from stdin, ex: `cat some_file | gitleaks detect --pipe`")
detectCmd.Flags().Bool("follow-symlinks", false, "scan files that are symlinks to other files")
detectCmd.Flags().StringP("source", "s", ".", "path to source")
detectCmd.Flags().String("log-opts", "", "git log options")
}

var detectCmd = &cobra.Command{
Use: "detect",
Short: "detect secrets in code",
Run: runDetect,
Use: "detect",
Short: "detect secrets in code",
Run: runDetect,
Hidden: true,
}

func runDetect(cmd *cobra.Command, args []string) {
Expand Down Expand Up @@ -101,5 +123,10 @@ func runDetect(cmd *cobra.Command, args []string) {
}
}

// set follow symlinks flag
if detector.FollowSymlinks, err = cmd.Flags().GetBool("follow-symlinks"); err != nil {
log.Fatal().Err(err).Msg("")
}

findingSummaryAndExit(findings, cmd, cfg, exitCode, start, err)
}
64 changes: 64 additions & 0 deletions cmd/directory.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
package cmd

import (
"time"

"github.com/rs/zerolog/log"
"github.com/spf13/cobra"

"github.com/zricethezav/gitleaks/v8/report"
"github.com/zricethezav/gitleaks/v8/sources"
)

func init() {
rootCmd.AddCommand(directoryCmd)
directoryCmd.Flags().Bool("follow-symlinks", false, "scan files that are symlinks to other files")
}

var directoryCmd = &cobra.Command{
Use: "dir [flags] [path]",
Aliases: []string{"file", "directory"},
Short: "scan directories or files for secrets",
Run: runDirectory,
}

func runDirectory(cmd *cobra.Command, args []string) {
initConfig()
var (
findings []report.Finding
err error
)

// setup config (aka, the thing that defines rules)
cfg := Config(cmd)

// start timer
start := time.Now()

// grab source
source, err := cmd.Flags().GetString("source")
if err != nil {
log.Fatal().Err(err).Msg("could not get source")
}
detector := Detector(cmd, cfg, source)

// set exit code
exitCode, err := cmd.Flags().GetInt("exit-code")
if err != nil {
log.Fatal().Err(err).Msg("could not get exit code")
}

var paths <-chan sources.ScanTarget
paths, err = sources.DirectoryTargets(source, detector.Sema, detector.FollowSymlinks)
if err != nil {
log.Fatal().Err(err)
}

findings, err = detector.DetectFiles(paths)
if err != nil {
// don't exit on error, just log it
log.Error().Err(err).Msg("failed scan directory")
}

findingSummaryAndExit(findings, cmd, cfg, exitCode, start, err)
}
72 changes: 72 additions & 0 deletions cmd/git.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
package cmd

import (
"time"

"github.com/rs/zerolog/log"
"github.com/spf13/cobra"

"github.com/zricethezav/gitleaks/v8/report"
"github.com/zricethezav/gitleaks/v8/sources"
)

func init() {
rootCmd.AddCommand(gitCmd)
gitCmd.Flags().Bool("staged", false, "scan staged commits (good for pre-commit)")
gitCmd.Flags().Bool("pre-commit", false, "scan using git diff")
gitCmd.Flags().String("log-opts", "", "git log options")
}

var gitCmd = &cobra.Command{
Use: "git [flags] [repo]",
Short: "scan git repositories for secrets",
Args: cobra.MaximumNArgs(1),
Run: runGit,
}

func runGit(cmd *cobra.Command, args []string) {
initConfig()
var (
findings []report.Finding
err error
)

// setup config (aka, the thing that defines rules)
cfg := Config(cmd)

// start timer
start := time.Now()

// grab source
source, err := cmd.Flags().GetString("source")
if err != nil {
log.Fatal().Err(err).Msg("could not get source")
}
detector := Detector(cmd, cfg, source)

// set exit code
exitCode, err := cmd.Flags().GetInt("exit-code")
if err != nil {
log.Fatal().Err(err).Msg("could not get exit code")
}

var (
gitCmd *sources.GitCmd
logOpts string
)
logOpts, err = cmd.Flags().GetString("log-opts")
if err != nil {
log.Fatal().Err(err).Msg("could not call GetString() for log-opts")
}
gitCmd, err = sources.NewGitLogCmd(source, logOpts)
if err != nil {
log.Fatal().Err(err).Msg("could not create Git cmd")
}
findings, err = detector.DetectGit(gitCmd)
if err != nil {
// don't exit on error, just log it
log.Error().Err(err).Msg("failed to scan Git repository")
}

findingSummaryAndExit(findings, cmd, cfg, exitCode, start, err)
}
9 changes: 6 additions & 3 deletions cmd/protect.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,16 @@ import (

func init() {
protectCmd.Flags().Bool("staged", false, "detect secrets in a --staged state")
protectCmd.Flags().String("log-opts", "", "git log options")
protectCmd.Flags().StringP("source", "s", ".", "path to source")
rootCmd.AddCommand(protectCmd)
}

var protectCmd = &cobra.Command{
Use: "protect",
Short: "protect secrets in code",
Run: runProtect,
Use: "protect",
Short: "protect secrets in code",
Run: runProtect,
Hidden: true,
}

func runProtect(cmd *cobra.Command, args []string) {
Expand Down
Loading

0 comments on commit 44ad62e

Please sign in to comment.