SecureStore for Rust

This repository houses the ssclient command-line utility for creating and manipulating SecureStore secrets files as well as the securestore rust crate providing an API for retrieving and decrypting secrets from a SecureStore at runtime.

Usage

This rust implementation of the SecureStore protocol ships in two separate, complementary parts: a command line client (ssclient) and a rust library/crate (securestore). Both (mostly) expose the same functionality, but are intended to be used together for maximum productivity and ergonomics.

The typical workflow for deploying versioned secrets alongside a restricted-access binary (e.g. a web app, a kiosk, or similar where the application isn't distributed directly to end users and is running in what is considered to be a privileged environment) is demonstrated here.

Adding a SecureStore vault to your rust workspace

Start by installing a copy of ssclient, the interactive SecureStore cli. Pre-built binaries are available (see releases), but the majority of rust developers will find the most convenient option for distribution to be direct installation via cargo:

> cargo install ssclient
    Updating crates.io index
  Installing ssclient v0.100.0
   Compiling securestore v0.100.0
   Compiling ssclient v0.100.0
    Finished release [optimized] target(s) in 36.14s
  Installing /home/mqudsi/.cargo/bin/ssclient
   Installed package `ssclient v0.100.0` (executable `ssclient`)

ssclient will be installed and added to the cargo binary directory in your $PATH, making it available in a terminal session merely by invoking ssclient.

Let's imagine your worktree is a typical rust binary application with a layout matching a typical Cargo-based rust project:

> tree
.
├── .git/
├── Cargo.toml
└── src
    └─ main.rs

We'll be storing our secrets in a folder called secure in the root of our cargo workspace. Open a terminal, cd into your rust project, and execute the following to create a new secure directory and use ssclient to create a new secrets vault called secrets.json (which is the SecureStore protocol default name):

> cd my-rust-project
> mkdir secure
> cd secure
> ssclient create secrets.json --export-key secrets.key
Password: ************
Confirm password: ************
Saving newly generated key to secrets.key
Excluding key file in newly-created VCS ignore file .gitignore

ssclient will prompt you to enter (then confirm) a password before it creates a new secrets.json file (currently a skeleton SecureStore vault without any secrets). The vault is symmetrically encrypted but can be decrypted either with the password you just entered or with the key file we just exported (secrets.key). When updating or interacting with the SecureStore vault via ssclient at the command line, we'll use password-based encryption/decryption, but when we deploy the app in dev or production, we'll be using key-based encryption/decryption instead.

The layout of our project folder now looks like this:

> tree .
.
├── .git/
├── Cargo.toml
├── secure
│   ├── .gitignore
│   ├── secrets.json
│   └── secrets.key
└── src
    └── main.rs

As you can see, there are three new files under the secrets subdirectory: secrets.json (the human-readable SecureStore vault), secrets.key (the sensitive & secure master key that lets us bypass password protection to decrypt secrets in production without manual intervention), and a .gitignore file that was helpfully created by ssclient and prevents secrets.key from being accidentally committed to the git repo.

The SecureStore protocol specifies that secrets.json should be added to the git repository and versioned alongside the rest of the code that uses its secrets, and absolutely forbids secrets.key from being committed to a VCS.

Here's what the "empty" secrets.json file looks like right now, before we've added any secrets:

{
  "version": 3,
  "iv": "eebJviP8bIC6XF6fp1g4Cw==",
  "sentinel": {
    "iv": "dVEOSg1OaM6LLF2fB7W7jg==",
    "hmac": "mY1LP5gBDQaYFenC2oHHRb7LXSg=",
    "payload": "7mFgSJSYclBBPo+Xbel0DA5y8e24QKqUh7m8EXy5+8bSagUoHGoIi2sJSKlSDP4X"
  },
  "secrets": {}
}

This is the basic "skeleton" of a SecureStore (v3) vault, containing just enough info to let ssclient or the securestore crate verify that we're using the correct encryption/decryption key the next time we open the vault.

The generated secure/.gitignore file contains the following (one) rule:

secrets.key

All it does is stop secure/secrets.key from being accidentally added to the git repo.

Go ahead and add the secure directory to git, then move on to the next section to see how we'll add secrets and then access them at run-time from our rust application:

git add secure/
git commit -m "Add empty SecureStore vault"
git push

Adding secrets to your SecureStore vault

At this point, the vault has been created and we're ready to add one or more secrets to the vault. The secrets will be stored encrypted in secrets.json and versioned in git alongside the same code using them, making it easy to roll back to earlier versions and make sure that managed secrets are kept in lock-step with the code that depends on them, across merges and PRs.

Starting from the terminal where we left off, let's add a secret or two (say, the credentials for accessing PostgreSQL and the AWS IAM credentials to upload or sign S3 urls). We can either specify the secret value at the command line (ssclient set secret_name secret_value) or enter it at a prompt provided by ssclient for more security (e.g. to avoid the secret being stored in our bash history) or for convenience (e.g. entering special characters that would otherwise need to be escaped under bash) by only specifying the secret name (ssclient set secret_name). We can omit the store name/path because we're using the default (secrets.json) and we don't need to specify -p or --password because ssclient defaults to password-based decryption.

Let's cd into the secure directory and set the first secret:

> cd secure/
> ssclient set db:username
Password: ************
Value: pgadmin

To demonstrate how the SecureStore protocol allows interchangeable encryption/decryption with either a password or a key file, we'll set the next secret value using the key file we generated earlier (secrets.key) and we'll notice that the secret is added without ssclient prompting us to enter a password (but it's still encrypted, of course):

> ssclient -k secrets.key set db:password pgsql123
Value: pgsql123

Let's go ahead and set two more secrets, representing the AWS S3 IAM username and password:

> ssclient -k secrets.key set aws:s3:access_id AKIAIOSFODNN7
> ssclient -k secrets.key set aws:s3:access_key wJalrXUtnFEMI/K7MDENG/bPxRfiCY

The secrets vault (secrets.json) now contains four secrets (two not-actually-secret usernames and two actually-secret passwords). The only file that has changed in our git worktree is the secrets vault:

> git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
    (use "git restore <file>..." to discard changes in working directory)
            modified:   secrets.json

Let's see what our secrets.json file looks like now:

{
  "version": 3,
  "iv": "eebJviP8bIC6XF6fp1g4Cw==",
  "sentinel": {
    "iv": "dVEOSg1OaM6LLF2fB7W7jg==",
    "hmac": "mY1LP5gBDQaYFenC2oHHRb7LXSg=",
    "payload": "7mFgSJSYclBBPo+Xbel0DA5y8e24QKqUh7m8EXy5+8bSagUoHGoIi2sJSKlSDP4X"
  },
  "secrets": {
    "aws:s3:access_id": {
      "iv": "4YEmTY9rwW7pq7/Iv6Vncg==",
      "hmac": "tFO2cN/Fh/jO7ijbAXh98yrm2Nk=",
      "payload": "ejKs4D2anovxS1OyX8e4Eg=="
    },
    "aws:s3:access_key": {
      "iv": "czwulU+ejDXD0dryqA6aaA==",
      "hmac": "w/egLShBDwu9L/Wagk/EKQVzpK0=",
      "payload": "OfJM7ZDLOkGhD8OwjfOOQrHNgyeQqYPuDZKwARxN/nw="
    },
    "db:password": {
      "iv": "rN0a3GVuTkBctAbCO51VQA==",
      "hmac": "ko8YB33XxhCEIge8Rxdyab1EA4Y=",
      "payload": "uvq/NsjCfVkc6Upv8EWg8A=="
    },
    "db:username": {
      "iv": "CkWYqyIWOTquFco/NZGiKA==",
      "hmac": "ODcnI82hkzQ+9feyf/1WlV3yxt8=",
      "payload": "oEmn2rQE9Hva97XwdNYkoA=="
    }
  }
}

We can see there are four new named JSON objects under secrets, one for each secret. They're encrypted and encoded per the cross-platform SecureStore protocol in a way that's human-readable (plain-text, compact base64 representation of binary payloads, well-formatted), git-friendly (sorted by name, formatted with new lines between each secret and delimiting each secrets component, etc), and cross-platform compatible (standardized encryption and base64 encoding).

If you're following along in your own terminal, you'll hopefully notice that while the basic structure of your secrets.json file is the same as this one, the secret values themselves differ - this isn't just because we picked different passwords, though! This is part of what makes a SecureStore vault so secure, and prevents attackers from guessing your password or the encrypted secret contents if they get their hands on your secrets.json (which is safe enough to publish to GitHub or even the front page of The New York Times without worry) -- as long as your secrets.key stays secure, of course!

Go ahead and commit your updated secrets to git:

> git add secrets.json
> git commit -m "Add secrets for db and s3 access"

Retrieving secrets at runtime

At this point, we're going to set ssclient aside and focus on the rust side of things to see how secrets can be retrieved at runtime. First, let's add a dependency on the securestore crate to our Cargo.toml. We'll also add a dependency on once_cell to use securestore::SecretsManager as a singleton for demonstration purposes:

[package]
name = "sstemp"
version = "0.1.0"
edition = "2021"

[dependencies]
once_cell = "1.13.0"
securestore = "0.100.0"

After which we can open src/main.rs and add some code to open the secrets file and decrypt + retrieve one or more secrets at runtime. The ssclient documentation covers how the securestore crate and its primary SecretsManager type are used, but we'll demo the basics below.

At the top of main.rs, let's add some imports and a once_cell::sync::Lazy instance to hold our SecretsManager, since it's highly recommended to only have one SecretsManager instance servicing all secrets requests for the lifetime of the application:

use securestore::{KeySource, SecretsManager};
use std::path::Path;
use once_cell::sync::Lazy;

static SECRETS: Lazy<SecretsManager> = Lazy::new(|| {
    let keyfile = Path::new("secure/secrets.key");
    SecretsManager::load("secure/secrets.json", KeySource::File(keyfile))
        .expect("Failed to load SecureStore vault!")
})

As for accessing the secrets, let's add a helper function get_db_credentials() that will look up the secrets named db:username and db:password that we previously saved to the vault and return them as a (String, String) tuple:

fn get_db_credentials() -> Result<(String, String), securestore::Error> {
    let username = SECRETS.get("db:username")?;
    let password = SECRETS.get("db:password")?;
    Ok((username, password))
}

and verify that the application builds without any issues:

> cargo build
   Compiling sstemp v0.1.0 (/tmp/sstemp)
    Finished dev [unoptimized + debuginfo] target(s) in 0.74s

And that's all there is to it! We can demonstrate that it works by adding a test asserting the contents of the secrets (of course this would be a no-no in the real world!):

#[test]
fn verify_db_credentials() -> Result<(), securestore::Error> {
    let expected = ("pgadmin".to_owned(), "pgsql123".to_owned());
    let actual = get_db_credentials()?;

    assert_eq!(expected, actual);
    return Ok(());
}

Then run the test:

> cargo test
    Finished test [unoptimized + debuginfo] target(s) in 0.06s
         Running unittests src/main.rs (tar et/debug/deps/sstemp-b3b06ea1ff58c9da)

         running 1 test
         test verify_db_credentials ... ok

We just need to make sure that when we deploy our code (via whatever method we normally deploy our app into production), we also send a copy of the ./secure folder and store it alongside the executable (or in the CWD we execute our app under) -- this includes both the versioned and safe-to-share-with-the-world secure/secrets.json as well as the highly secret secure/secrets.key that mustn't ever be leaked to a non-secure environment.

Advanced secrets management topics

You'll notice that in the guide above, we stored two secrets (db:password and aws:s3:access_key) and two not-so-secret values (db:username and aws:s3:access_id) that could have been instead hard-coded into the application. What gives?

The answer lies in how SecureStore should be used in the real world, with teams of n > 1 and where not everyone may have the commit bit. Ideally, your dev and production infrastructure should be completely separate, as should your secrets. The SecureStore protocol makes this easy because it's really just a glorified (but secure!) key-value database, versioned and stored alongside your code itself. In practice, you won't just have secrets.key and secrets.json - you'll actually have a vault/key pair for each of your dev/staging/production environments, with all the vaults saved under the secure/ folder and committed to the repository, but with the corresponding secrets.key file for each environment's vault only shared with the people that should have access to it. Perhaps everyone has access to secrets.dev.key and can add/retrieve/remove secrets from secrets.dev.json but the production secrets.prod.key is only stored on the CI servers as a protected asset deployed to production but protected from being retrieved by any team members except for those with direct access to the secure CI environment or remote access to the production servers (your actual scenario will likely differ, but the basic idea stands).

The generic KV-nature of the SecureStore protocol and the SecureStore crate make it easy to abstract over the different dev/staging/prod configurations by simply loading a different store at startup depending on the environment (identified, say, by an environment variable), combined with storing any configuration-dependent non-secret values (usernames, connection strings, host addresses, etc) as if they were secrets in the same SecureStore vault, all automatically resolved just by determining and loading the correct store just once. For example, your actual SECRETS declaration could look like this:

static SECRETS: Lazy<SecretsManager> = Lazy::new(|| {
    let (store_path, key_path) = match std::env::var("MYWEBAPP_ENV").as_ref().map(|s| s.as_str()) {
        Ok("STAGING") => ("secure/secrets.staging.json", Path::new("secure/secrets.staging.key")),
        Ok("PRODUCTION") => ("secure/secrets.prod.json", Path::new("secure/secrets.prod.key")),
        _ => ("secure/secrets.dev.json", Path::new("secure/secrets.dev.key")),
    };

    SecretsManager::load(store_path, KeySource::File(key_path))
        .expect("Failed to load SecureStore vault!")
});

So while we could have hard-coded db:username, that would have meant also testing MYWEBAPP_ENV in get_db_credentials() to figure out what username to return in addition to the MYWEBAPP_ENV check in the SECRETS declaration that determined which secrets store to load. But if we store db:username as a secret the same way we store db:password, all that differentiation is done automatically for us. In addition, it makes sense to keep all the credentials info in one place (the secrets vault), wholly separate from the runtime logic - if you need to update the username and password in the future, you just need to update them in one place (the vault) rather than needing to patch both secrets.json and src/main.rs (in addition to remembering to do so).