Update Docs Post-Rewrite (#112)

* Update Docs Post-Rewrite

* checkpoint! updating so that alembic works programmatically from within the fides codebase [ci skip]

* cleans up the migration commands [ci skip]

* the config will now use defaults instead of failing hard, finished the deployment rewrite

* updates the getting started guides [ci skip]

* updates the CLI page

* starts updating the tutorial [ci skip]

* fix some formatting issues

* fixes all of the ci issues

* clean up all of the comments from review

Co-authored-by: Thomas La Piana <tal103020@icloud.com>

Makefile

@@ -16,38 +16,30 @@ help:
 	@echo --------------------
 	@echo Development Targets:
 	@echo ----
-	@echo clean - Runs various Docker commands to clean up the docker environment including containers, images, volumes, etc.
+	@echo clean - Runs various Docker commands to clean up the docker environment including containers, images, volumes, etc. This will wipe out everything!
 	@echo ----
-	@echo cli - Spins up the database, the api, and starts a shell within a docker container with the local fidesctl files mounted.
+	@echo cli - Spins up the database, the api, and starts a shell within a Docker container with the local Fidesctl files mounted.
 	@echo ----
-	@echo build - Builds the fidesctl Docker image.
+	@echo build - Builds the Fidesctl Docker image.
 	@echo ----
-	@echo check-all - Run all of the available CI checks for fidesctl locally.
+	@echo check-all - Run all of the available CI checks for Fidesctl locally.
 	@echo ----
-	@echo init-db - Initializes the database docker container and runs migrations. Run this if your database seems to be the cause of test failures.
+	@echo init-db - Run any available migrations.
 	@echo ----
-	@echo api - Spins up the database and fidesapi, reachable from the host machine at localhost.
+	@echo api - Spins up the database and API, reachable from the host machine at localhost.
 	@echo ----
-	@echo cli - Spins up the database, fidesapi, and starts a shell within the fidesapi container to run fidesctl commands
+	@echo cli - Spins up the database, API, and starts a shell within the API container to run Fidesctl CLI commands.
 	@echo --------------------
 
 ####################
 # Dev
 ####################
 
-.PHONY: check-db
-check-db: compose-build
-	@echo "Check for new migrations to run..."
-	@docker-compose down
-	@docker-compose run $(IMAGE_NAME) alembic upgrade head
-	@make teardown
-
 .PHONY: init-db
 init-db: compose-build
-	@echo "Drop the db and run the migrations..."
+	@echo "Check for new migrations to run..."
 	@docker-compose down
-	@docker volume prune -f
-	@docker-compose run $(IMAGE_NAME) alembic upgrade head
+	@docker-compose run $(IMAGE_NAME) fidesctl init-db
 	@make teardown
 
 .PHONY: api

docker-compose.yml

@@ -5,7 +5,7 @@ services:
       dockerfile: Dockerfile
     command: uvicorn --host 0.0.0.0 --port 8080 --reload fidesapi.main:app
     depends_on:
-      fidesdb:
+      fidesctl-db:
         condition: service_healthy
     expose:
       - 8080
@@ -19,7 +19,7 @@ services:
     env_file:
       - env_files/fides.env
 
-  fidesdb:
+  fidesctl-db:
     image: postgres:12
     healthcheck:
       test: ["CMD-SHELL", "pg_isready -U postgres"]

docs/fides/docs/cli.md

@@ -1,13 +1,19 @@
-# Fidesctl
+# Fidesctl CLI
 
-Fidesctl wraps the functionality of the Fides Server into a CLI tool to be used by either engineers or within CI/CD pipelines.
+---
+
+The Fidesctl CLI wraps the entire functionality of Fidesctl into a few succint commands.
 
 ## Commands
 
+This is a non-exhaustive list of available Fidesctl CLI commands:
+
 * `fidesctl apply <manifest_dir> [--dry] [--diff]` - Creates or Updates resources found within the YAML file(s) at the specified path.
 * `fidesctl evaluate [-k,--fides-key] [-m, --message] [--dry]` - Runs an evaluation of all policies, but a single policy can be specified using the `--fides-key` parameter.
-* `fidesctl find <resource_type> <fides_key>` - Looks up a specific resource on the server by its type and `fides_key`.
-* `fidesctl ls <resource_type>` - Shows a list of resources of a certain type that exist on the server.
-* `fidesctl ping` - Pings the server to make sure that a connection can be established.
+* `fidesctl initdb` - Sets up the database by running all missing migrations.
+* `fidesctl get <resource_type> <fides_key>` - Looks up a specific resource on the server by its type and `fides_key`.
+* `fidesctl ls <resource_type>` - Shows a list of all resources of a certain type that exist on the server.
+* `fidesctl ping` - Pings the API's healthcheck endpoint to make sure that it is reachable and ready for requests.
+* `fidesctl resetdb` - Tears down the database, erasing all data.
 * `fidesctl version` - Shows the version of Fides that is installed.
-* `fidesctl view-config`- Show a JSON representation of the config that Fides is using.
+* `fidesctl view-config`- Show a JSON representation of the config that Fidesctl is using.

docs/fides/docs/deployment.md

@@ -1,42 +1,61 @@
 # Deployment
 
-We recommend deploying Fides with `Docker` using the included docker-compose file [found here in the repo](https://github.com/ethyca/fides/blob/main/docker-compose.yml). There are separate containers for `Fidesctl`, the `API` and the `DB`.
+For production deployments of Fidesctl, we suggest deploying everything individually as opposed to using the included docker-compose file. While docker-compose is great for development and experimentation, it isn't suited to production use. The following guide will walk you through setting up each component of Fidesctl as a production-grade deployment.
 
 ## Requirements
 
-1. Install MySQL 8
+1. Install Postgres 12
 1. Install Python 3.8 or newer (including pip)
-1. Install Docker
+1. Install Fidesctl via pip -> `pip install fidesctl`
 
 ## Setup
 
-The setup is done via Docker, with configuration values injected at runtime into the container.
+### Step 1: Database
 
-1. Spin up a MySQL DB with the following env vars:
+Spin up a Postgresql DB and configure a user, password and database for Fidesctl to use. For example:
 
-    ```env
-    MYSQL_ROOT_PASSWORD="<root_password>"
-    MYSQL_USER="<user>"
-    MYSQL_PASSWORD="<user_password>"
-    MYSQL_DATABASE="<db>"
-    ```
+```env
+POSTGRES_USER="fidesctl_db"
+POSTGRES_PASSWORD="f1d3sctl_dB"
+POSTGRES_DATABASE="fidesctl_db"
+```
 
-1. Start the FidesAPI -> `docker run -p "127.0.0.1:8080:8080/tcp" --env FIDES_DB_JDBC_URL="jdbc:mysql://<server_address>:3306/<db>" --env FIDES_DB_JDBC_USER="<user>" --env FIDES_DB_JDBC_PASSWORD="<user_password>" ethyca/fidesapi:latest /bin/bash -c "sbt flywayMigrate && sbt ~jetty:start"`
-1. Install Fidesctl -> `pip install fidesctl`
-1. Configuration of Fidesctl will be done via an `ini` file that will then be mounted onto the docker container. Fidesctl will automatically look for a file called `fides.ini` in the local directory, or at a location set by the `FIDES_CONFIG_PATH` environment variable. 
-    <details>
-        <summary>Here's an example ini file to get you started</summary>
-        
-    ```ini
-    [user]
-    user_id = 1
-    api_key = test_api_key
+### Step 2: Create a Config
 
-    [cli]
-    server_url = http://fidesapi:8080
-    ```
-    </details>
+The next step is to create a `fidesctl.toml` config file. This is used to pass important variables to the Fidesctl applications for connections to the database, api, etc. Make sure that the username, password and database in the `database_url` connection string match what you used to configure your database.
 
-    Write your `fides.ini` file and set your `FIDES_CONFIG_PATH` environment variable as needed
+Fidesctl will automatically look for the `fidesctl.toml` file in the current directory, in the user directory, or at the path specified by an optional `FIDESCTL_CONFIG_PATH` environment variable.
 
-5. Run `fidesctl` to see a list of possible commands!
+Additionally, any variable can be overriden by using a properly formatted environment variable. For instance to overwrite the `database_url` configuration value, you would set the `FIDESCTL__API__DATABASE_URL` environment variable.
+
+The following is an example `fidesctl.toml`:
+
+```toml
+[cli]
+server_url = "http://localhost:8080"
+
+[api]
+database_url = "postgresql+psycopg2://fidesctl_db:fidesdb@localhost:5432/fidesctl_db"
+```
+
+### Step 3: Fidesctl API
+
+Next we need to prepare the database to be used by the API. Run the initial database setup via the fidesctl CLI:
+
+```bash
+fidesctl init-db
+```
+
+Now open a new terminal to run the API, as it will run there indefinitely:
+
+`fidesctl webserver`
+
+The webserver should now be available at `localhost:8080`, and docs are available at `localhost:8080/docs`, however the API docs can be safely ignored, as the Fidesctl CLI will abstract the API layer.
+
+### Step 4: Fidesctl CLI
+
+The last step is to check that everything is working! Open a new terminal window and run the following:
+
+`fidesctl ping`
+
+If everything is configured correctly, it will let you know that the command was successful! You've now successfully completed a complete Fidesctl deployment.

docs/fides/docs/getting_started/docker.md

@@ -1,57 +1,65 @@
-# Getting Started with Fides in Docker
+# Getting Started with Fidesctl in Docker
 
-The recommended way to get Fides running is to launch it using the supplied `make` commands. The `make` commands wrap `docker-compose` commands that will spin up each piece of the project.
+---
+
+The recommended way to get Fidesctl running is to launch it using the supplied `make` commands. The `make` commands wrap `docker-compose` commands that will spin up each piece of the project.
 
 ## System Requirements
 
 1. Install `make` locally (see [Make on Homebrew (Mac)](https://formulae.brew.sh/formula/make), [Make for Windows](http://gnuwin32.sourceforge.net/packages/make.htm), or your preferred installation)
 1. Install `docker` locally (see [Docker Desktop](https://www.docker.com/products/docker-desktop) or your preferred installation; use a recent enough version so that `docker-compose` is also available)
-1. Clone the [Fides repo](https://github.com/ethyca/fides) and `cd` into the `fides` directory
+1. Clone the [Fidesctl repo](https://github.com/ethyca/fides) and `cd` into the `fides` directory
 
 ## Docker Setup
 
 The following commands should all be run from the top-level `fides` directory (where the Makefile is):
-1. `make init-db` -> Builds the require images, spins up the database, and runs the initialization scripts
-```
-~/git/fides% make init-db
-Build the images required in the docker-compose file...
-...
-Building fidesapi
-...
-Building fidesctl
-...
-Building docs
-...
-Reset the db and run the migrations...
-...
-Tearing down the dev environment...
-...
-Teardown complete
-```
+
+1. `make init-db` -> Builds the required images, spins up the database, and runs the initialization scripts:
+
+    ```bash
+    ~/git/fides% make init-db
+    Build the images required in the docker-compose file...
+    ...
+    Building fidesapi
+    ...
+    Building fidesctl
+    ...
+    Building docs
+    ...
+    Reset the db and run the migrations...
+    ...
+    Tearing down the dev environment...
+    ...
+    Teardown complete
+    ```
+
 2. `make cli` -> This will spin up the entire project and open a shell within the `fidesctl` container, with the `fidesapi` being accessible. This command will "hang" for a bit, as `fidesctl` will wait for the API to be healthy before launching the shell. Once you see the `fidesctl#` prompt, you know you're ready to go:
-```
-~/git/fides% make cli
-Build the images required in the docker-compose file...
-...
-Building fidesapi
-...
-Building fidesctl
-...
-Building docs
-...
-Check for new migrations to run...
-...
-root@1a742083cedf:/fides/fidesctl#
-```
+
+    ```bash
+    ~/git/fides% make cli
+    Build the images required in the docker-compose file...
+    ...
+    Building fidesapi
+    ...
+    Building fidesctl
+    ...
+    Building docs
+    ...
+    Check for new migrations to run...
+    ...
+    root@1a742083cedf:/fides/fidesctl#
+    ```
+
 3. `fidesctl ping` -> This confirms that your `fidesctl` CLI can reach the server and everything is ready to go!
-```
-root@796cfde906f1:/fides/fidesctl# fidesctl ping
-Pinging http://fidesapi:8080...
-Ping Successful!
-```
+
+    ```bash
+    root@796cfde906f1:/fides/fidesctl# fidesctl ping
+    Pinging http://fidesapi:8080...
+    Ping Successful!
+    ```
 
 ## Next Steps
 
-Now that you're up and running, you can use `fidesctl` from the shell to get a list of all the possible CLI commands. You're now ready to start enforcing privacy with Fides!
+Now that you're up and running, you can use `fidesctl` from the shell to get a list of all the possible CLI commands. You're now ready to start enforcing privacy with Fidesctl!
 
-See the [Tutorial](../tutorial.md) page for a step-by-step guide on setting up a Fides data privacy workflow.
+See the [Tutorial](../tutorial.md) page for a step-by-step guide on setting up a Fidesctl data privacy workflow.

docs/fides/docs/getting_started/local.md

@@ -1,27 +1,10 @@
-# Getting Started with Fides Locally (without Docker)
+# Getting Started with Fidesctl Locally (without Docker)
 
-Fides can also be spun up locally without relying on Docker or Make, however there are significantly more prerequisites. This is __not the recommended way to run Fides!__ Please see the [Getting Started with Docker](docker.md) guide for the recommended setup experience.
+---
 
-## Local Requirements
+Fidesctl can be spun up locally without relying on Docker or Make, however it is a bit more complicated. If you'd like something simpler, please see the [Getting Started with Docker](docker.md) guide for the recommended setup experience.
 
-1. Python 3.8
-1. pip Version 21 or later
-1. Java 11
-1. sbt Version 2.12
-1. MySQL DB Version 8
-1. Clone the [Fides repo](https://github.com/ethyca/fides)
-
-## Local Setup
-
-1. Spin up the MySQL database with your desired credentials
-1. `cd fidesapi/`
-1. Update the `application.conf` in `src/main/resources/` with the database credentials
-1. `sbt flywayMigrate`
-1. `sbt ~jetty:start` -> You now have a Fides Server instance up and running powered by MySQL!
-1. In a new shell -> `cd fidesctl/ && pip install -e .`
-1. Set the `FIDES_SERVER_URL` environment variable to `localhost:8080` and restart your shell as needed
-1. Set the `FIDES_CONFIG_PATH` environment variable and write a config based off of the `fidesctl/example_config.ini`, changing fields as needed.
-1. You can now run `fidesctl ping` to verify that your installation is set up properly
+The guide for getting Fidesctl up and running locally is the same as is described in our [Deployment Guide](../deployment.md), so please head over there and follow the steps to get going.
 
 ## Next Steps
 

docs/fides/docs/tutorial.md

@@ -1,22 +1,22 @@
 # Tutorial
 
-This tutorial walks you through the process of getting up and running with Fides.
+This tutorial walks you through the process of getting up and running with Fidesctl.
 
 ## Getting Started
 
-Use either the [Docker](getting_started/docker.md) (**recommended**) or [Local](getting_started/local.md) guide to get Fides up and running on your machine.
+Use either the [Docker](getting_started/docker.md) (**recommended**) or [Local](getting_started/local.md) guide to get Fidesctl up and running on your machine.
 
 ## Writing Manifest Files
 
-The next step is to write the manifest files that describe your privacy data usage with the Fides privacy ontology. Manifest files are written in YAML and are used to create and update resources via the FidesAPI.
+The next step is to write the manifest files that describe your privacy data usage with the Fides privacy ontology. Manifest files are written in YAML and are used to create and update resources via the Fidesctl API.
 
 First create a directory for the manifests to live in:
 
-`mkdir fides_manifests/`
+`mkdir fides_resources/`
 
 Next, you'll need to write a System manifest file and a Policy manifest file. These are the only two required resources for Fides to function. For an exhaustive set of example manifests see the [Fides Resources](fides_resources.md) page. Included below are the examples we'll assume are being used for the sake of the tutorial.
 
-=== "fides_manifests/policy.yml"
+=== "fides_resources/policy.yml"
 
     ```yaml
     policy:
@@ -75,7 +75,7 @@ Next, you'll need to write a System manifest file and a Policy manifest file. Th
             action: REJECT
     ```
 
-### "fides_manifests/dataset.yml"
+### "fides_resources/dataset.yml"
 
     ```yaml
     dataset:
@@ -103,7 +103,7 @@ Next, you'll need to write a System manifest file and a Policy manifest file. Th
             path: "sample_db_dataset.food_preference"
     ```
 
-=== "fides_manifests/system.yml"
+=== "fides_resources/system.yml"
 
     ```yaml
     system:
@@ -131,9 +131,9 @@ Once you've finished writing your manifest files, it's time to apply them to the
 
 If we assume the same directory name as before for where our manifests are located, the command would be:
 
-`fidesctl apply fides_manifests/`
+`fidesctl apply fides_resources/`
 
-This will load all files ending in either `.yaml` or `.yml` within that directory. Any file formatting issues within the manifests will be caught and shown to the user.
+This will load all files ending in either `.yaml` or `.yml` within that directory. Any invalid resource definitions within the manifests will be caught and shown to the user.
 
 ## Evaluation
 

env_files/db.env

@@ -1,3 +1,3 @@
-POSTGRES_USER="fidesdb"
-POSTGRES_PASSWORD="fidesdb"
-POSTGRES_DATABASE="fidesdb"
+POSTGRES_USER="fidesctl"
+POSTGRES_PASSWORD="fidesctl"
+POSTGRES_DATABASE="fidesctl"

env_files/fides.env

@@ -1 +1 @@
-FIDES_CONFIG_PATH="/fides/fidesctl/example_config.toml"
+FIDES_CONFIG_PATH="/fides/fidesctl/fides.toml"

fidesctl/MANIFEST.in

@@ -3,3 +3,4 @@ include requirements.txt
 include dev-requirements.txt
 include versioneer.py
 include src/fidesctl/_version.py
+include src/alembic.ini