Reorganize application code

.github/workflows/fidesops_unsafe_pr_checks.yml

@@ -13,6 +13,7 @@ env:
 jobs:
   Build:
     runs-on: ubuntu-latest
+    if: contains(github.event.pull_request.labels.*.name, 'run unsafe ci checks')
     steps:
       - name: Checkout
         uses: actions/checkout@v3

CHANGELOG.md

@@ -39,6 +39,7 @@ The types of changes are:
 ### Changed
 
 * Users should be able to click on the full field of a dropdown-type filter to open up the dropdown [#747](https://github.com/ethyca/fidesops/pull/903)
+* Reorganize the core code logic to be easier to merge with fidesctl [#1058](https://github.com/ethyca/fidesops/pull/1058)
 * Updated the python docker base image from slim-buster to slim-bullseye [#928](https://github.com/ethyca/fidesops/pull/928)
 * Removed ipython from the docker install [#928](https://github.com/ethyca/fidesops/pull/928)
 * Run static nox checks outside of Docker [#1053](https://github.com/ethyca/fidesops/pull/1053)

Dockerfile

@@ -71,7 +71,7 @@ ENV PYTHONUNBUFFERED=TRUE
 ENV RUNNING_IN_DOCKER=true
 
 # Make a static files directory
-RUN mkdir -p /fidesops/src/fidesops/build/static/
+RUN mkdir -p /fidesops/src/fidesops/ops/build/static/
 
 EXPOSE 8080
 CMD [ "fidesops", "webserver" ]
@@ -93,4 +93,4 @@ RUN python setup.py sdist
 RUN pip install dist/fidesops-*.tar.gz
 
 # Copy frontend build over
-COPY --from=frontend /fidesops/clients/ops/admin-ui/out/ /fidesops/src/fidesops/build/static/
+COPY --from=frontend /fidesops/clients/ops/admin-ui/out/ /fidesops/src/fidesops/ops/build/static/

MANIFEST.in

@@ -3,11 +3,11 @@ include README.md
 include requirements.txt
 include dev-requirements.txt
 include versioneer.py
-include src/fidesops/alembic.ini
+include src/fidesops/ops/alembic.ini
 include src/fidesops/_version.py
 include src/fidesops/py.typed
-graft src/fidesops/migrations
-exclude src/fidesops/migrations/README
-exclude src/fidesops/migrations/script.py.mako
+graft src/fidesops/ops/migrations
+exclude src/fidesops/ops/migrations/README
+exclude src/fidesops/ops/migrations/script.py.mako
 
 global-exclude *.pyc

clients/ops/admin-ui/package.json

@@ -16,7 +16,7 @@
     "analyze:server": "cross-env BUNDLE_ANALYZE=server next build",
     "analyze:browser": "cross-env BUNDLE_ANALYZE=browser next build",
     "export": "next build && next export",
-    "copy-export": "rsync -a --delete out/ ../../../src/fidesops/build/static/",
+    "copy-export": "rsync -a --delete out/ ../../../src/fidesops/ops/build/static/",
     "prod-export": "npm run export && npm run copy-export"
   },
   "dependencies": {

docs/fidesops/docs/development/contributing_details.md

@@ -5,13 +5,14 @@
 ## API Endpoints
 
 ### Postman API Collection
+
 Our [fidesops Postman Collection](../postman/Fidesops.postman_collection.json) can be used to test fidesops endpoints.
 
 Follow our [Using Postman](../postman/using_postman.md) guide to learn more about the how to use the collection.
 
 ### API URLs
 
-We define API URLs for specific API versions as constants within `app.api.v1.urn_registry` (where `v1` can be substituted for that particular API version), then import those URLs into their specific API views. Since we are on the first version, there is no clear precedent set for overriding URLs between versions yet. The most likely change is that we'll override the `APIRouter` class instantiation with a different base path (ie. `/api/v2` instead of `/api/v1`). For example:
+We define API URLs for specific API versions as constants within `fidesops.ops.api.v1.urn_registry` (where `v1` can be substituted for that particular API version), then import those URLs into their specific API views. Since we are on the first version, there is no clear precedent set for overriding URLs between versions yet. The most likely change is that we'll override the `APIRouter` class instantiation with a different base path (ie. `/api/v2` instead of `/api/v1`). For example:
 
 ```
 PRIVACY_REQUEST = "/privacy-request"
@@ -20,18 +21,17 @@ PRIVACY_REQUEST_DETAIL = "/privacy-request/{privacy_request_id}"
 
 would both resolve as `/api/v1/privacy-request` and `/api/v1/privacy-request/{privacy_request_id}` respectively.
 
-
 ## Database and Models
 
 ### The ORM -- SQLAlchemy
 
-SQLAlchemy is an Object Relational Mapper, allowing us to avoid writing direct database queries within our codebase, and access the database via Python code instead. The ORM provides an additional configuration layer allowing user-defined Python classes to be mapped to database tables and other constructs, as well as an object persistence mechanism known as the `Session`. Some common uses cases are listed below, for a more comprehensive guide see: https://docs.sqlalchemy.org/en/14/tutorial/index.html
-
+SQLAlchemy is an Object Relational Mapper, allowing us to avoid writing direct database queries within our codebase, and access the database via Python code instead. The ORM provides an additional configuration layer allowing user-defined Python classes to be mapped to database tables and other constructs, as well as an object persistence mechanism known as the `Session`. Some common uses cases are listed below, for a more comprehensive guide see: <https://docs.sqlalchemy.org/en/14/tutorial/index.html>
 
 ### Adding models
 
-Database tables are defined with model classes. Model files should live in `src/app/models/`. Individual model classes must inherit from our custom base class at `app.db.base_class.Base` to ensure uniformity within the database. Multiple models per file are encouraged so long as they fit the same logical delineation within the project. An example model declaration is added below. For a comprehensive guide see: https://docs.sqlalchemy.org/en/14/orm/mapping_styles.html#declarative-mapping
+Database tables are defined with model classes. Model files should live in `src/app/models/`. Individual model classes must inherit from our custom base class at `app.db.base_class.Base` to ensure uniformity within the database. Multiple models per file are encouraged so long as they fit the same logical delineation within the project. An example model declaration is added below. For a comprehensive guide see: <https://docs.sqlalchemy.org/en/14/orm/mapping_styles.html#declarative-mapping>
 You should also import your model in src/fidesops/db/base.py so it is visible for alembic.
+
 ```
 class Book(Base):
     __tablename__ = 'book'
@@ -41,20 +41,23 @@ class Book(Base):
     page_count = Column(Integer, nullable=True)
     author_id = Column(Integer, ForeignKey("author.id"), nullable=False)
 ```
-When models are added to the project, we must then add them to the database in a recordable and repeatable fashion using migrations.
 
+When models are added to the project, we must then add them to the database in a recordable and repeatable fashion using migrations.
 
 ### Using the database via models
 
-Once you've added database tables via project models, you're ready to read, write and update them via Python code. Some examples of common use cases here are listed below. Official documentation is here: https://docs.sqlalchemy.org/en/14/orm/query.html#sqlalchemy.orm.Query.
+Once you've added database tables via project models, you're ready to read, write and update them via Python code. Some examples of common use cases here are listed below. Official documentation is here: <https://docs.sqlalchemy.org/en/14/orm/query.html#sqlalchemy.orm.Query>.
 
-- Import our application's database session: `from fidesops.db.session import get_db_session`
+- Import our application's database session: `from fidesops.ops.db.session import get_db_session`
 - Instantiate the database interaction object:
+
 ```
 SessionLocal = get_db_session(config)
 db = SessionLocal()
 ```
+
 - Create a new row in a table:
+
 ```
 db_obj = User(
     email="admin@fidesops.app",
@@ -66,10 +69,12 @@ db.add(db_obj)
 db.commit()
 db.refresh(db_obj)
 ```
+
 - Fetch all objects in a table: `users = db.query(User).all()`
 - Fetch all objects in a table that meet some criteria: `active_users = db.query(User).filter(User.is_active == True)`
 - Get a specific row in a table: `user = db.query(User).get(User.email == "admin@fidesops.app")`
 - Update a specific row in a table:
+
 ```
 user.email = "updated@fidesops.app"
 db.add(user)
@@ -78,17 +83,17 @@ db.refresh()
 ```
 
 ### Connecting to the database
+
 When you run `nox -s dev`, the database will be spun up in a Docker container with port `5432` exposed on localhost. You can connect to it using the credentials found in `.fidesops.toml`, e.g.
 
 - Hostname: `localhost`
 - Port: `5432`
 - Username: see `database.user` in `.fidesops.toml`
 - Password: see `database.password` in `.fidesops.toml`
 
-
 ### Alembic migrations
 
-Some common Alembic commands are listed below. For a comprehensive guide see: https://alembic.sqlalchemy.org/en/latest/tutorial.html. 
+Some common Alembic commands are listed below. For a comprehensive guide see: <https://alembic.sqlalchemy.org/en/latest/tutorial.html>.
 
 The commands will need to be run inside a shell on your Docker containers, which can be opened with `nox -s dev -- shell`.
 
@@ -99,7 +104,7 @@ In the `/src/fidesops` directory:
 - Automatically generate a new migration: `alembic revision --autogenerate -m "<a message describing your changes>"`
 - Create a new migration file to manually fill out: `alembic revision -m "<a message describing your changes>"`
 - Migrate your database to a specific state `alembic upgrade <revision-id>` or `alembic downgrade <revision-id>`, (or if you want to be smart `alembic upgrade <revision-id> || alembic downgrade <revision-id>` is handy when you don't know whether the target revision is an upgrade or downgrade)
-    - NB. You can find the `revision-id` inside each migration file in `alembic/versions/` on line 3 next to `Revision ID: ...`
+  - NB. You can find the `revision-id` inside each migration file in `alembic/versions/` on line 3 next to `Revision ID: ...`
 
 When working on a PR with a migration, ensure that `down_revision` in the generated migration file correctly references the previous migration before submitting/merging the PR.
 
@@ -116,12 +121,10 @@ def some_method():
     raise SomeException("a message")
 ```
 
-
 ## General debugging -- pdb
 
 The project uses `pdb` for debugging as a `dev-requirement`. You can set breakpoints with `pdb` in much the same way you'd set them using `debugger` in Javascript. Insert `import pdb; pdb.set_trace()` into the line where you want the breakpoint to set, then run your Python code.
 
-
 ## Docker
 
 Occasionally when developing you'll run into issues where it's beneficial to remove all existing Docker instances in order to recreate them based on some updated spec. Some commands to do this are below:
@@ -131,4 +134,4 @@ Occasionally when developing you'll run into issues where it's beneficial to rem
 - Delete all local Docker volumes: `docker volume rm $(docker volume ls -q)`
 - Remove temp. files, installed dependencies, all local Docker containers and all local Docker volumes: `nox -s clean`
 - Delete all stopped containers, all networks not used by a container, all dangling images, and all build cache: `docker system prune`
-- Recreate the project: `nox -s "build(dev)"`
+- Recreate the project: `nox -s "build(dev)"`