Generate Dataset Squashes Tables From Multiple Schemas

[brentonmallen1]

Issue:

• when using generate-dataset on a the slice database (mysql and/or postgres), which has multiple user created databases/schemas (mypizza and test), the resulting auto-generated dataset.yml file only contains the mypizza schema with the test schemas appended to the list of collection objest.

Expected:

• The resulting database.yml file should have two databases/schemas each containing their respective tables under their collection list

Investigation:

• I believe this nested dict (https://github.com/ethyca/fides/blob/main/fidesctl/src/fidesctl/core/generate_dataset.py#L28), while written to allow multiple schemas, doesn't separate them in a way that would yield multiple databases/schemas for the dataset.yml file generation.

Impact:

• Users who take advantage of the generate-dataset functionality in the CLI may not notice extra tables associated with a given dataset in the output file.
• Potential impact on annotate-dataset, though I will try to make it so it doesn't either way
• Not sure what the impact, if any, there will be on the fides system in general.

Potential solutions:

• Fix the nested dict highlighted above so that it handles multiple schemas/databases appropriately
• Enforce one dataset.yml file per database/schema
• I'm sure there are other approaches

[brentonmallen1]

Further impact/consideration:

• Currently, write_manifest is expecting a single dataset dictionary and will write a single file

[brentonmallen1]

I looked into this a bit more due to the conversation here: #176 (comment)

It seems to me that a few functions need to be refactored to support multiple datasets in a single dataset.yml file. specifically the following functions in generate_dataset.py:

• https://github.com/ethyca/fides/blob/main/fidesctl/src/fidesctl/core/generate_dataset.py#L12
• https://github.com/ethyca/fides/blob/main/fidesctl/src/fidesctl/core/generate_dataset.py#L39
• https://github.com/ethyca/fides/blob/main/fidesctl/src/fidesctl/core/generate_dataset.py#L65

They seem to be written to expect that a single dataset exits.

I'm not sure if manifests.write_manifest is actually impacted due to it being a yaml dump. I imagine if the the dataset dictionary is correctly compiled that function shouldn't have to change.

For a few reasons, most importantly due to the proximity to the web summit, I don't feel comfortable addressing this at this time. I at least wanted to scope out the level of effort here as the result will cause a refactor in annotate_dataset.py

[brentonmallen1]

Taking care of this in #176 since it's dependent