feat(ui): UI-based ingestion (as featured in Dec Townhall)

[jjoyce0510]

Summary

This PR introduces support for UI-Based Metadata Ingestion, as originally presented in the December 2021 Townhall.

Highlights

• New Entities: DataHubIngestionSource - Represents a unique source of metadata ingestion on DataHub. DataHubExecutionRequest - Represents a request to execute a named task, in this case ingestion. DataHubSecret which stores encrypted secrets in DataHub's backend.
• Support MCL "hooks", or code that is invoked for each MCL received from Kafka.
• Embedded Ingestion Scheduler hook responsible for creating ingestion execution requests as specified CRON intervals.
• Ingestion Tab in the UI
• Adding acryl-datahub-actions container as a dependency for Quickstart.

Access Control

To view and manage UI ingestion, users will need to be assigned the following privileges:

1. Manage Metadata Ingestion
2. Manage Secrets

via a policy. The root datahub user has these by default.

Usage

To use UI ingestion, simply navigate to the 'Ingestion' tab, create some secrets, and create a new ingestion source.
Note that for quickstart deployments, you'll need to point your ingestion toward http://datahub-gms:8080 if your NOT using Metadata Service Authentication yet. If you are, you can point to http://localhost:9002/api/gms

Configuration

Disabling UI Ingestion

Managed ingestion can be completely disabled, such that it does not appear in the UI, by setting the environment variable UI_INGESTION_ENABLED=false on the datahub-gms container / pod.

Encryption

By default, all Secrets created on DataHub are encrypted using AES encryption with a 128-bit encryption key. The final key used to encrypt + decrypt secrets is generated by running the value provided in the datahub-gms SECRET_SERVICE_ENCRYPTION_KEY environment variable through SHA-1 digest, and then taking the first 16-bytes of the result.

Screenshots

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable)

[github-actions]

Unit Test Results (build & test)

  60 files  +15    60 suites  +15   17m 0s ⏱️ + 4m 39s
558 tests +58  499 ✔️ +56  59 💤 +3  0 ❌  - 1 
558 runs  +57  499 ✔️ +56  59 💤 +3  0 ❌  - 2 

Results for commit 82affe3. ± Comparison against base commit 40bae0d.

♻️ This comment has been updated with latest results.

[github-actions]

Unit Test Results (metadata ingestion)

       5 files         5 suites   38m 55s ⏱️
   274 tests    273 ✔️   0 💤 1 ❌
1 261 runs  1 232 ✔️ 28 💤 1 ❌

For more details on these failures, see this check.

Results for commit 82affe3.

♻️ This comment has been updated with latest results.