parsons-prefect-gcp-deployment

This repo is an example of how to use a Prefect automatic deployment and provisioning of workflow on GCP.

Here are all the relevant Prefect Docs.

https://docs.prefect.io/latest/guides/deployment/push-work-pools/

Some prereqs are that you need to have a GCP account with a billing account and you need Docker installed and running.

Getting Started:

1. Clone the repository:

   git clone https://github.com/matthewkrausse/parsons-prefect-gcp-deployment
   

2. Create a Virtual Environment:

   python3.10 -m venv venv
   

3. Activate the venv:

   source venv/bin/activate
   

4. Install the reqs:

   pip install -r requirements.txt
   

5. Install the gcloud cli client:

   Install the gcloud CLI (https://cloud.google.com/sdk/docs/install) and authenticate with your GCP project (https://cloud.google.com/docs/authentication/gcloud).

   Also pull any updates

   gcloud components update
   

6. Download Docker:

   https://docs.docker.com/get-docker/

7. Create the work pool: Using the --provision-infra flag will allow you to select a GCP project to use for your work pool and automatically configure it to be ready to execute flows via Cloud Run. In your GCP project, this command will activate the Cloud Run API, create a service account, and create a key for the service account, if they don't already exist. In your Prefect workspace, this command will create a GCPCredentials block for storing the service account key.

   prefect work-pool create --type cloud-run:push --provision-infra my-cloud-run-pool
   

8. Run the built in prefect flow deployment or create your own and run it

   python test_deployment.py
   

9. Run the flow using the prefect UI or locally

   It's optional to use the UI but you can create a free cloud account. When you run the above command, it will provide you code to run the flow through the CLI.

10. Let's go check the logs in Google Cloud and see that it really ran in the cloud!

11. That's it!