From 4f34ef519a19075beb1cd63b265194cfbf707fd8 Mon Sep 17 00:00:00 2001 From: Natalie Weires Date: Thu, 19 Oct 2023 16:37:20 -0400 Subject: [PATCH 1/3] Start outline of doc updates for running on GCP --- docs/installation.rst | 7 +++++++ docs/project_defn.rst | 7 +++++++ docs/run_sims.rst | 49 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 63 insertions(+) diff --git a/docs/installation.rst b/docs/installation.rst index edc910a3..1b0d6c39 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -188,3 +188,10 @@ The installation instructions are the same as the :ref:`local-install` installation. You will need to use an AWS account with appropriate permissions. The first time you run ``buildstock_aws`` it may take several minutes, especially over a slower internet connection as it is downloading and building a docker image. + + +Google Cloud Platform +~~~~~~~~~~~~~~~~~~~~~ + +TODO: Add any GCP-specific installation instructions +TODO: Need to install and run docker? diff --git a/docs/project_defn.rst b/docs/project_defn.rst index 35d46e71..98329c99 100644 --- a/docs/project_defn.rst +++ b/docs/project_defn.rst @@ -218,6 +218,13 @@ on the `AWS Batch `_ service. .. _instance type: https://aws.amazon.com/ec2/instance-types/ +.. _gcp-config: + +GCP Configuration +~~~~~~~~~~~~~~~~~ +TODO: Add GCP configuration details + + .. _postprocessing: Postprocessing diff --git a/docs/run_sims.rst b/docs/run_sims.rst index 1389a47a..7b3ea883 100644 --- a/docs/run_sims.rst +++ b/docs/run_sims.rst @@ -110,3 +110,52 @@ When the simulation and postprocessing is all complete, run ``buildstock_aws --clean your_project_file.yml``. This will clean up all the AWS resources that were created on your behalf to run the simulations. Your results will still be on S3 and queryable in Athena. + +Google Cloud Platform +~~~~~~~~~~~~~~~~~~~~~ + +Running a batch on GCP is done by calling the ``buildstock_gcp`` command line +tool. + +.. command-output:: buildstock_gcp --help + :ellipsis: 0,8 + +GCP Specific Project configuration +.................................. + +TODO: add more GCP configuration details here + +For the project to run on GCP, you will need to add a section to your config +file, something like this: + +.. code-block:: yaml + + gcp: + job_identifier: national01 + gcs: + bucket: mybucket + prefix: national01_run01 + gcp_project: myorg_project + use_spot: true + batch_array_size: 10000 + notifications_email: your_email@somewhere.com + region: us-central1 + +See :ref:`gcp-config` for details. + + +List existing jobs +.................. + +Run ``buildstock_gcp --list_jobs your_project_file.yml`` to see a list of all existing +jobs matching the project specified. This can show you whether a previously-started job +has completed or is still running. + + +Cleaning up after yourself +.......................... + +When the simulation and postprocessing is all complete, run ``buildstock_gcp +--clean your_project_file.yml``. This will clean up all the GCP resources that +were created to run the specified project. If the project is still running, it +will be cancelled. Your output files will still be available in GCS. From 5a658f2f434a42aaf097795671fcbc616ec369e7 Mon Sep 17 00:00:00 2001 From: Natalie Weires Date: Fri, 20 Oct 2023 10:10:04 -0400 Subject: [PATCH 2/3] Update example config --- docs/run_sims.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/run_sims.rst b/docs/run_sims.rst index 7b3ea883..a687684e 100644 --- a/docs/run_sims.rst +++ b/docs/run_sims.rst @@ -132,14 +132,15 @@ file, something like this: gcp: job_identifier: national01 + gcp_project: myorg_project + region: us-central1 + artifact_registry: buildstockbatch gcs: bucket: mybucket prefix: national01_run01 - gcp_project: myorg_project use_spot: true batch_array_size: 10000 notifications_email: your_email@somewhere.com - region: us-central1 See :ref:`gcp-config` for details. From 327431cb1eb63e3b14b85d3329de50ace274b91d Mon Sep 17 00:00:00 2001 From: Natalie Weires Date: Fri, 20 Oct 2023 14:56:09 -0400 Subject: [PATCH 3/3] Add TODO --- docs/run_sims.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/run_sims.rst b/docs/run_sims.rst index a687684e..927287cd 100644 --- a/docs/run_sims.rst +++ b/docs/run_sims.rst @@ -156,6 +156,8 @@ has completed or is still running. Cleaning up after yourself .......................... +TODO: Review and update this after implementing cleanup option. + When the simulation and postprocessing is all complete, run ``buildstock_gcp --clean your_project_file.yml``. This will clean up all the GCP resources that were created to run the specified project. If the project is still running, it