Code

Add GCP implementation#423

Code

Merged

nmerket merged 111 commits intoNREL:developNREL/buildstockbatch:developfrom rewiringamerica:gcprewiringamerica/buildstockbatch:gcpCopy head branch name to clipboardMar 29, 2024

+1,756-51

Commits

Commits on Oct 19, 2023

• Add the beginnings of support for buildstockbatch on GCP

  Show description for 6da2cb2
  lathanh

  committed
  6da2cb2

  Copy full SHA for 6da2cb2

Commits on Oct 20, 2023

• Merge pull request #1 from rewiringamerica/gcp-start

  Show description for cbce5c8
  lathanh

  authored
  cbce5c8

  Copy full SHA for cbce5c8

Commits on Oct 24, 2023

• Start outline of documentation updates for running on GCP (#2)

  nweires

  authored
  2cbb454

  Copy full SHA for 2cbb454

Commits on Oct 25, 2023

• Run basic GCP batch job (#3)

  nweires

  authored
  6d83a7d

  Copy full SHA for 6d83a7d

Commits on Oct 30, 2023

• Flesh out GCP documentation for installation and project definition

  lathanh

  committed
  954d4b7

  Copy full SHA for 954d4b7

• Flesh out GCP documentation for installation and project definition

  lathanh

  committed
  fa33798

  Copy full SHA for fa33798

Commits on Oct 31, 2023

• Add GCP config validation (#6)

  nweires

  authored
  454dc5d

  Copy full SHA for 454dc5d

• Add basic terraform file (#5)

  Show description for 07d5e8a
  nweires

  authored
  07d5e8a

  Copy full SHA for 07d5e8a

• Improvements and fixes from PR comments

  lathanh

  committed
  4c2ee04

  Copy full SHA for 4c2ee04

• Merge pull request #7 from rewiringamerica/lathanh/gcp-docs

  Show description for d841947
  lathanh

  authored
  d841947

  Copy full SHA for d841947

Commits on Nov 1, 2023

• Run building simulations on GCP Batch (#4)

  nweires

  authored
  1a5be50

  Copy full SHA for 1a5be50

Commits on Nov 3, 2023

• Show status of individual tasks (#9)

  nweires

  authored
  048b66a

  Copy full SHA for 048b66a

• Add postprocessing support to GCP (#11)

  Show description for 0ad0279
  lathanh

  authored
  0ad0279

  Copy full SHA for 0ad0279

Commits on Nov 6, 2023

• Update settings and defaults for job resources (#8)

  nweires

  authored
  0e93574

  Copy full SHA for 0e93574

• Show batch job progress (#12)

  Show description for 45d51b7
  nweires

  authored
  45d51b7

  Copy full SHA for 45d51b7

• Fix write path for postprocessing for GCP (#13)

  lathanh

  authored
  8bb583c

  Copy full SHA for 8bb583c

Commits on Nov 7, 2023

• Run Black formatter (#14)

  nweires

  authored
  ab0e460

  Copy full SHA for ab0e460

• Remove notifications email option (#15)

  nweires

  authored
  eba8910

  Copy full SHA for eba8910

Commits on Nov 8, 2023

• Fix documentation cross-reference (#17)

  lathanh

  authored
  e3ce11c

  Copy full SHA for e3ce11c

• Merge branch 'develop' into gcp (#16)

  Show description for fd3a7a9
  

  3 peopleauthored
  fd3a7a9

  Copy full SHA for fd3a7a9

Commits on Nov 9, 2023

• Omit GCP code from test coverage checks (#19)

  nweires

  authored
  74708de

  Copy full SHA for 74708de

• Merge branch 'develop' into natalie/merge

  nweires

  committed
  d2ab4dd

  Copy full SHA for d2ab4dd

• Fix broken 'Create a repository' link in installation.rst

  lathanh

  authored
  50afeda

  Copy full SHA for 50afeda

• Merge pull request #20 from rewiringamerica/natalie/merge

  Show description for 350bae4
  nweires

  authored
  350bae4

  Copy full SHA for 350bae4

Commits on Nov 10, 2023

• Use shared code in GCP implementation (#22)

  nweires

  authored
  495455f

  Copy full SHA for 495455f

Commits on Nov 13, 2023

• Changes to how we handle job IDs (#18)

  nweires

  authored
  9e2452f

  Copy full SHA for 9e2452f

Commits on Nov 14, 2023

• Make GCP's arg name in documentation match the script's name

  lathanh

  authored
  4dbb2d0

  Copy full SHA for 4dbb2d0

• Make job_identifier actually optional (#24)

  lathanh

  authored
  ad67859

  Copy full SHA for ad67859

• Use default application credentials (when available) to push to AR (#25)

  nweires

  authored
  f5b0011

  Copy full SHA for f5b0011

Commits on Nov 15, 2023

• Add support for running post-processing on GCP Cloud Run (#21)

  lathanh

  authored
  68faf2f

  Copy full SHA for 68faf2f

Commits on Nov 17, 2023

• Log basic job stats (#28)

  nweires

  authored
  796e471

  Copy full SHA for 796e471

• Merge branch 'develop' into gcp

  nweires

  committed
  e92d1db

  Copy full SHA for e92d1db

• Skip post-processing of upgrade directories that fail to list (#29)

  Show description for 400f0d6
  lathanh

  authored
  400f0d6

  Copy full SHA for 400f0d6

• Add exception handling to Cloud Run job start attempt

  lathanh

  committed
  921662e

  Copy full SHA for 921662e

Commits on Nov 21, 2023

• Apply Black formatter (#30)

  nweires

  authored
  0db948b

  Copy full SHA for 0db948b

• Update --list_jobs option

  nweires

  committed
  c0506da

  Copy full SHA for c0506da

• Update task timeouts and retries

  nweires

  committed
  71112e4

  Copy full SHA for 71112e4

• Clean up copy

  nweires

  committed
  e6118f1

  Copy full SHA for e6118f1

• Merge pull request #32 from rewiringamerica/develop

  Show description for a019c83
  nweires

  authored
  a019c83

  Copy full SHA for a019c83

Commits on Nov 27, 2023

• Respond to review comments

  nweires

  committed
  ac67e71

  Copy full SHA for ac67e71

• Merge pull request #31 from rewiringamerica/natalie/list_jobs

  Show description for e034567
  nweires

  authored
  e034567

  Copy full SHA for e034567

Commits on Nov 29, 2023

• Add retry to post-processing job start (#35)

  lathanh

  authored
  4e555ce

  Copy full SHA for 4e555ce

• Refactor docker_base to use inversion-of-control, and GCP to use that (#34)

  lathanh

  authored
  2829d9e

  Copy full SHA for 2829d9e

Commits on Nov 30, 2023

• Make post-processing wait until completion, then report stats (#37)

  lathanh

  authored
  12524dd

  Copy full SHA for 12524dd

Commits on Dec 1, 2023

• Validate post-processing settings (#36)

  nweires

  authored
  05c927b

  Copy full SHA for 05c927b

• Include AR docker images in cleanup (#38)

  nweires

  authored
  ef99f20

  Copy full SHA for ef99f20

• Fix TsvLogger to use instance variables

  lathanh

  committed
  ed7c3fa

  Copy full SHA for ed7c3fa

• Make combine_results fail rather than stall upon exception with Dask (#39)

  lathanh

  authored
  a9675af

  Copy full SHA for a9675af

Commits on Dec 2, 2023

• Fix string with formatting that should be an f-string

  lathanh

  authored
  d5d9630

  Copy full SHA for d5d9630

Commits on Dec 6, 2023

• Quickly fail if an existing job has the same ID (#40)

  nweires

  authored
  f0bdd19

  Copy full SHA for f0bdd19

• Run sampling before uploading assets (#41)

  nweires

  authored
  932cbef

  Copy full SHA for 932cbef

• Include failed tasks as progress towards batch completion

  lathanh

  committed
  3618560

  Copy full SHA for 3618560

Commits on Dec 8, 2023

• Merge branch 'develop' into gcp

  nweires

  committed
  a65239d

  Copy full SHA for a65239d

Commits on Dec 11, 2023

• Apply refactor to gcp.py (#43)

  nweires

  authored
  10426a4

  Copy full SHA for 10426a4

• Add support for non-default service accounts (#42)

  nweires

  authored
  7f3cae7

  Copy full SHA for 7f3cae7

Commits on Dec 15, 2023

• Add validation ensuring all EPW weather files needed by the batch are present (#45)

  Show description for 06d5f0b
  lathanh

  authored
  06d5f0b

  Copy full SHA for 06d5f0b

• Add a parallelism parameter for GCP Batch (#47)

  lathanh

  authored
  c3090b4

  Copy full SHA for c3090b4

• Improve weather validation error message wording

  lathanh

  committed
  fc03008

  Copy full SHA for fc03008

Commits on Dec 20, 2023

• Add Cloud Storage links for easy browsing of simulation and post-processing output

  lathanh

  committed
  14fea33

  Copy full SHA for 14fea33

Commits on Dec 21, 2023

• Add job_identifier as a label to the batch job and post-processing Run job

  lathanh

  committed
  1903679

  Copy full SHA for 1903679

Commits on Jan 2, 2024

• Merge pull request #49 from rewiringamerica/develop

  Show description for 7eda492
  nweires

  authored
  7eda492

  Copy full SHA for 7eda492

• Log summary of simulation statuses (#46)

  nweires

  authored
  427edca

  Copy full SHA for 427edca

Commits on Jan 8, 2024

• Update docs, remove unused option, small cleanups (#50)

  nweires

  authored
  094b678

  Copy full SHA for 094b678

• Merge branch 'develop' into gcp

  nweires

  committed
  4614282

  Copy full SHA for 4614282

Commits on Jan 9, 2024

• Collect simulation errors in results json (#51)

  nweires

  authored
  38a022c

  Copy full SHA for 38a022c

Commits on Jan 10, 2024

• Add option to set upload chunk size in project config (#53)

  nweires

  authored
  3657fb4

  Copy full SHA for 3657fb4

• Add labels to more of the resources used by Batch (#56)

  Show description for 8e03769
  lathanh

  authored
  8e03769

  Copy full SHA for 8e03769

Commits on Jan 11, 2024

• Fix the ability to skip baseline sims in Docker-based implementations (#52)

  lathanh

  authored
  7ef4514

  Copy full SHA for 7ef4514

• Clean up docker containers (#55)

  nweires

  authored
  03b7e88

  Copy full SHA for 03b7e88

Commits on Jan 12, 2024

• Add option to request extra boot disk space for Batch Tasks (#57)

  lathanh

  authored
  2522573

  Copy full SHA for 2522573

Commits on Jan 16, 2024

• Simplify docker cleanup (#58)

  nweires

  authored
  5f3d52a

  Copy full SHA for 5f3d52a

• Turn off external IPs (#54)

  nweires

  authored
  5000f6b

  Copy full SHA for 5000f6b

Commits on Jan 22, 2024

• Increase retry limit (#60)

  nweires

  authored
  b7fad48

  Copy full SHA for b7fad48

Commits on Jan 24, 2024

• Add the option to rerun failed tasks.

  nweires

  committed
  d9460e2

  Copy full SHA for d9460e2

• Cleanups

  nweires

  committed
  08b13ee

  Copy full SHA for 08b13ee

• Refactor: make missing_only an instance variable

  nweires

  committed
  6eff54f

  Copy full SHA for 6eff54f

• More refactoring and comment fixes

  nweires

  committed
  6fa23a3

  Copy full SHA for 6fa23a3

• formatting

  nweires

  committed
  10268b4

  Copy full SHA for 10268b4

Commits on Jan 25, 2024

• Update GCP docs (#59)

  nweires

  authored
  8b691c8

  Copy full SHA for 8b691c8

• Merge branch 'gcp' into natalie/retry-missing

  nweires

  committed
  93c587f

  Copy full SHA for 93c587f

Commits on Jan 26, 2024

• Review comment updates

  nweires

  committed
  a941448

  Copy full SHA for a941448

Commits on Jan 29, 2024

• Docs and logging improvements (#63)

  nweires

  authored
  cdb1649

  Copy full SHA for cdb1649

Commits on Jan 31, 2024

• Merge pull request #62 from rewiringamerica/natalie/retry-missing

  Show description for feb1607
  mfathollahzadeh

  authored
  feb1607

  Copy full SHA for feb1607

Commits on Feb 2, 2024

• Update how we start post-processing job (#64)

  nweires

  authored
  75c951e

  Copy full SHA for 75c951e

Commits on Feb 6, 2024

• Add check for existing output files during validation (#65)

  nweires

  authored
  bcc584c

  Copy full SHA for bcc584c

• Option to delete existing prefix with user input

  mfathollahzadeh

  committed
  b4e7acc

  Copy full SHA for b4e7acc

• skip optional deletion with missingonly flag

  mfathollahzadeh

  committed
  1a77d11

  Copy full SHA for 1a77d11

Commits on Feb 7, 2024

• skip optional deletion of existing files in the bucket with missingonly flag

  mfathollahzadeh

  committed
  ee35fae

  Copy full SHA for ee35fae

• removing the second user input for missingonly flg

  mfathollahzadeh

  committed
  27be5a1

  Copy full SHA for 27be5a1

Commits on Feb 8, 2024

• skip optional deletion works with missingonly flag

  mfathollahzadeh

  committed
  e6bad58

  Copy full SHA for e6bad58

Commits on Feb 9, 2024

• Merge pull request #66 from rewiringamerica/mfath

  Show description for 7f60659
  mfathollahzadeh

  authored
  7f60659

  Copy full SHA for 7f60659

Commits on Feb 15, 2024

• Clean up output dir validation (#67)

  nweires

  authored
  65c8703

  Copy full SHA for 65c8703

Commits on Feb 27, 2024

• Separate GCP dependencies (#68)

  nweires

  authored
  11ea6ff

  Copy full SHA for 11ea6ff

Commits on Feb 28, 2024

• Small resource docs updates (#69)

  nweires

  authored
  500d7e5

  Copy full SHA for 500d7e5

Commits on Mar 14, 2024

• skip deletion in postprocessonly flag

  mfathollahzadeh

  committed
  2f56442

  Copy full SHA for 2f56442

• skip deletion in postprocessonly flag. rebase

  mfathollahzadeh

  committed
  687018c

  Copy full SHA for 687018c

• deletion rebase

  mfathollahzadeh

  committed
  cdaa68d

  Copy full SHA for cdaa68d

Commits on Mar 15, 2024

• Fix rebase

  nweires

  committed
  b1c467e

  Copy full SHA for b1c467e

Commits on Mar 20, 2024

• Merge in AWS updates (#72)

  nweires

  authored
  374794b

  Copy full SHA for 374794b

Commits on Mar 21, 2024

• A few fixes for compatibility with ComStock (#73)

  nweires

  authored
  afb7c6d

  Copy full SHA for afb7c6d

Commits on Mar 26, 2024

• Merge branch 'develop' into gcp

  nweires

  committed
  7a06247

  Copy full SHA for 7a06247

• Merge branch 'gcp' of https://github.com/rewiringamerica/buildstockbatch into gcp

  nweires

  committed
  a219571

  Copy full SHA for a219571

• Merge branch 'develop' into gcp

  nweires

  committed
  0cd324d

  Copy full SHA for 0cd324d

• Clean up installation docs

  nweires

  committed
  96d0857

  Copy full SHA for 96d0857

• Merge branch 'develop' into gcp

  nweires

  committed
  3c561ea

  Copy full SHA for 3c561ea

• Add config option for simulation duration (#76)

  nweires

  authored
  e3720d8

  Copy full SHA for e3720d8

• Cleanups and merge fixes

  nweires

  committed
  ec61843

  Copy full SHA for ec61843

Commits on Mar 27, 2024

• Cleanup

  nweires

  committed
  797d614

  Copy full SHA for 797d614

• Fix test

  nweires

  committed
  56684e7

  Copy full SHA for 56684e7

• Add changelog entry

  nweires

  committed
  0838e4a

  Copy full SHA for 0838e4a

Commits on Mar 28, 2024

• Remove unused imports

  nweires

  committed
  2cb74a8

  Copy full SHA for 2cb74a8