GCP: Support for custom VPC.

[concretevitamin]

Support specifying a specific VPC to use for GCP.

In ~/.sky/config.yaml:

gcp:
  # VPC to use (optional).
  #
  # Default: None, which implies the following behavior. First, the VPC named
  # 'default' is checked against minimal recommended firewall rules for
  # SkyPilot to function. If it satisfies these rules, this VPC is used.
  # Otherwise, a new VPC named 'skypilot-vpc' is automatically created with
  # the minimal recommended firewall rules and will be used.
  #  
  # If this field is set, SkyPilot will use the VPC with this name. Useful for
  # when users want to manually set up a VPC and precisely control its
  # firewall rules. If no region restrictions are given, SkyPilot only
  # provisions in regions for which a subnet of this VPC exists. Errors are
  # thrown if VPC with this name is not found. The VPC does not get modified
  # in any way, except when opening ports (e.g., via `resources.ports`) in
  # which case new firewall rules permitting public traffic to those ports
  # will be added.
  vpc_name: skypilot-vpc

TODO

• check if statement "VPC does not get modified" above is correct; e.g., does exposing ports modify the VPC
  • incorrect; reworded

Tested (run the relevant ones):

• Code formatting: bash format.sh

• Any manual or new tests for this PR (please specify below)

  • With 'default' vpc, that only has 1 subnet (us-central1):
    • sky launch --cloud gcp --cpus 2+ -c dbg2 --region us-west4 correctly fails + skips the entire region.
  • With a 'nonexistent' vpc
    • sky launch --cpus 2+ -c dbg --use-spot correctly skips GCP and fails over to AWS
    • sky launch --cloud gcp --cpus 2+ -c dbg --use-spot correctly errors out with no failover
    • sky tpunode correctly errors out with no failover
  • Existing GCP spot controller under 'default' vpc (has 1 region, us-central1)
    • set config.yaml to use 'skypilot-vpc' vpc (has all regions), launch a new spot job in GCP us-west4, check that new spot cluster is launched under this VPC and region

• All smoke tests: pytest tests/test_smoke.py: pytest tests/test_smoke.py --gcp

• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name

• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[Michaelvll]

Thanks for adding this @concretevitamin! This is very useful for our users. Just left sevral comments, and I will try it out soon.

[concretevitamin]

Thanks, PTAL.

[Michaelvll]

Thanks for the update @concretevitamin! LGTM.

[Michaelvll]

May need to coordinate with #2772. We can merge this PR first and then update #2772.

[concretevitamin]

Done, PTAL.

[Michaelvll]

Thanks for adding the doc and the fixes @concretevitamin! LGTM.

[Michaelvll]

assert specific_vpc_to_use is None or add a comment for readability : )

[concretevitamin]

Added # Should not reach here.

[concretevitamin]

• pytest tests/test_smoke.py --gcp

[dongreenberg]

Thank you @concretevitamin !!