Switch to cloud.json for cloud-netblocks

[cagataygurturk]

From https://cloud.google.com/compute/docs/faq#find_ip_range

Important: In the past, Google Cloud instructed users to inspect the _cloud-netblocks.googleusercontent.com DNS TXT record (and the records it referenced). Please update your scripts or software libraries so that they read from the cloud.json file instead. The JSON file includes additional information, such as the region to which a regional external IP address is attached.

while the module is still querying the DNS record.

Maybe it would be better to switch to the JSON file as the documentation recommends.

[c2thorn]

Similar to the work done with google-netblocks here:
GoogleCloudPlatform/magic-modules#3689

Seems reasonable to do, I'll add the appropriate tag

[edwardday7]

Since the IP addresses between the DNS record and the JSON file are different we have experienced some traffic being blocked unexpectedly. Just an FYI for anyone reading, you might want to hard code the IP ranges for the time being until this gets updated.

[mi1os]

in case anyone is looking for workaround this is what I've implemented on our side until the provider is fixed

@@ -73,15 +73,19 @@ module "XXX" {
   private_ip_google_access = true
 }

-data "google_netblock_ip_ranges" "netblock" {
-  range_type = "google-netblocks"
+data "external" "google_api_ranges" {
+  program = ["${path.module}/helpers/google-private-service-ips.py"]
+}
+
+locals {
+  google_private_service_ips = split(",", data.external.google_api_ranges.result.ipv4)
 }

 resource "google_compute_route" "XXX" {
-  count = length(data.google_netblock_ip_ranges.netblock.cidr_blocks_ipv4)
+  count = length(local.google_private_service_ips)

   name             = "XXX-private-access-${count.index}"
-  dest_range       = data.google_netblock_ip_ranges.netblock.cidr_blocks_ipv4[count.index]
+  dest_range       = local.google_private_service_ips[count.index]
   network          = google_compute_network.network.self_link
   next_hop_gateway = "default-internet-gateway"
   tags             = ["XXX"]

and the helper script is slightly modified version of the script from doc at https://cloud.google.com/vpc/docs/configure-private-google-access#ip-addr-defaults

#
# https://cloud.google.com/vpc/docs/configure-private-google-access#ip-addr-defaults

import base64
import json
import netaddr
import urllib.request

goog_url="www.gstatic.com/ipranges/goog.json"
cloud_url="www.gstatic.com/ipranges/cloud.json"

def read_url(url, fallback_to_http=False,attempts=2):
   if fallback_to_http:
      url = "http://" + url
   else:
      url = "https://" + url
   try:
      s = urllib.request.urlopen(url).read()
      return json.loads(s)
   except urllib.error.HTTPError:
      print("Invalid HTTP response from %s" % url)
      return {}
   except json.decoder.JSONDecodeError:
      print("Could not parse HTTP response from %s" % url)
      return {}
   except urllib.error.URLError:
      if attempts > 1:
         url = url.replace("https://","").replace("http://","")
         attempts -=1
         print("Error opening URL; trying HTTP instead of HTTPS.")
         return read_url(url,fallback_to_http=True,attempts=attempts)
      else:
         print("Error opening URL.")
         return {}

def main():
   goog_json=read_url(goog_url)
   cloud_json=read_url(cloud_url)

   if goog_json and cloud_json:
#      print("{} published: {}".format(goog_url,goog_json.get('creationTime')))
#      print("{} published: {}".format(cloud_url,cloud_json.get('creationTime')))
      goog_cidrs = netaddr.IPSet()
      for e in goog_json['prefixes']:
         if e.get('ipv4Prefix'):
            goog_cidrs.add(e.get('ipv4Prefix'))
      cloud_cidrs = netaddr.IPSet()
      for e in cloud_json['prefixes']:
         if e.get('ipv4Prefix'):
            cloud_cidrs.add(e.get('ipv4Prefix'))
#      print("IP ranges for Google APIs and services default domains:")
      result = []
      for i in goog_cidrs.difference(cloud_cidrs).iter_cidrs():
#        print(i)
        result.append(str(i))
      print(json.dumps({"ipv4": ",".join(result)}))

if __name__=='__main__':
   main()

[cagataygurturk]

I have sent a PR to fix this #7157

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!