Fetch Google netblock from JSON on gstatic domain #3689

bcadiot · 2020-06-18T22:14:48Z

Release Note Template for Downstream PRs (will be copied)

netblock: fixed the google netblock ranges returned by the `google_netblock_ip_ranges` by targeting json on gstatic domain instead of reading SPF dns records (solution provided by network team)

We experienced the issue after seeing that some google domains are outside of the IPs provided by the SPF records.
For example :

ig datastudio.google.com
; <<>> DiG 9.10.6 <<>> datastudio.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39814
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;datastudio.google.com.		IN	A
;; ANSWER SECTION:
datastudio.google.com.	299	IN	A	142.250.31.101
datastudio.google.com.	299	IN	A	142.250.31.102
datastudio.google.com.	299	IN	A	142.250.31.138
datastudio.google.com.	299	IN	A	142.250.31.113
datastudio.google.com.	299	IN	A	142.250.31.100
datastudio.google.com.	299	IN	A	142.250.31.139
;; Query time: 96 msec
;; SERVER: 10.114.228.14#53(10.114.228.14)
;; WHEN: Fri Jun 12 11:33:54 CEST 2020
;; MSG SIZE  rcvd: 146

Currently the datasource return:

data.google_netblock_ip_ranges.netblock: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

cidr_blocks = [
  "35.190.247.0/24",
  "64.233.160.0/19",
  "66.102.0.0/20",
  "66.249.80.0/20",
  "72.14.192.0/18",
  "74.125.0.0/16",
  "108.177.8.0/21",
  "173.194.0.0/16",
  "209.85.128.0/17",
  "216.58.192.0/19",
  "216.239.32.0/19",
  "2001:4860:4000::/36",
  "2404:6800:4000::/36",
  "2607:f8b0:4000::/36",
  "2800:3f0:4000::/36",
  "2a00:1450:4000::/36",
  "2c0f:fb50:4000::/36",
  "172.217.0.0/19",
  "172.217.32.0/20",
  "172.217.128.0/19",
  "172.217.160.0/20",
  "172.217.192.0/19",
  "172.253.56.0/21",
  "172.253.112.0/20",
  "108.177.96.0/19",
  "35.191.0.0/16",
  "130.211.0.0/22",
]

With the changes proposed it return:

data.google_netblock_ip_ranges.netblock: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

cidr_blocks = [
  "8.8.4.0/24",
  "8.8.8.0/24",
  "8.34.208.0/20",
  "8.35.192.0/20",
  "23.236.48.0/20",
  "23.251.128.0/19",
  "34.64.0.0/10",
  "35.184.0.0/13",
  "35.192.0.0/14",
  "35.196.0.0/15",
  "35.198.0.0/16",
  "35.199.0.0/17",
  "35.199.128.0/18",
  "35.200.0.0/13",
  "35.208.0.0/12",
  "35.224.0.0/12",
  "35.240.0.0/13",
  "64.15.112.0/20",
  "64.233.160.0/19",
  "66.102.0.0/20",
  "66.249.64.0/19",
  "70.32.128.0/19",
  "72.14.192.0/18",
  "74.114.24.0/21",
  "74.125.0.0/16",
  "104.154.0.0/15",
  "104.196.0.0/14",
  "104.237.160.0/19",
  "107.167.160.0/19",
  "107.178.192.0/18",
  "108.59.80.0/20",
  "108.170.192.0/18",
  "108.177.0.0/17",
  "130.211.0.0/16",
  "136.112.0.0/12",
  "142.250.0.0/15",
  "146.148.0.0/17",
  "162.216.148.0/22",
  "162.222.176.0/21",
  "172.110.32.0/21",
  "172.217.0.0/16",
  "172.253.0.0/16",
  "173.194.0.0/16",
  "173.255.112.0/20",
  "192.158.28.0/22",
  "192.178.0.0/15",
  "199.36.154.0/23",
  "199.36.156.0/24",
  "199.192.112.0/22",
  "199.223.232.0/21",
  "207.223.160.0/20",
  "208.65.152.0/22",
  "208.68.108.0/22",
  "208.81.188.0/22",
  "208.117.224.0/19",
  "209.85.128.0/17",
  "216.58.192.0/19",
  "216.73.80.0/20",
  "216.239.32.0/19",
  "2001:4860::/32",
  "2404:6800::/32",
  "2404:f340::/32",
  "2600:1900::/28",
  "2607:f8b0::/32",
  "2620:11a:a000::/40",
  "2620:120:e000::/40",
  "2800:3f0::/32",
  "2a00:1450::/32",
  "2c0f:fb50::/32",
]

The solution has been provided by the GCP support (networking team).

modular-magician · 2020-06-18T22:15:58Z

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@c2thorn, please review this PR or find an appropriate assignee.

modular-magician · 2020-06-18T22:21:28Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 1 file changed, 56 insertions(+), 6 deletions(-))
Terraform Beta: Diff ( 1 file changed, 56 insertions(+), 6 deletions(-))

modular-magician · 2020-06-18T22:21:37Z

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=121461"

modular-magician · 2020-06-19T09:02:02Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 57 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 2 files changed, 57 insertions(+), 7 deletions(-))

modular-magician · 2020-06-19T09:02:11Z

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=121663"

bcadiot · 2020-06-19T09:13:18Z

Without reading the data from CloudBuild it's difficult to understand where is the problem... The tests are running well locally.
Can someone copy/paste the data here? Thanks!

c2thorn · 2020-06-19T17:03:31Z

Hi @bcadiot thanks for the contribution, I will review this shortly.
For the build error, looks like our linter didn't like a few variables:

==> Checking source code against linters...
google/data_source_google_netblock_ip_ranges.go:20:2: structtag: struct field tag `json:ipv4Prefix` not compatible with reflect.StructTag.Get: bad syntax for struct tag value (govet)
	Ipv4Prefix string `json:ipv4Prefix`
	^
google/data_source_google_netblock_ip_ranges.go:21:2: structtag: struct field tag `json:ipv6Prefix` not compatible with reflect.StructTag.Get: bad syntax for struct tag value (govet)
	Ipv6Prefix string `json:ipv6Prefix`
	^
make: *** [lint] Error 1
GNUmakefile:26: recipe for target 'lint' failed

modular-magician · 2020-06-19T17:23:11Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 57 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 2 files changed, 57 insertions(+), 7 deletions(-))

modular-magician · 2020-06-19T17:23:19Z

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=121673"

c2thorn

LGTM, and verified locally. Thanks for this!

Fetch Google netblock from JSON

1a9be87

googlebot added the cla: yes label Jun 18, 2020

modular-magician requested a review from c2thorn June 18, 2020 22:15

Update docs link

0c84218

Add missing quotes

e452f3b

c2thorn approved these changes Jun 19, 2020

View reviewed changes

c2thorn merged commit d3ef0dc into GoogleCloudPlatform:master Jun 19, 2020

This was referenced Jun 19, 2020

Fetch Google netblock from JSON on gstatic domain hashicorp/terraform-provider-google#6650

Merged

Fetch Google netblock from JSON on gstatic domain hashicorp/terraform-provider-google-beta#2210

Merged

c2thorn mentioned this pull request Aug 11, 2020

Switch to cloud.json for cloud-netblocks hashicorp/terraform-provider-google#6996

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fetch Google netblock from JSON on gstatic domain #3689

Fetch Google netblock from JSON on gstatic domain #3689

bcadiot commented Jun 18, 2020 •

edited by c2thorn

Loading

modular-magician commented Jun 18, 2020

modular-magician commented Jun 18, 2020

modular-magician commented Jun 18, 2020

modular-magician commented Jun 19, 2020

modular-magician commented Jun 19, 2020

bcadiot commented Jun 19, 2020

c2thorn commented Jun 19, 2020

modular-magician commented Jun 19, 2020

modular-magician commented Jun 19, 2020

c2thorn left a comment

Fetch Google netblock from JSON on gstatic domain #3689

Fetch Google netblock from JSON on gstatic domain #3689

Conversation

bcadiot commented Jun 18, 2020 • edited by c2thorn Loading

modular-magician commented Jun 18, 2020

modular-magician commented Jun 18, 2020

Diff report:

modular-magician commented Jun 18, 2020

modular-magician commented Jun 19, 2020

Diff report:

modular-magician commented Jun 19, 2020

bcadiot commented Jun 19, 2020

c2thorn commented Jun 19, 2020

modular-magician commented Jun 19, 2020

Diff report:

modular-magician commented Jun 19, 2020

c2thorn left a comment

Choose a reason for hiding this comment

bcadiot commented Jun 18, 2020 •

edited by c2thorn

Loading