Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing sweeper(s): CloudIdsEndpoint #13375

Open
roaks3 opened this issue Jan 3, 2023 · 1 comment
Open

Failing sweeper(s): CloudIdsEndpoint #13375

roaks3 opened this issue Jan 3, 2023 · 1 comment
Labels
forward/linked service/ids size/s test-failure
Milestone
Near-Term Goals

Comments

@roaks3
Copy link
Collaborator

roaks3 commented Jan 3, 2023 •
edited by modular-magician
Loading

Affected Resource(s)

  • google_cloud_ids_endpoint

This sweeper was failing consistently with a 502, and because of the retry logic, it took about 5 minutes before failing and moving on to the next sweeper. This is a significant portion of our sweeper task, which times out after 11 minutes.

The 502 appears to come from the location parameter, which is assumed to take a region, but it actually only accepts a zone.

This sweeper should be fixed to properly handle the 502 so that we can re-add it. Note that the sweeper is being disabled while we work on a fix for this issue.

Message:

[04:11:53] :	 [Step 5/7] ---[ REQUEST ]---------------------------------------
[04:11:53] :	 [Step 5/7] GET /v1/projects/ci-test-project-188019/locations/us-central1/endpoints?alt=json HTTP/1.1
[04:11:53] :	 [Step 5/7] Host: ids.googleapis.com
[04:11:53] :	 [Step 5/7] Content-Type: application/json
[04:11:53] :	 [Step 5/7] Accept-Encoding: gzip
[04:11:53] :	 [Step 5/7] 
[04:11:53] :	 [Step 5/7] 
[04:11:53] :	 [Step 5/7] -----------------------------------------------------
[04:12:13] :	 [Step 5/7] 2023/01/03 04:12:13 [DEBUG] Google API Response Details:
[04:12:13] :	 [Step 5/7] ---[ RESPONSE ]--------------------------------------
[04:12:13] :	 [Step 5/7] HTTP/2.0 502 Bad Gateway
[04:12:13] :	 [Step 5/7] Content-Length: 1613
[04:12:13] :	 [Step 5/7] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
[04:12:13] :	 [Step 5/7] Content-Type: text/html; charset=UTF-8
[04:12:13] :	 [Step 5/7] Date: Tue, 03 Jan 2023 04:12:13 GMT
[04:12:13] :	 [Step 5/7] Referrer-Policy: no-referrer
[04:12:13] :	 [Step 5/7] 
[04:12:13] :	 [Step 5/7] <!DOCTYPE html>
[04:12:13] :	 [Step 5/7] <html lang=en>
[04:12:13] :	 [Step 5/7]   <meta charset=utf-8>
[04:12:13] :	 [Step 5/7]   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
[04:12:13] :	 [Step 5/7]   <title>Error 502 (Server Error)!!1</title>
[04:12:13] :	 [Step 5/7]   <style>
[04:12:13] :	 [Step 5/7]     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
[04:12:13] :	 [Step 5/7]   </style>
[04:12:13] :	 [Step 5/7]   <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
[04:12:13] :	 [Step 5/7]   <p><b>502.</b> <ins>That’s an error.</ins>
[04:12:13] :	 [Step 5/7]   <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>

...

[04:12:19] :	 [Step 5/7] 2023/01/03 04:12:19 [ERROR] WaitForState exceeded refresh grace period
[04:12:19] :	 [Step 5/7] 2023/01/03 04:12:19 [INFO][SWEEPER_LOG] Error in response from request https://ids.googleapis.com/v1/projects/ci-test-project-188019/locations/us-central1/endpoints: Get "https://ids.googleapis.com/v1/projects/ci-test-project-188019/locations/us-central1/endpoints?alt=json": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[04:12:19] :	 [Step 5/7] 2023/01/03 04:12:19 [DEBUG] Completed Sweeper (CloudIdsEndpoint) in region (us-central1) in 5m30.091551811s

Very similar to: #13338
Example build log: https://ci-oss.hashicorp.engineering/downloadBuildLog.html?buildId=364027&plain=true

b/299601139
@roaks3 roaks3 mentioned this issue Jan 3, 2023
Merged
5 tasks
This was referenced Jan 5, 2023
Merged
Merged
@rileykarson rileykarson added this to the Near-Term Goals milestone Jan 30, 2023
@github-actions github-actions bot added forward/review In review; remove label to forward service/ids labels Aug 17, 2023
@melinath melinath removed the forward/review In review; remove label to forward label Sep 7, 2023
@melinath
Copy link
Collaborator

melinath commented Sep 7, 2023 •
edited
Loading

Sweeper failures can cause quota issues over time, which can result in tests failing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
forward/linked service/ids size/s test-failure
Projects
None yet
Milestone
Near-Term Goals
Development

No branches or pull requests
4 participants
@melinath @rileykarson @roaks3 @modular-magician