Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FDS-2497] Wrap google API execute calls with a 5 attempt retry #1513

Merged
merged 4 commits into from
Oct 15, 2024
Merged

[FDS-2497] Wrap google API execute calls with a 5 attempt retry #1513

merged 4 commits into from
Oct 15, 2024

Conversation

BryanFauble
Copy link
Collaborator

Problem:

  1. Errors like the following kept popping up:
 Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.15/x64/bin/schematic", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/schematic/manifest/commands.py", line 251, in get_manifest
    result = create_single_manifest(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/schematic/manifest/commands.py", line 201, in create_single_manifest
    result = manifest_generator.get_manifest(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/schematic/manifest/generator.py", line 1789, in get_manifest
    manifest_url = self.get_empty_manifest(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/schematic/manifest/generator.py", line 1320, in get_empty_manifest
    manifest_url = self._create_empty_gs(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/schematic/manifest/generator.py", line 1245, in _create_empty_gs
    requests_body = self._create_requests_body(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/schematic/manifest/generator.py", line 1187, in _create_requests_body
    create_dropdown = self._request_dropdown(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/schematic/manifest/generator.py", line 991, in _request_dropdown
    validation_body = self._get_column_data_validation_values(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/schematic/manifest/generator.py", line 328, in _get_column_data_validation_values
    valid_values = self._store_valid_values_as_data_dictionary(
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/schematic/manifest/generator.py", line 309, in _store_valid_values_as_data_dictionary
    .execute()
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 502 when requesting https://sheets.googleapis.com/v4/spreadsheets/1hLLgWh8dEQHAn-IZQDGR3YbrL_SFMIIky2B3XJuplys/values/Sheet2%21F2%3AF8?valueInputOption=RAW&alt=json returned "Bad Gateway". Details: "<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    ****margin:0;padding:0***html,code***font:15px/22px arial,sans-serif***html***background:#fff;color:#222;padding:15px***body***margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px**** > body***background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px***p***margin:11px 0 22px;overflow:hidden***ins***color:#777;text-decoration:none***a img***border:0***@media screen and (max-width:772px)***body***background:none;margin-top:0;max-width:none;padding-right:0***#logo***background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px***@media only screen and (min-resolution:192dpi)***#logo***background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0***@media only screen and (-webkit-min-device-pixel-ratio:2)***#logo***background:url(//www.google
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>
">

Solution:

  1. Grabbing the retry logic used in other parts of the code to wrap the google API calls with retry.

Testing:

  1. Relying on the existing tests that are hitting this logic. I also ran the code locally to verify the test.

@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Oct 4, 2024

Quality Gate Passed Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
94.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

thomasyu888
thomasyu888 approved these changes Oct 4, 2024
View reviewed changes
Copy link
Member

@thomasyu888 thomasyu888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 LGTM! Lets wait for the develop endpoint to be deployed before the rest of the team reviews, but I'm going to pre-approve.

I believe this and #1505 will be the PR's that will resolve lots of the disconnect issues that I've been seeing

@thomasyu888 thomasyu888 changed the title Wrap google API execute calls with a 5 attempt retry [FDS-2497] Wrap google API execute calls with a 5 attempt retry Oct 4, 2024
@linglp
Copy link
Contributor

linglp commented Oct 14, 2024

@BryanFauble Could you share a way that we could reproduce the error that you put in the description? And we could add a test to verify that the solution works?

@BryanFauble
Copy link
Collaborator Author

@linglp

Could you share a way that we could reproduce the error that you put in the description? And we could add a test to verify that the solution works?

There is no way to reliable re-create this issue because it depends on the google API servers failing to serve the request. By virtue of us getting less failures during our integration tests we can implicitly see that it is working.

While I was writing this I had also semi-regularly gotten these failures, but once the code was in place I didn't get a single one.

@linglp
Copy link
Contributor

linglp commented Oct 15, 2024

@BryanFauble thanks, that makes sense. I will approve for now. Thanks for your PR.

@linglp linglp self-requested a review October 15, 2024 17:11
@BryanFauble BryanFauble merged commit 49c701c into develop Oct 15, 2024
7 checks passed
@BryanFauble BryanFauble deleted the wrap-google-calls-with-retry branch October 15, 2024 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@linglp linglp linglp approved these changes

@thomasyu888 thomasyu888 thomasyu888 approved these changes

@andrewelamb andrewelamb Awaiting requested review from andrewelamb

@GiaJordan GiaJordan Awaiting requested review from GiaJordan

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@BryanFauble @linglp @thomasyu888