Skip to content
This repository has been archived by the owner on Feb 22, 2023. It is now read-only.

Make translations more reliably present in all environments #2172

Merged
merged 4 commits into from
Feb 14, 2023
Merged

Make translations more reliably present in all environments #2172

merged 4 commits into from
Feb 14, 2023

Conversation

dhruvkb
Copy link
Member

@dhruvkb dhruvkb commented Feb 11, 2023 •
edited
Loading

Fixes

Fixes #2170 by @AetherUnbound

Description

This PR installs the axios-rate-limit module and sets up rate limiting for requests to GlotPress. This ensures that 429 errors are not raised and all translations are reliably downloaded.

This also adds logging in the get-translations.js script which prints the names of locales that fail to download.

Testing Instructions

  1. Check out the PR.
  2. Run pnpm i18n:get-translations with various rate limits in src/locales/scripts/axios.js. Very low limits like 10 or 20 will slow down the download considerably, whereas very high limits like 200 or 1000 will report large number of failures due to 429 errors. With the rate-limiting enabled and set to 99 (lower than 100 req/sec), no failures are recorded.

Example output when rate limit is set to 1000.

Successfully saved English translation to en.json.
Failed to fetch fr (429), he (429), hr (429), hsb (429), hy (429), ibo (429), jv (429), kaa (429), kab (429), kal (429), kin (429), kir (429), km (429), ko (429), lb (429), li (429), lij (429), lin (429), lmo (429), lo (429), lt (429), lv (429), mai (429), mfe (429), mg (429), mk (429), ml (429), mn (429), mr (429), mri (429), ms (429), mya (429), nb (429), nl-be (429), nl (429), nn (429), nqo (429), oci (429), ory (429), pa (429), pap-aw (429), pap-cw (429), pcd (429), pcm (429), pl (429), ps (429), pt-ao (429), pt-br (429), pt (429), rhg (429), ro (429), roh (429), ru (429), sa-in (429), sah (429), scn (429), si (429), sk (429), skr (429), sl (429), sna (429), snd (429), so (429), sq (429), sq-xk (429), sr (429), su (429), sv (429), sw (429), syr (429), szl (429), ta (429), ta-lk (429), tah (429), te (429), tg (429), th (429), tir (429), tl (429), tr (429), tt (429), tuk (429), twd (429), tzm (429), ug (429), uk (429), ur (429), uz (429), vec (429), vi (429), wol (429), xho (429), yor (429), zgh (429), zh-cn (429), zh-sg (429), zh-tw (429), zul (429)
Successfully saved 101 translations.

Checklist

  • My pull request has a descriptive title (not a vague title like
    Update index.md).
  • My pull request targets the default branch of the repository (main) or
    a parent feature branch.
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no visible
    errors.

Developer Certificate of Origin

Developer Certificate of Origin 
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@dhruvkb dhruvkb requested a review from a team as a code owner February 11, 2023 09:51
@dhruvkb dhruvkb requested review from zackkrida and obulat February 11, 2023 09:51
@github-actions
Copy link

github-actions bot commented Feb 11, 2023 •
edited
Loading

Storybook and Tailwind configuration previews: Ready

Storybook: https://wordpress.github.io/openverse-frontend/_preview/2172
Tailwind: https://wordpress.github.io/openverse-frontend/_preview/2172/tailwind

Please note that GitHub pages takes a little time to deploy newly pushed code, if the links above don't work or you see old versions, wait 5 minutes and try again.

You can check the GitHub pages deployment action list to see the current status of the deployments.

@openverse-bot openverse-bot added 🟧 priority: high Stalls work on the project or its dependents 🛠 goal: fix Bug fix ♿️ aspect: a11y Concerns related to the project's accessibility labels Feb 11, 2023
@github-actions
Copy link

github-actions bot commented Feb 11, 2023 •
edited
Loading

Size Change: -969 B (0%)

Total Size: 906 kB

Filename Size Change
./.nuxt/dist/client/233.js 0 B -272 B (removed) 🏆
./.nuxt/dist/client/233.modern.js 0 B -277 B (removed) 🏆
./.nuxt/dist/client/234.js 0 B -1.85 kB (removed) 🏆
./.nuxt/dist/client/app.js 151 kB -418 B (0%)
./.nuxt/dist/client/app.modern.js 124 kB -455 B (0%)
./.nuxt/dist/client/pages/audio/_id/index.modern.js 4.84 kB -20 B (0%)
./.nuxt/dist/client/pages/image/_id/index.modern.js 7.41 kB -36 B (0%)
./.nuxt/dist/client/runtime.js 2.7 kB -22 B (-1%)
./.nuxt/dist/client/runtime.modern.js 2.7 kB -21 B (-1%)
./.nuxt/dist/client/229.js 272 B +272 B (new file) 🆕
./.nuxt/dist/client/229.modern.js 277 B +277 B (new file) 🆕
./.nuxt/dist/client/230.js 1.85 kB +1.85 kB (new file) 🆕
ℹ️ View Unchanged
Filename Size Change
./.nuxt/dist/client/commons/app.js 87.7 kB 0 B
./.nuxt/dist/client/commons/app.modern.js 78.2 kB 0 B
./.nuxt/dist/client/components/loading-icon.js 748 B 0 B
./.nuxt/dist/client/components/loading-icon.modern.js 749 B 0 B
./.nuxt/dist/client/components/table-sort-icon.js 508 B 0 B
./.nuxt/dist/client/components/table-sort-icon.modern.js 513 B 0 B
./.nuxt/dist/client/components/v-all-results-grid.js 7.54 kB 0 B
./.nuxt/dist/client/components/v-all-results-grid.modern.js 5.05 kB 0 B
./.nuxt/dist/client/components/v-audio-cell.js 384 B 0 B
./.nuxt/dist/client/components/v-audio-cell.modern.js 393 B 0 B
./.nuxt/dist/client/components/v-audio-details.js 2.54 kB 0 B
./.nuxt/dist/client/components/v-audio-details.modern.js 1.79 kB 0 B
./.nuxt/dist/client/components/v-audio-track-skeleton.js 1.01 kB 0 B
./.nuxt/dist/client/components/v-audio-track-skeleton.modern.js 1.02 kB 0 B
./.nuxt/dist/client/components/v-audio-track.js 5.27 kB 0 B
./.nuxt/dist/client/components/v-audio-track.modern.js 5.22 kB 0 B
./.nuxt/dist/client/components/v-back-to-search-results-link.js 538 B 0 B
./.nuxt/dist/client/components/v-back-to-search-results-link.modern.js 543 B 0 B
./.nuxt/dist/client/components/v-bone.js 685 B 0 B
./.nuxt/dist/client/components/v-bone.modern.js 689 B 0 B
./.nuxt/dist/client/components/v-box-layout.js 1.23 kB 0 B
./.nuxt/dist/client/components/v-box-layout.modern.js 1.23 kB 0 B
./.nuxt/dist/client/components/v-content-link.js 1.11 kB 0 B
./.nuxt/dist/client/components/v-content-link.modern.js 1.09 kB 0 B
./.nuxt/dist/client/components/v-content-page.js 466 B 0 B
./.nuxt/dist/client/components/v-content-page.modern.js 471 B 0 B
./.nuxt/dist/client/components/v-content-report-button.js 774 B 0 B
./.nuxt/dist/client/components/v-content-report-button.modern.js 781 B 0 B
./.nuxt/dist/client/components/v-content-report-form.js 6.1 kB 0 B
./.nuxt/dist/client/components/v-content-report-form.modern.js 3.58 kB 0 B
./.nuxt/dist/client/components/v-content-report-popover.js 1.23 kB 0 B
./.nuxt/dist/client/components/v-content-report-popover.modern.js 4.23 kB 0 B
./.nuxt/dist/client/components/v-copy-button.js 3.99 kB 0 B
./.nuxt/dist/client/components/v-copy-button.modern.js 3.99 kB 0 B
./.nuxt/dist/client/components/v-copy-license.js 1 kB 0 B
./.nuxt/dist/client/components/v-copy-license.modern.js 1 kB 0 B
./.nuxt/dist/client/components/v-copy-license/components/v-error-image/components/v-media-reuse/components/v-search-grid/d219393b.js 9.88 kB +5 B (0%)
./.nuxt/dist/client/components/v-copy-license/components/v-error-image/components/v-media-reuse/components/v-search-grid/d219393b.modern.js 9.85 kB +4 B (0%)
./.nuxt/dist/client/components/v-dmca-notice.js 744 B 0 B
./.nuxt/dist/client/components/v-dmca-notice.modern.js 751 B 0 B
./.nuxt/dist/client/components/v-error-image.js 1.69 kB 0 B
./.nuxt/dist/client/components/v-error-image.modern.js 1.68 kB 0 B
./.nuxt/dist/client/components/v-error-section.js 373 B 0 B
./.nuxt/dist/client/components/v-error-section.modern.js 376 B 0 B
./.nuxt/dist/client/components/v-external-search-form.js 2.06 kB 0 B
./.nuxt/dist/client/components/v-external-search-form.modern.js 2.05 kB 0 B
./.nuxt/dist/client/components/v-external-source-list.js 901 B 0 B
./.nuxt/dist/client/components/v-external-source-list.modern.js 899 B 0 B
./.nuxt/dist/client/components/v-full-layout.js 1.59 kB 0 B
./.nuxt/dist/client/components/v-full-layout.modern.js 1.59 kB 0 B
./.nuxt/dist/client/components/v-grid-skeleton.js 1.62 kB 0 B
./.nuxt/dist/client/components/v-grid-skeleton.modern.js 1.62 kB 0 B
./.nuxt/dist/client/components/v-home-gallery.js 4.8 kB 0 B
./.nuxt/dist/client/components/v-home-gallery.modern.js 4.77 kB 0 B
./.nuxt/dist/client/components/v-homepage-content.js 1.72 kB 0 B
./.nuxt/dist/client/components/v-homepage-content.modern.js 1.69 kB 0 B
./.nuxt/dist/client/components/v-image-carousel.js 4.77 kB 0 B
./.nuxt/dist/client/components/v-image-carousel.modern.js 4.73 kB 0 B
./.nuxt/dist/client/components/v-image-cell-square.js 1.02 kB 0 B
./.nuxt/dist/client/components/v-image-cell-square.modern.js 1.02 kB 0 B
./.nuxt/dist/client/components/v-image-cell.js 1.45 kB 0 B
./.nuxt/dist/client/components/v-image-cell.modern.js 1.44 kB 0 B
./.nuxt/dist/client/components/v-image-details.js 2.15 kB 0 B
./.nuxt/dist/client/components/v-image-details.modern.js 1.42 kB 0 B
./.nuxt/dist/client/components/v-image-grid.js 4.93 kB 0 B
./.nuxt/dist/client/components/v-image-grid.modern.js 2.47 kB 0 B
./.nuxt/dist/client/components/v-license-tab-panel.js 521 B 0 B
./.nuxt/dist/client/components/v-license-tab-panel.modern.js 525 B 0 B
./.nuxt/dist/client/components/v-load-more.js 3.16 kB 0 B
./.nuxt/dist/client/components/v-load-more.modern.js 684 B 0 B
./.nuxt/dist/client/components/v-media-license.js 819 B 0 B
./.nuxt/dist/client/components/v-media-license.modern.js 827 B 0 B
./.nuxt/dist/client/components/v-media-reuse.js 1.62 kB 0 B
./.nuxt/dist/client/components/v-media-reuse.modern.js 1.61 kB 0 B
./.nuxt/dist/client/components/v-media-tag.js 428 B 0 B
./.nuxt/dist/client/components/v-media-tag.modern.js 434 B 0 B
./.nuxt/dist/client/components/v-no-results.js 752 B 0 B
./.nuxt/dist/client/components/v-no-results.modern.js 751 B 0 B
./.nuxt/dist/client/components/v-old-homepage-content.js 1.88 kB 0 B
./.nuxt/dist/client/components/v-old-homepage-content.modern.js 1.85 kB 0 B
./.nuxt/dist/client/components/v-radio.js 1.51 kB 0 B
./.nuxt/dist/client/components/v-radio.modern.js 1.47 kB 0 B
./.nuxt/dist/client/components/v-related-audio.js 1.25 kB 0 B
./.nuxt/dist/client/components/v-related-audio.modern.js 1.25 kB 0 B
./.nuxt/dist/client/components/v-related-images.js 1.05 kB 0 B
./.nuxt/dist/client/components/v-related-images.modern.js 3.03 kB 0 B
./.nuxt/dist/client/components/v-report-desc-form.js 965 B 0 B
./.nuxt/dist/client/components/v-report-desc-form.modern.js 968 B 0 B
./.nuxt/dist/client/components/v-row-layout.js 1.7 kB 0 B
./.nuxt/dist/client/components/v-row-layout.modern.js 1.71 kB 0 B
./.nuxt/dist/client/components/v-scroll-button.js 813 B 0 B
./.nuxt/dist/client/components/v-scroll-button.modern.js 819 B 0 B
./.nuxt/dist/client/components/v-search-grid.js 5.87 kB 0 B
./.nuxt/dist/client/components/v-search-grid.modern.js 5.8 kB 0 B
./.nuxt/dist/client/components/v-search-results-title.js 596 B 0 B
./.nuxt/dist/client/components/v-search-results-title.modern.js 599 B 0 B
./.nuxt/dist/client/components/v-search-type-radio.js 791 B 0 B
./.nuxt/dist/client/components/v-search-type-radio.modern.js 767 B 0 B
./.nuxt/dist/client/components/v-server-timeout.js 299 B 0 B
./.nuxt/dist/client/components/v-server-timeout.modern.js 303 B 0 B
./.nuxt/dist/client/components/v-sketch-fab-viewer.js 3.37 kB 0 B
./.nuxt/dist/client/components/v-sketch-fab-viewer.modern.js 894 B 0 B
./.nuxt/dist/client/components/v-skip-to-content-container.js 887 B 0 B
./.nuxt/dist/client/components/v-skip-to-content-container.modern.js 896 B 0 B
./.nuxt/dist/client/components/v-snackbar.js 1.18 kB 0 B
./.nuxt/dist/client/components/v-snackbar.modern.js 1.19 kB 0 B
./.nuxt/dist/client/components/v-sources-table.js 16.6 kB 0 B
./.nuxt/dist/client/components/v-sources-table.modern.js 16.6 kB 0 B
./.nuxt/dist/client/components/v-warning-suppressor.js 298 B 0 B
./.nuxt/dist/client/components/v-warning-suppressor.modern.js 303 B 0 B
./.nuxt/dist/client/pages/about.js 1.52 kB 0 B
./.nuxt/dist/client/pages/about.modern.js 1.53 kB 0 B
./.nuxt/dist/client/pages/audio/_id/index.js 8.01 kB -1 B (0%)
./.nuxt/dist/client/pages/external-sources.js 1.54 kB +1 B (0%)
./.nuxt/dist/client/pages/external-sources.modern.js 1.54 kB -1 B (0%)
./.nuxt/dist/client/pages/feedback.js 1.32 kB +1 B (0%)
./.nuxt/dist/client/pages/feedback.modern.js 1.32 kB 0 B
./.nuxt/dist/client/pages/image/_id/index.js 9.32 kB 0 B
./.nuxt/dist/client/pages/image/_id/report.js 3.59 kB 0 B
./.nuxt/dist/client/pages/image/_id/report.modern.js 4.24 kB -1 B (0%)
./.nuxt/dist/client/pages/index.js 8.65 kB -2 B (0%)
./.nuxt/dist/client/pages/index.modern.js 8.51 kB -2 B (0%)
./.nuxt/dist/client/pages/preferences.js 1.23 kB +1 B (0%)
./.nuxt/dist/client/pages/preferences.modern.js 1.22 kB 0 B
./.nuxt/dist/client/pages/privacy.js 995 B +1 B (0%)
./.nuxt/dist/client/pages/privacy.modern.js 995 B 0 B
./.nuxt/dist/client/pages/search-help.js 1.62 kB 0 B
./.nuxt/dist/client/pages/search-help.modern.js 1.62 kB +2 B (0%)
./.nuxt/dist/client/pages/search.js 5.18 kB 0 B
./.nuxt/dist/client/pages/search.modern.js 2.67 kB -1 B (0%)
./.nuxt/dist/client/pages/search/audio.js 6.13 kB +1 B (0%)
./.nuxt/dist/client/pages/search/audio.modern.js 3.63 kB -1 B (0%)
./.nuxt/dist/client/pages/search/image.js 595 B -1 B (0%)
./.nuxt/dist/client/pages/search/image.modern.js 2.72 kB +1 B (0%)
./.nuxt/dist/client/pages/search/index.js 480 B 0 B
./.nuxt/dist/client/pages/search/index.modern.js 486 B 0 B
./.nuxt/dist/client/pages/search/model-3d.js 242 B -1 B (0%)
./.nuxt/dist/client/pages/search/model-3d.modern.js 246 B 0 B
./.nuxt/dist/client/pages/search/search-page.types.js 266 B +1 B (0%)
./.nuxt/dist/client/pages/search/search-page.types.modern.js 271 B 0 B
./.nuxt/dist/client/pages/search/video.js 240 B 0 B
./.nuxt/dist/client/pages/search/video.modern.js 244 B +1 B (0%)
./.nuxt/dist/client/pages/sources.js 1.56 kB +1 B (0%)
./.nuxt/dist/client/pages/sources.modern.js 1.55 kB -5 B (0%)
./.nuxt/dist/client/vendors/app.js 63.8 kB 0 B
./.nuxt/dist/client/vendors/app.modern.js 63.1 kB 0 B

compressed-size-action

obulat
obulat approved these changes Feb 13, 2023
View reviewed changes
Copy link
Contributor

@obulat obulat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great find!

zackkrida
zackkrida approved these changes Feb 14, 2023
View reviewed changes
Copy link
Member

@zackkrida zackkrida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. How was 99 chosen for the rate limiting. Is that a GlotPress value documented somewhere?

@dhruvkb
Copy link
Member Author

dhruvkb commented Feb 14, 2023

It was trial and error tbh, I noticed that it was only downloading about a 100 languages and 429-ing for the rest. I searched for official rate-limiting documentation but couldn't find any.

@dhruvkb @zackkrida
Update src/locales/scripts/get-translations.js
b1075ac 
Co-authored-by: Zack Krida <zackkrida@pm.me>
@github-actions
Copy link

Playwright Failure Test Results

It looks like some of the Playwright tests failed. You can download the Playwright trace
output for both Storybook and Nuxt Playwright tests that have failed at the bottom of this
page, under the "Artifacts" section:

https://github.com/WordPress/openverse-frontend/actions/runs/4174618643

Read more about how to use this artifact here: https://github.com/WordPress/openverse-frontend/blob/main/test/playwright/README.md#debugging

@dhruvkb dhruvkb merged commit ec68839 into main Feb 14, 2023
@dhruvkb dhruvkb deleted the ensure_translations branch February 14, 2023 14:31
github-actions bot pushed a commit that referenced this pull request Feb 14, 2023
@openverse-bot
Clean preview for PR #2172
3f7176e
This was referenced Feb 14, 2023
Merged
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@zackkrida zackkrida zackkrida approved these changes

@obulat obulat obulat approved these changes

Assignees
No one assigned
Labels
♿️ aspect: a11y Concerns related to the project's accessibility 🛠 goal: fix Bug fix 🟧 priority: high Stalls work on the project or its dependents
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Not all locales show up in picker
4 participants
@dhruvkb @zackkrida @obulat @openverse-bot