feat: use atlas in make pull_translations

[OmarIthawi]

Changes

• Add new ATLAS_PULL_LANGS variable similar to the deprecated transifex_langs variable in other repos.
• Add atlas into make pull_translations when OPENEDX_ATLAS_PULL environment variable is set.

Testing

Mostly the following commands needs to be run, I won't go into super detailed steps though:

# Rebuild the image and restart the devstack
course-discovery $ docker build --target=dev -t edxops/discovery-dev:latest .

# Then run the pull command with `static.dev`
devstack $ make dev.shell.discovery
discovery $ OPENEDX_ATLAS_PULL=yes make pull_translations static.dev

• Test pulled translations

  • Deletes the existing language
  • Pulls new language files from the the openedx-translations repo: https://github.com/openedx/openedx-translations/tree/main/translations/course-discovery/course_discovery/conf/locale (See latest comments for testing log)

• Quick QA for translated content (source po file):
  

References

This pull request is part of the FC-0012 project which is sparked by the Translation Infrastructure update OEP-58.

Join the conversation on Open edX Slack #translations-project-fc-0012.

Check the links above for full information about the overall project.

Internalization is being rearchitected in Open edX Python, XBlock, Micro-frontend, and other projects. There are a number of immediately visible changes:

• Remove source and language translations from the repositories, hence no .json, .po or .mo files will be committed into the repos.
• Add standardized make extract_translations in all repositories
• Push user-facing messages strings into openedx/openedx-translations.
• Integrate root repositories with openedx/openedx-atlas to pull translations on build/deploy time

Breaking Changes

One of the primary goals of the project is to avoid breaking changes. If you noticed any suspicious code, please raise your concern. But before that, please know the strategy we're following to avoid breaking changes:

For XBlocks and Python projects:

• The standard translation path must be conf/locale.
• At the end of the project, a clear change log will be added and all translations will be handled by Atlas. Thus, the local translation will be removed from the repo.
• A notification for the community will be published to ensure that everyone knows why translations directories are removed from all repos.

[openedx-webhooks]

Thanks for the pull request, @OmarIthawi! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[brian-smith-tcril]

@DawoudSheraz I requested a review from you on this because you are listed as "Review required from" on the spreadsheet. Thanks in advance for taking a look!

[OmarIthawi]

Thanks @DawoudSheraz and Brian.

Let's hold on a bit before merging. I might need to refactor the environment variables first.

[OmarIthawi]

I'm more confident now that this works. I've tested it in devstack and it looks good to me.

@brian-smith-tcril and @DawoudSheraz would you mind taking another look and merging if it looks good?

Testing results

devstack $ make dev.remove-containers
devstack $ make dev.up.discovery
$ make dev.shell.discovery
docker-compose exec discovery env TERM=xterm-256color bash -c '/bin/bash'
root@discovery:/edx/app/discovery/discovery# atlas --version
v0.4.4

devstack $ docker-compose run discovery bash -c 'OPENEDX_ATLAS_PULL=yes make pull_translations && ls -R course_discovery/conf'
Creating devstack_discovery_run ... done
find course_discovery/conf/locale -mindepth 1 -maxdepth 1 -type d -exec rm -r {} \;
atlas pull  translations/course-discovery/course_discovery/conf/locale:course_discovery/conf/locale
Pulling translation files
 - directory: translations/course-discovery/course_discovery/conf/locale:course_discovery/conf/locale
 - repository: openedx/openedx-translations
 - branch: main
 - filter: Not Specified
Creating a temporary Git repository to pull translations into "./translations_TEMP"...
Done.
Setting git sparse-checkout rules...
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 8 (delta 6), reused 5 (delta 4), pack-reused 0
Receiving objects: 100% (8/8), 52.13 KiB | 1.06 MiB/s, done.
Resolving deltas: 100% (6/6), done.
Done.
Pulling translation files from the repository...
Your branch is up to date with 'origin/main'.
Done.
Copying translations from "./translations_TEMP/translations/course-discovery/course_discovery/conf/locale" to "./course_discovery/conf/locale"...
Done.
Removing temporary directory...
Done.

Translations pulled successfully!
python manage.py compilemessages
2023-08-18 09:25:38,282 INFO 195 [algoliasearch_django.registration] /edx/app/discovery/venvs/discovery/lib/python3.8/site-packages/algoliasearch_django/registration.py:77 - REGISTER <class 'course_discovery.apps.course_metadata.algolia_models.AlgoliaProxyProduct'>
processing file django.po in /edx/app/discovery/discovery/course_discovery/conf/locale/ar/LC_MESSAGES
processing file djangojs.po in /edx/app/discovery/discovery/course_discovery/conf/locale/ar/LC_MESSAGES
processing file django.po in /edx/app/discovery/discovery/course_discovery/conf/locale/en/LC_MESSAGES
processing file djangojs.po in /edx/app/discovery/discovery/course_discovery/conf/locale/en/LC_MESSAGES
processing file django.po in /edx/app/discovery/discovery/course_discovery/conf/locale/de/LC_MESSAGES
processing file djangojs.po in /edx/app/discovery/discovery/course_discovery/conf/locale/de/LC_MESSAGES
processing file django.po in /edx/app/discovery/discovery/course_discovery/conf/locale/fr_CA/LC_MESSAGES
processing file djangojs.po in /edx/app/discovery/discovery/course_discovery/conf/locale/fr_CA/LC_MESSAGES
course_discovery/conf:
locale

course_discovery/conf/locale:
ar  config.yaml  de  en  fr_CA

course_discovery/conf/locale/ar:
LC_MESSAGES

course_discovery/conf/locale/ar/LC_MESSAGES:
djangojs.mo  djangojs.po  django.mo  django.po

course_discovery/conf/locale/de:
LC_MESSAGES

course_discovery/conf/locale/de/LC_MESSAGES:
djangojs.mo  djangojs.po  django.mo  django.po

course_discovery/conf/locale/en:
LC_MESSAGES

course_discovery/conf/locale/en/LC_MESSAGES:
djangojs.mo  djangojs.po  django.mo  django.po

course_discovery/conf/locale/fr_CA:
LC_MESSAGES

course_discovery/conf/locale/fr_CA/LC_MESSAGES:
djangojs.mo  djangojs.po  django.mo  django.po

[OmarIthawi]

Used the pip-tools --upgrade-package=openedx-atlas to upgrade this requirement and avoid getting potentially stuck with others.

[openedx-webhooks]

@OmarIthawi 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.