feat: fetching of a secured Algolia key [BB-8083]

[0x29a]

Description

Adds a fallback mechanism, so when the ALGOLIA_SEARCH_API_KEY environment variable is not set, the key is fetched from the endpoint defined by the ALGOLIA_SECURED_KEY_ENDPOINT environment variable. It expects this endpoint to return a JSON like:

{
  "key": "<ALGOLIA_SEARCH_API_KEY>"
}

This is intended, but not limited to, to be used with openedx/edx-enterprise#1962.

Testing steps

We're going to test this and openedx/edx-enterprise#1962 PRs at once by creating three courses and three enterprise customers, uploading them to Algolia through enterprise-catalog and checking that learners are able to browse catalogs of their enterprises with the help of this MFE, but unable to access courses of enterprises they don't belong to.

Prerequisites

You'll need an Algolia account. Create an application and enterprise-catalog index. Write down the application id, admin API key, and search API key somewhere.

Installing master devstack

mkdir master && cd master
export DEVSTACK_WORKSPACE=$(pwd)
git clone git@github.com:openedx/devstack.git && cd devstack
virtualenv -p python3 .fenv
source .fenv/bin/activate
make requirements
make dev.clone
make dev.provision
make dev.up.large-and-slow

After that:

1. cd ../edx-platform
2. Add ENTERPRISE_ALGOLIA_SEARCH_API_KEY = '<YOUR_ALGOLIA_SEARCH_(NOT ADMIN)_API_KEY>' to the end of lms/envs/devstack.py.

Installing edx-enterprise fork

sudo chown $USER:$USER ../src
cd ../src
git clone -b 0x29a/bb8083/per-user-algolia-key git@github.com:open-craft/edx-enterprise.git
cd ../devstack
make lms-shell
cd /edx/src/edx-enterprise/
pip uninstall edx-enterprise
pip install -e .
exit
make lms-shell
python manage.py lms migrate
exit
make cms-shell
cd /edx/src/edx-enterprise/
pip uninstall edx-enterprise
pip install -e .
exit
make lms-restart
make cms-restart

Installing enterprise-catalog

cd ..
git clone git@github.com:openedx/enterprise-catalog.git && cd enterprise-catalog

After that:

1. Open enterprise_catalog/apps/catalog/constants.py and add 'source' to the CONTENT_PRODUCT_SOURCE_ALLOW_LIST set.
2. Add the following to the end of enterprise_catalog/settings/devstack.py:

   ALGOLIA = {
       'INDEX_NAME': 'enterprise-catalog',
       'APPLICATION_ID': '<YOUR_ALGOLIA_APPLICATION_ID>',
       'API_KEY': '<YOUR_ADMIN_ALGOLIA_API_KEY>',
   }

3. Run make dev.provision.

Installing frontend-app-learner-portal-enterprise

cd ..
git clone -b 0x29a/bb8083/per-user-algolia-key git@github.com:open-craft/frontend-app-learner-portal-enterprise.git && cd frontend-app-learner-portal-enterprise
export ALGOLIA_APP_ID=<YOUR_ALGOLIA_APPLICATION_ID>
export ALGOLIA_INDEX_NAME=enterprise-catalog
nvm use
npm install
npm start

Leave this terminal open, do all the remaining steps in a separate one.

Creating test entities

1. Create three courses here: http://localhost:18010/home/. Use ABC, XYZ, and PSY for all fields, like here:
   
2. Add a section with a dropdown unit to each course, publish each.
3. Set Course Start Date and Enrollment Start Date for each course here to 01/01/2019.
4. Add all three courses here, like this:
   
5. Create a source with a Source name here: http://localhost:18381/admin/course_metadata/source/add/
6. Go here and create three queries with the following content filters for each course:

   {
       "content_type":[
           "course",
           "courserun"
       ],
       "aggregation_key":[
           "course:ABC+ABC",
           "courserun:ABC+ABC"
       ],
       "partner":"edx",
       "availability":[
           "Current",
           "Starting Soon",
           "Upcoming"
       ],
       "status":"published"
   }

   Change ABC to XYZ and PSY for each query.
7. Go here and create three enterprise customers: XXX, YYY, ZZZ. Change their slugs respectively.
8. Go here and create three catalogs for each customer selecting different course queries we created earlier. XXX should be assigned to the query selecting ABC course, YYY -> XYZ and ZZZ -> PSY.

Indexing

Go to the <DEVSTACK_WORKSPACE> and replace in course_discovery/settings/base.py this:

DEFAULT_PRODUCT_SOURCE_NAME = 'edX'
DEFAULT_PRODUCT_SOURCE_SLUG = 'edx'

with this:

DEFAULT_PRODUCT_SOURCE_NAME = 'Source'
DEFAULT_PRODUCT_SOURCE_SLUG = 'source'

Then, go to the <DEVSTACK_WORKSPACE>/devstack directory run the following to pull the data from LMS to Course Discovery:

make discovery-shell
./manage.py refresh_course_metadata

Then:

1. Go here and set a name for each organization.
2. Go here and change every course run to be "Published"
3. Run the following in the discovery shell to fill in the product source for each course and update the ES index:

./manage.py populate_default_product_source
./manage.py update_index --disable-change-limit
exit

Now we need to copy course queries and other enterprise-related data from LMS to enterprise-catalog. Run the following:

make lms-shell
./manage.py lms migrate_enterprise_catalogs --api_user enterprise_catalog_worker

After that you should see three of your course queries here.

Now, go to the <DEVSTACK_WORKSPACE>/enterprise-catalog and run the following to fetch content metadata (courses and course runs) from Course Discovery:

make app-shell
./manage.py update_content_metadata --force

After that you should see Course and Course Run objects for each of your courses here. If you don't, then something went wrong. Also, Json metadata for each course should contain a non-empty advertised_course_run_uuid.

Now you can upload all courses to Algolia:

./manage.py reindex_algolia

If everything go fine, you should see three courses in your enterprise-catalog Algolia index.

Enrolling users

1. Register a new user: abc@example.com.
2. Go the XXX enterprise customer and click Manager Learners (example of the admin panel URL).
3. Now, enroll abc@example.com user to the course-v1:ABC+ABC+ABC course. Specify Audit track and test reason for manual enrollment.
4. Repeat points 2 and 3, but for YYY enterprise customer and course-v1:XYZ+XYZ+XYZ course, so we have a single learner present in two different enterprises.

Testing frontend-app-learner-portal-enterprise

1. Sign out as abc@example.com.
2. Go to http://localhost:8734.
3. You should be redirected to the LMS.
4. Upon signing in again as abc@example.com, you should be offered to choose an organization. Choose XXX.
5. You should be redirected to the MFE. You shoud see the ABC course here.
6. Now, if you visit this URL again and select YYY, you should see the XYZ course.

Now, open the browser dev tools and locate a query like this:

https://ql3i9veweo-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia for JavaScript (4.6.0); Browser; JS Helper (3.12.0); react (17.0.2); react-instantsearch (6.38.1)

1. Click the right mouse button on it and Edit and Resend (that's for Firefox).
2. Paste the following JSON to the Body field and click Send:

   {
     "requests": [
       {
         "indexName": "enterprise-catalog",
         "params": "facetingAfterDistinct=true&facets=%5B%5D&highlightPostTag=%3C%2Fais-highlight-0000000000%3E&highlightPreTag=%3Cais-highlight-0000000000%3E&tagFilters="
       }
     ]
   }

   Normally, this request would fetch all courses from the index. But the MFE is using our secured Algolia key now.
3. Observe what this query returned. There should be only 2 hits, for the ABC and XYZ courses, omitting the PSY course.

To test the default MFE behavior, try setting the ALGOLIA_SEARCH_API_KEY environment variable, re-starting the npm server and testing that the MFE is pulling all courses from the index.

[openedx-webhooks]

Thanks for the pull request, @0x29a!

What's next?

Please work through the following steps to get your changes ready for engineering review:

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Let us know that your PR is ready for review:

Who will review my changes?

This repository is currently unmaintained.

To get help with finding a technical reviewer, tag the community contributions project manager for this PR in a comment and let them know that your changes are ready for review:

1. On the right-hand side of the PR, find the Contributions project, click the caret in the top right corner to expand it, and check the "Primary PM" field for the name of your PM.
2. Find their GitHub handle here.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[Agrendalath]

👍

• I tested this: in feat: fetching of a secured Algolia key [BB-8083] open-craft/frontend-app-learner-portal-enterprise#1, which matches the code from this PR (this PR has an additional replacement of the duplicated code in the AcademyDetailPage)
• I read through the code
• I checked for accessibility issues: n/a
• Includes documentation: n/a

[adamstankiewicz]

[inform] We've been starting to use @tanstack/react-query's useQuery hook to make API GET requests moving forward, as it removes a fair amount of boilerplate (e.g., loading states, useEffect, etc.), the need for custom client-side caching support, etc.

[0x29a]

Thank you for letting me know, @adamstankiewicz. Would you like me to re-write this using useQuery, or it's fine as is?

[e0d]

@0x29a I've run the tests, but looks like there are some conflicts that need attention.

[codecov]

Codecov Report

Attention: Patch coverage is 79.31034% with 6 lines in your changes missing coverage. Please review.

Project coverage is 84.97%. Comparing base (d36b78a) to head (52c8526).
Report is 343 commits behind head on master.

Files with missing lines	Patch %	Lines
src/utils/common.js	70.00%	2 Missing and 1 partial ⚠️
src/index.jsx	0.00%	2 Missing ⚠️
src/components/search/Search.jsx	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #887      +/-   ##
==========================================
+ Coverage   84.87%   84.97%   +0.10%     
==========================================
  Files         320      338      +18     
  Lines        6399     7102     +703     
  Branches     1552     1739     +187     
==========================================
+ Hits         5431     6035     +604     
- Misses        941     1035      +94     
- Partials       27       32       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[0x29a]

@e0d, thanks. I rebased the branch and fixed all linting & tests issues.

[mphilbrick211]

Hi @0x29a! Flagging that some branch conflicts have popped up for when you have a minute. Thanks!

[0x29a]

Hi @mphilbrick211, unless the base branch gets some major changes that make changes in dependent PRs fundamentally wrong, I'd prefer to fix conflicts after PRs are reviewed and approved or refused. Here is why:

• Rebasing doesn't take much time (around 10-15 minutes usually) but given how active is frontend-app-learner-portal-enterprise and edx-enterprise development, it quickly adds up.
• For the same reason conflicts often pop up even before a maintainer has a chance to take another look, making approval impossible.
• Finally, PRs with conflicts can't be merged, and only maintainers can merge PRs, so they always can do final checks before clicking "Merge" and request changes if something is off. But conflicts alone, in my opinion, shouldn't block approval / refusal if the code is fine / needs rework.

Does this make sense? @e0d, maybe you have suggestions on what we can improve from our side to streamline the review process?

[mphilbrick211]

Hi @openedx/2u-titans! Is someone able to review / merge this for us?

[bradenmacdonald]

This is the same approach we've taken with Meilisearch in Redwood for the new Course Search feature: a unique API key is generated for every user, so they can make queries against Meilisearch directly but only see results that they're allowed to see.

Question: why would we want to leave the option in place for the "old" way with a shared key - is that still useful?

[0x29a]

Question: why would we want to leave the option in place for the "old" way with a shared key - is that still useful?

I think it is, @bradenmacdonald. Here is what @tecoholic wrote in our internal ticket:

the issue seems to arise from the difference between "Filtering" which HMX wants to limit who can see what content and "Faceting" which edx.org wants to make sure everyone can see everything to maximize visibility (a typical e-com company use case I guess).

I didn't realize this when was implementing the backend part. If we want to get rid of the "old" way, the algolia_key endpoint has to be re-worked to cover edX's use case.

[mphilbrick211]

Hi @0x29a! Is this pull request still in progress?

[mphilbrick211]

Hi @0x29a! Is this pull request still in progress?

Friendly ping on this, @0x29a!