Remove play-iteratees

[ioannakok]

What does this change?

As part of the Scala 2.13 upgrade we need to remove the play-iteratees library because it is no longer supported and there is no 2.13 version. We're replacing it with the paginateAccum CAPI client method. The play-iteratees library was used to fully list all keyword & series tags, for two purposes:

• creating index pages - Index pages help Google crawl our site, so they're good for SEO - they were first introduced with SEO: keyword and contributors index pages #5199 in July 2014
• detecting when redirects should occur away from legacy urls in PublicationController (this functionality was introduced with Prepare for dated newspaper page requests #11150)

  frontend/article/app/controllers/PublicationController.scala

  Line 58 in 6b1adee

  bookAgent.getTags(publication).exists(_.id == tag) || bookSectionAgent.getTags(publication).exists(_.id == tag)

Index pages

• https://www.theguardian.com/index/contributors

• https://www.theguardian.com/index/contributors/b

• https://www.theguardian.com/index/subjects

• https://www.theguardian.com/index/subjects/b

Does this change need to be reproduced in dotcom-rendering ?

• No
• Yes (please indicate your plans for DCR Implementation)

Screenshots

What is the value of this and can you measure success?

Checklist

Does this affect other platforms?

• AMP
• Apps
• Other (please specify)

Does this affect GLabs Paid Content Pages? Should it have support for Paid Content?

• No
• Yes (please give details)

Does this change break ad-free?

• No
• It did, but tests caught it and I fixed it
• It did, but there was no test coverage so I added that then fixed it

Does this change update the version of CAPI we're using?

• No, all the existing database files are just fine
• Yes, and I have re-run all the tests locally and checked in all the updated data/database/xyz files

Accessibility test checklist

• Tested with screen reader
• Navigable with keyboard
• Colour contrast passed

Tested

• Locally
• On CODE (optional)

[ioannakok]

We grabbed a copy of the index data that is stored by the RebuildIndexJob for CODE using this command:

aws s3 cp s3://aws-frontend-pressed/CODE/index . --recursive --profile frontend

We are planning to delete them from the bucket, deploy this branch to CODE and see if the bucket gets populated with the data again.

indexes-s3-content.zip

[ioannakok]

Index data got deleted from bucket and our branch was deployed in CODE. Bucket got populated with new data:
after-deploying-our-branch.zip

I ran diff -rq indexes-s3-content after-deploying-our-branch and we got some diffs

Files indexes-s3-content/contributors/1-9.json and after-deploying-our-branch/contributors/1-9.json differ
Files indexes-s3-content/contributors/e.json and after-deploying-our-branch/contributors/e.json differ
Files indexes-s3-content/contributors/g.json and after-deploying-our-branch/contributors/g.json differ
Files indexes-s3-content/contributors/r.json and after-deploying-our-branch/contributors/r.json differ
Files indexes-s3-content/keywords/c.json and after-deploying-our-branch/keywords/c.json differ
Files indexes-s3-content/keywords/t.json and after-deploying-our-branch/keywords/t.json differ
Files indexes-s3-content/keywords_by_section/news.json and after-deploying-our-branch/keywords_by_section/news.json differ
Only in indexes-s3-content/keywords_by_section: weather.json
Files indexes-s3-content/keywords_by_section/world.json and after-deploying-our-branch/keywords_by_section/world.json differ

Because indexes-s3-content got downloaded from the bucket on 1/7, I redeployed main in CODE, waited for the job to run (https://github.com/guardian/frontend/blob/main/admin/app/model/AdminLifecycle.scala#L85) and downloaded the new data:
after-deploying-main.zip

diff -rq after-deploying-main after-deploying-our-branch gave no diff.

[OllysCoding]

+1 Great work :)

[prout-bot]

Seen on PROD (merged by @ioannakok 11 minutes and 53 seconds ago)

• Check your changes on www.theguardian.com ✔️
• Keep an eye on the deploy dashboard 📉

[rtyley]

Deploy to PROD completed at 3:13pm. Since then, we've seen new S3 files generated by the RebuildIndexJob:

The SEO index pages look unchanged:

• https://www.theguardian.com/index/contributors
  

• https://www.theguardian.com/index/contributors/b
  

• https://www.theguardian.com/index/subjects
  

• https://www.theguardian.com/index/subjects/b
  

https://logs.gutools.co.uk/s/dotcom/goto/cdd97830-fc74-11ec-835d-fb89df6438e4

[ioannakok]

This PR also required:

• Update CAPI & FAPI clients to latest versions (also remove hack required for Lightbox) #25139
• Stop capi client updates from requiring new database files #25098
• Extend auto-pagination to support TagsQuery content-api-scala-client#359