Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add S3 based optimization cache support #30

Open
satyamtg opened this issue Jul 8, 2020 · 1 comment
Open

Add S3 based optimization cache support #30

satyamtg opened this issue Jul 8, 2020 · 1 comment
Labels
enhancement New feature or request stale

Comments

@satyamtg
Copy link
Contributor

satyamtg commented Jul 8, 2020

We use kiwix_storagelib for implementing S3 based optimization cache in the scrapers. However, this gives rise to redundant code. We put a version of the file along with the optimizer version as the metadata always. So, this can be better implemented in scraperlib. For a start, we can have a caching module that can have 3 functions, (or maybe a class containing methods). The primary 3 things we need are -

  • download_from_cache()
  • upload_to_cache()
  • check_credentials()

There can be several ways to have this, but it should at least fulfill the following -

  • Compare optimizer_version
  • Compare file_version

Optional things can be to check file upload date and discard if it's older than a specified amount of time. If we go for a class based approach, we can also explore possibilities to improve performance.

@satyamtg satyamtg added the enhancement New feature or request label Jul 8, 2020
@satyamtg satyamtg changed the title Add S3 based optimization cache helpers Add S3 based optimization cache support Jul 8, 2020
@kelson42 kelson42 pinned this issue Aug 9, 2020
@stale
Copy link

stale bot commented Sep 6, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

@stale stale bot added the stale label Sep 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request stale
Projects
None yet
Development

No branches or pull requests

1 participant