Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor common cache entry #634

Merged
merged 40 commits into from
Nov 18, 2022
Merged

Refactor common cache entry #634

merged 40 commits into from
Nov 18, 2022

Conversation

severo
Copy link
Collaborator

@severo severo commented Nov 9, 2022

No description provided.

@codecov-commenter
Copy link

codecov-commenter commented Nov 9, 2022

Codecov Report

Base: 84.98% // Head: 91.43% // Increases project coverage by +6.44% 🎉

Coverage data is based on head (dfa89b1) compared to base (076a0dd).
Patch coverage: 89.48% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #634      +/-   ##
==========================================
+ Coverage   84.98%   91.43%   +6.44%     
==========================================
  Files          37       16      -21     
  Lines        1359      841     -518     
==========================================
- Hits         1155      769     -386     
+ Misses        204       72     -132     
Flag Coverage Δ
jobs_mongodb_migration 84.00% <56.66%> (?)
libs_libcache 100.00% <100.00%> (?)
services_admin ?
services_api ?
workers_first_rows ?
workers_splits ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
jobs/mongodb_migration/tests/test_plan.py 96.15% <ø> (ø)
...s/mongodb_migration/src/mongodb_migration/check.py 38.70% <7.14%> (ø)
...grations/_20221117223000_cache_generic_response.py 64.86% <64.86%> (ø)
...ngodb_migration/src/mongodb_migration/collector.py 100.00% <100.00%> (ø)
...bs/mongodb_migration/src/mongodb_migration/plan.py 100.00% <100.00%> (ø)
libs/libcache/src/libcache/simple_cache.py 100.00% <100.00%> (ø)
libs/libcache/tests/test_simple_cache.py 100.00% <100.00%> (ø)
services/admin/src/admin/app.py
services/admin/src/admin/dataset.py
services/admin/src/admin/prometheus.py
... and 46 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

instead of one collection for each kind of cached entry (splits,
first-rows, etc). Some admin methods have bveen disabled for now and
will be reintroduced later.
The preferred way to refresh the cache is to specifically call the
webhook for each dataset you want to refresh
since the cache is considered stale if the worker version or the git
revision has changed.
also: rename internal method _clean_database to _clean_cache_database
So that the hook will stop the update of the Helm chart. If not, the
code will be updated (and expects the new database) while the database
migration has failed
@severo severo marked this pull request as ready for review November 18, 2022 09:00
@severo severo merged commit d17a26e into main Nov 18, 2022
@severo severo deleted the refactor-common-cache-entry branch November 18, 2022 12:12
@severo
Copy link
Collaborator Author

severo commented Nov 18, 2022

Migration script worked (even if it took more than the default Helm timeout for hooks -> deploy failed, without a consequence, since the migration job kept up until finished with success; I just redeployed, and the migration was skipped since already applied)

Defaulted container "datasets-server-mongodb-migration" out of: datasets-server-mongodb-migration, prepare-assets (init)
DEBUG: 2022-11-18 12:16:18,963 - root - Log level set to: DEBUG
INFO: 2022-11-18 12:16:18,983 - root - Assets directory: /assets
INFO: 2022-11-18 12:16:18,997 - root - Start migrations
INFO: 2022-11-18 12:16:19,117 - root - 2 migrations have already been applied. They will be skipped.
INFO: 2022-11-18 12:16:19,117 - root - Migrate 20221117223000: add to the migrations collection
INFO: 2022-11-18 12:16:19,129 - root - Migrate 20221117223000: apply
INFO: 2022-11-18 12:16:19,130 - root - Create the cachedResponsesBlue collection, and fill it with the data from splits and first-rows
INFO: 2022-11-18 12:32:45,657 - root - Migrate 20221117223000: validate
INFO: 2022-11-18 12:32:45,657 - root - Validate the migrated documents
INFO: 2022-11-18 12:36:20,652 - root - Migrate 20221117223000: done
INFO: 2022-11-18 12:36:20,652 - root - All migrations have been applied

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants