-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Backport-2.x][Tiered Caching] Indices Request cache stalekey management (#12625) #12707
[Backport-2.x][Tiered Caching] Indices Request cache stalekey management (#12625) #12707
Conversation
…h-project#12625) * Introduce IndicesRequestCacheCleanupManager Signed-off-by: Kiran Prakash <awskiran@amazon.com> * using cleanup mgr to enqueue cleanups Signed-off-by: Kiran Prakash <awskiran@amazon.com> * readability improvements Signed-off-by: Kiran Prakash <awskiran@amazon.com> * update updateCleanupKeyToCountMap on new cache entry Signed-off-by: Kiran Prakash <awskiran@amazon.com> * create IndicesRequestCacheCleanupManager & settings and validators Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Add IRC specific cache cleaner and remove from IndicesService Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Unit Tests Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <awskiran@amazon.com> * move cachecleaner inside mgr Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove processCleanupKeys Signed-off-by: Kiran Prakash <awskiran@amazon.com> * minor cleanups Signed-off-by: Kiran Prakash <awskiran@amazon.com> * add updateCleanupKeyToCountMapOnCacheEviction Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove locks and make all methods synchronized Signed-off-by: Kiran Prakash <awskiran@amazon.com> * spotless Signed-off-by: Kiran Prakash <awskiran@amazon.com> * updateCleanupKeyToCountMapOnCacheEviction Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Testing Signed-off-by: Kiran Prakash <awskiran@amazon.com> * add Reschedule back to indices service Signed-off-by: Kiran Prakash <awskiran@amazon.com> * rename updateStaleKeysCount to incrementStaleKeysCount Signed-off-by: Kiran Prakash <awskiran@amazon.com> * rename getStaleKeysCountForTesting to getStaleKeysCount Signed-off-by: Kiran Prakash <awskiran@amazon.com> * rename threshold to stalenessThreshold Signed-off-by: Kiran Prakash <awskiran@amazon.com> * check for cleanupKey.entity == null Signed-off-by: Kiran Prakash <awskiran@amazon.com> * use computeIfPresent with keycountmap Signed-off-by: Kiran Prakash <awskiran@amazon.com> * log both staleKeysInCache & Staleness in debug logs Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Use HashMap instead of ConcurrentMap Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Address b/w compatibility Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove synchronized for updateCleanupKeyToCountMapOnCacheEviction Signed-off-by: Kiran Prakash <awskiran@amazon.com> * make cleanCache synchronized Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove shouldRemoveKey Signed-off-by: Kiran Prakash <awskiran@amazon.com> * spotlessApply Signed-off-by: Kiran Prakash <awskiran@amazon.com> --------- Signed-off-by: Kiran Prakash <awskiran@amazon.com> (cherry picked from commit b4da802)
Compatibility status:Checks if related components are compatible with change ca503bc Incompatible componentsSkipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git] |
❌ Gradle check result for 6697fec: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Kiran Prakash <awskiran@amazon.com>
❌ Gradle check result for 4e937ca: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
❌ Gradle check result for 4e937ca: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
This was due to flaky test that has issue created - #10704 |
This was due to flaky test that has issue created - #12236 |
Signed-off-by: Kiran Prakash <awskiran@amazon.com>
❌ Gradle check result for 9d4f011: Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Kiran Prakash <awskiran@amazon.com>
Signed-off-by: Kiran Prakash <awskiran@amazon.com>
❌ Gradle check result for ca503bc: Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
❌ Gradle check result for 9708f79: Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 2.x #12707 +/- ##
============================================
- Coverage 71.28% 71.22% -0.07%
- Complexity 60145 60173 +28
============================================
Files 4957 4963 +6
Lines 282799 283268 +469
Branches 41409 41466 +57
============================================
+ Hits 201591 201752 +161
- Misses 64189 64427 +238
- Partials 17019 17089 +70 ☔ View full report in Codecov by Sentry. |
Description
Backports #12625
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.