Capability to support loading terminated jobs after leader election #595

sundargates · 2023-11-29T23:22:18Z

Context

^^^

Checklist

./gradlew build compiles code correctly
Added new tests where applicable
./gradlew test passes all tests
Extended README or added javadocs where applicable

github-actions · 2023-11-29T23:29:46Z

Test Results

487 tests - 62 481 ✔️ - 60 7m 15s ⏱️ -10s
131 suites ±  0     6 💤 -   2
131 files ±  0     0 ❌ ±  0

Results for commit f381695. ± Comparison against base commit 1dfc742.

This pull request removes 68 and adds 6 tests. Note that renamed tests count towards both.

io.mantisrx.master.jobcluster.JobClusterTest ‑ testCronTriggersSLAToKillOld
io.mantisrx.master.jobcluster.JobClusterTest ‑ testExpireOldJobs
io.mantisrx.master.jobcluster.JobClusterTest ‑ testGetJobDetailsForArchivedJob
io.mantisrx.master.jobcluster.JobClusterTest ‑ testGetLastSubmittedJob
io.mantisrx.master.jobcluster.JobClusterTest ‑ testGetLastSubmittedJobSubject
io.mantisrx.master.jobcluster.JobClusterTest ‑ testGetLastSubmittedJobSubjectWithWrongClusterNameFails
io.mantisrx.master.jobcluster.JobClusterTest ‑ testGetLastSubmittedJobWithCompletedOnly
io.mantisrx.master.jobcluster.JobClusterTest ‑ testGetLastSubmittedJobWithNoJobs
io.mantisrx.master.jobcluster.JobClusterTest ‑ testJobClusterArtifactUpdate
io.mantisrx.master.jobcluster.JobClusterTest ‑ testJobClusterArtifactUpdateMultipleTimes
…

io.mantisrx.master.jobcluster.CompletedJobStoreTest ‑ testInitializationOfCompletedJobStore
io.mantisrx.master.jobcluster.CompletedJobStoreTest ‑ testJobClusterDeletion
io.mantisrx.master.jobcluster.CompletedJobStoreTest ‑ testLazyLoadingOfNewPages
io.mantisrx.master.jobcluster.CompletedJobStoreTest ‑ testWhenJobGetsCompleted
io.mantisrx.master.jobcluster.CompletedJobStoreTest ‑ testWhenJobIsNotThere
io.mantisrx.server.master.store.KeyValueStoreTest ‑ testUpsertOrdered

♻️ This comment has been updated with latest results.

github-actions · 2023-11-29T23:33:47Z

Uploaded Artifacts

To use these artifacts in your Gradle project, paste the following lines in your build.gradle.

resolutionStrategy {
    force "io.mantisrx:mantis-client:0.1.0-20231207.221317-455"
    force "io.mantisrx:mantis-common-serde:0.1.0-20231207.221317-454"
    force "io.mantisrx:mantis-common:0.1.0-20231207.221317-454"
    force "io.mantisrx:mantis-discovery-proto:0.1.0-20231207.221317-454"
    force "io.mantisrx:mantis-runtime:0.1.0-20231207.221317-455"
    force "io.mantisrx:mantis-remote-observable:0.1.0-20231207.221317-455"
    force "io.mantisrx:mantis-runtime-loader:0.1.0-20231207.221317-455"
    force "io.mantisrx:mantis-testcontainers:0.1.0-20231207.221317-124"
    force "io.mantisrx:mantis-connector-iceberg:0.1.0-20231207.221317-453"
    force "io.mantisrx:mantis-connector-job:0.1.0-20231207.221317-455"
    force "io.mantisrx:mantis-connector-kafka:0.1.0-20231207.221317-455"
    force "io.mantisrx:mantis-network:0.1.0-20231207.221317-454"
    force "io.mantisrx:mantis-control-plane-client:0.1.0-20231207.221317-454"
    force "io.mantisrx:mantis-control-plane-core:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-connector-publish:0.1.0-20231207.221317-454"
    force "io.mantisrx:mantis-control-plane-server:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-examples-groupby-sample:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-examples-jobconnector-sample:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-examples-mantis-publish-sample:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-examples-core:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-shaded:0.1.0-20231207.221317-453"
    force "io.mantisrx:mantis-examples-sine-function:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-examples-synthetic-sourcejob:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-publish-core:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-examples-wordcount:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-examples-twitter-sample:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-server-agent:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-server-worker:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-server-worker-client:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-source-job-kafka:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-source-job-publish:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-publish-netty-guice:0.1.0-20231207.221317-448"
    force "io.mantisrx:mantis-publish-netty:0.1.0-20231207.221317-447"
}

Andyz26 · 2023-11-30T17:47:07Z

...mantis-control-plane-server/src/main/java/io/mantisrx/master/jobcluster/JobClusterActor.java

+        // let's load 1 page of completed jobs from DB and then let the rest be loaded lazily
+        // todo(sundaram): Use a clock here
+        Instant end = Instant.now();
+        List<CompletedJob> completedJobs = jobStore.loadCompletedJobsForCluster(name, end.minus(Duration.ofDays(7)), end);


is this a blocking IO call? can it be converted to a future?

sundargates · 2023-12-07T21:16:44Z

...ntis-control-plane-server/src/main/java/io/mantisrx/master/jobcluster/CompletedJobStore.java

+
+interface ICompletedJobsStore {
+
+    void initialize() throws IOException;


Add comments

sundargates · 2023-12-08T06:52:09Z

...mantis-control-plane-server/src/main/java/io/mantisrx/master/jobcluster/JobClusterActor.java

-
+        void initialize() {
+            try {
+                logger.info("Loading completed jobs for cluster {}", name);


Change this to debug instead.

Andyz26 · 2023-12-08T23:07:52Z

...ntis-control-plane-server/src/main/java/io/mantisrx/master/jobcluster/CompletedJobStore.java

+    List<CompletedJob> getCompletedJobs(int limit) throws IOException;
+
+    /**
+     * Gets a list of completed jobs
+     * @param limit number of jobs to return
+     * @param endExclusive end job id
+     * @return list of completed jobs
+     * @throws IOException if there is an error
+     */
+    List<CompletedJob> getCompletedJobs(int limit, JobId endExclusive) throws IOException;


So this only supports reading first x jobs? Is there a way to read from the list paginated?

Andyz26 · 2023-12-08T23:08:33Z

...ntis-control-plane-server/src/main/java/io/mantisrx/master/jobcluster/CompletedJobStore.java

+        while (!jobs.isEmpty()) {
+            for (CompletedJob completedJob : jobs) {
+                try {
+//                    todo(sundaram): Clean this up. This is a hack to get around the fact that the job store


incomplete comment?

sundargates requested review from nickmahilani, jeffchao, piygoyal, calvin681, Andyz26, hmitnflx, markcho and fdc-ntflx as code owners November 29, 2023 23:22

sundargates temporarily deployed to Integrate Pull Request November 29, 2023 23:22 — with GitHub Actions Inactive

sundargates had a problem deploying to Integrate Pull Request November 30, 2023 00:01 — with GitHub Actions Failure

sundargates had a problem deploying to Integrate Pull Request November 30, 2023 17:10 — with GitHub Actions Failure

sundargates had a problem deploying to Integrate Pull Request November 30, 2023 17:15 — with GitHub Actions Failure

Andyz26 reviewed Nov 30, 2023

View reviewed changes

sundargates had a problem deploying to Integrate Pull Request November 30, 2023 18:11 — with GitHub Actions Failure

sundargates had a problem deploying to Integrate Pull Request November 30, 2023 22:18 — with GitHub Actions Failure

sundargates had a problem deploying to Integrate Pull Request December 1, 2023 04:36 — with GitHub Actions Failure

sundargates had a problem deploying to Integrate Pull Request December 2, 2023 07:57 — with GitHub Actions Failure

sundargates had a problem deploying to Integrate Pull Request December 2, 2023 20:57 — with GitHub Actions Failure

sundargates temporarily deployed to Integrate Pull Request December 5, 2023 08:57 — with GitHub Actions Inactive

sundargates temporarily deployed to Integrate Pull Request December 6, 2023 05:44 — with GitHub Actions Inactive

sundargates added 8 commits December 6, 2023 23:56

Intermediate commit

70a823c

Capability to load completed jobs after master restart

879f71a

Ensuring the end timestamp is being updated after job completion

dc618b3

Fixing the test

a3c8ab1

Fixing the test

7af1341

Assume endedAtInstant will be set for a completed job

f745ff6

Fixing the test

9f53131

Trying something

840f169

sundargates added 6 commits December 6, 2023 23:56

Renaming CompletedJobs Table

fbb8d0c

Cleaning up

2063588

Tests for CompletedJobStore

ceecbdf

Fixing the test

d53eb67

Adding more logs

97bf13f

Increasing retention

962a49d

sundargates force-pushed the sundaram/jobs_restart branch from 805b66a to 962a49d Compare December 7, 2023 07:57

sundargates had a problem deploying to Integrate Pull Request December 7, 2023 07:57 — with GitHub Actions Failure

Cleaning up

0c62b7a

sundargates had a problem deploying to Integrate Pull Request December 7, 2023 21:48 — with GitHub Actions Failure

Adding metrics

961c8af

sundargates temporarily deployed to Integrate Pull Request December 7, 2023 22:03 — with GitHub Actions Inactive

sundargates commented Dec 8, 2023

View reviewed changes

Addressing comments

f381695

sundargates had a problem deploying to Integrate Pull Request December 8, 2023 06:59 — with GitHub Actions Failure

sundargates merged commit c5c2a9d into master Dec 8, 2023

sundargates deleted the sundaram/jobs_restart branch December 8, 2023 07:07

Andyz26 reviewed Dec 8, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Capability to support loading terminated jobs after leader election #595

Capability to support loading terminated jobs after leader election #595

sundargates commented Nov 29, 2023

github-actions bot commented Nov 29, 2023 •

edited

Loading

github-actions bot commented Nov 29, 2023 •

edited

Loading

Andyz26 Nov 30, 2023

sundargates Dec 7, 2023

sundargates Dec 8, 2023

Andyz26 Dec 8, 2023

Andyz26 Dec 8, 2023


		interface ICompletedJobsStore {

		void initialize() throws IOException;

Capability to support loading terminated jobs after leader election #595

Capability to support loading terminated jobs after leader election #595

Conversation

sundargates commented Nov 29, 2023

Context

Checklist

github-actions bot commented Nov 29, 2023 • edited Loading

Test Results

github-actions bot commented Nov 29, 2023 • edited Loading

Uploaded Artifacts

Andyz26 Nov 30, 2023

Choose a reason for hiding this comment

sundargates Dec 7, 2023

Choose a reason for hiding this comment

sundargates Dec 8, 2023

Choose a reason for hiding this comment

Andyz26 Dec 8, 2023

Choose a reason for hiding this comment

Andyz26 Dec 8, 2023

Choose a reason for hiding this comment

github-actions bot commented Nov 29, 2023 •

edited

Loading

github-actions bot commented Nov 29, 2023 •

edited

Loading