snapshots #113

MarinPostma · 2021-03-20T10:53:20Z

This pr adds support for snapshoting.

The snapshoting process for an index requires that no other update is processing at the same time. A mutex lock has been added to prevent a snapshot from occuring at the same time as an update, while still premitting updates to be pushed.

The list of the indexes to snapshot is first retrieved from the UuidResolver which also performs its snapshot.

This list is passed to the update store, which attempts to acquire a lock on the update store while it snaphots itself and it's associated index store.

This means that a snapshot can only be completed once all indexes have finished their ongoing update.

This pr also adds refactoring of the code to allow unit testing and mocking, and unit test the snapshot creation.

irevoire · 2021-03-24T17:04:52Z

meilisearch-http/src/index_controller/update_actor/handle_impl.rs

+    async fn get_all_updates_status(&self, uuid: Uuid) -> Result<Vec<UpdateStatus>> {
+        let (ret, receiver) = oneshot::channel();
+        let msg = UpdateMsg::ListUpdates { uuid, ret };
+        let _ = self.sender.send(msg).await;
+        receiver.await.expect("update actor killed.")
+    }
+
+    async fn update_status(&self, uuid: Uuid, id: u64) -> Result<UpdateStatus> {
+        let (ret, receiver) = oneshot::channel();
+        let msg = UpdateMsg::GetUpdate { uuid, id, ret };
+        let _ = self.sender.send(msg).await;
+        receiver.await.expect("update actor killed.")
+    }
+
+    async fn delete(&self, uuid: Uuid) -> Result<()> {
+        let (ret, receiver) = oneshot::channel();
+        let msg = UpdateMsg::Delete { uuid, ret };
+        let _ = self.sender.send(msg).await;
+        receiver.await.expect("update actor killed.")
+    }
+
+    async fn create(&self, uuid: Uuid) -> Result<()> {
+        let (ret, receiver) = oneshot::channel();
+        let msg = UpdateMsg::Create { uuid, ret };
+        let _ = self.sender.send(msg).await;
+        receiver.await.expect("update actor killed.")
+    }
+
+    async fn snapshot(&self, uuid: Uuid, path: PathBuf) -> Result<()> {
+        let (ret, receiver) = oneshot::channel();
+        let msg = UpdateMsg::Snapshot { uuid, path, ret };
+        let _ = self.sender.send(msg).await;
+        receiver.await.expect("update actor killed.")
+    }
+}


Do you think it would be interesting to write a macro that generates all these functions? (for another PR though, this one is already complex enough 😂 )

Yes I thought about that a bit, it's not straightforward, but could be a fun little project :)

@irevoire lol I tried to do so and it looks pretty strange and I'm not sure it became any better 😄
3e875b0

It works, however!

curquiza

I tried cargo run --release --schedule-snapshot=true

and I get in the logs: Snashot scheduled every 86400s

But the snapshot was not created. It should be created when launching:

curquiza · 2021-03-24T19:08:24Z

meilisearch-http/src/index_controller/snapshot.rs

+
+    pub async fn run(self) {
+        info!(
+            "Snashot scheduled every {}s.",


Suggested change

"Snashot scheduled every {}s.",

"Snapshot scheduled every {}s.",

curquiza · 2021-03-24T19:12:43Z

Also, I tried: cargo run --release -- --schedule-snapshot=true --snapshot-interval-sec 10 and after 10sec I got:

By default, a snapshot should be created in the snapshots directory

Then I created the expected snapshots folder and I re-launched and I got:

I re-tried by using: cargo run --release -- --schedule-snapshot=true --snapshot-interval-sec 10 --snapshot-dir snapshots/ but still the second error Is a directory

And I've never got any snapshot in the snapshots folder, but I get .tmp files:

MarinPostma · 2021-03-24T20:20:03Z

Interesting, thanks I'll look into this !

MarinPostma · 2021-03-29T06:15:12Z

bors merge

113: snapshots r=MarinPostma a=MarinPostma This pr adds support for snapshoting. The snapshoting process for an index requires that no other update is processing at the same time. A mutex lock has been added to prevent a snapshot from occuring at the same time as an update, while still premitting updates to be pushed. The list of the indexes to snapshot is first retrieved from the `UuidResolver` which also performs its snapshot. This list is passed to the update store, which attempts to acquire a lock on the update store while it snaphots itself and it's associated index store. This means that a snapshot can only be completed once all indexes have finished their ongoing update. This pr also adds refactoring of the code to allow unit testing and mocking, and unit test the snapshot creation. Co-authored-by: mpostma <postma.marin@protonmail.com> Co-authored-by: tamo <irevoire@protonmail.ch> Co-authored-by: marin <postma.marin@protonmail.com>

bors · 2021-03-29T06:23:22Z

Build failed:

tests

curquiza · 2021-03-29T16:01:40Z

😭 Bors hates us

MarinPostma · 2021-03-29T17:19:47Z

bors try

bors · 2021-03-29T17:28:28Z

try

Build succeeded:

tests

MarinPostma · 2021-03-30T08:05:00Z

bors merge

113: snapshots r=MarinPostma a=MarinPostma This pr adds support for snapshoting. The snapshoting process for an index requires that no other update is processing at the same time. A mutex lock has been added to prevent a snapshot from occuring at the same time as an update, while still premitting updates to be pushed. The list of the indexes to snapshot is first retrieved from the `UuidResolver` which also performs its snapshot. This list is passed to the update store, which attempts to acquire a lock on the update store while it snaphots itself and it's associated index store. This means that a snapshot can only be completed once all indexes have finished their ongoing update. This pr also adds refactoring of the code to allow unit testing and mocking, and unit test the snapshot creation. Co-authored-by: mpostma <postma.marin@protonmail.com> Co-authored-by: tamo <irevoire@protonmail.ch> Co-authored-by: marin <postma.marin@protonmail.com>

bors · 2021-03-30T08:13:50Z

Build failed:

tests

curquiza · 2021-03-31T17:19:30Z

@MarinPostma can we remove the failing test since you created https://github.com/meilisearch/transplant/issues/130?

MarinPostma · 2021-03-31T17:27:37Z

Yes this is on my to-do, you want to make a release?

curquiza · 2021-03-31T17:37:19Z

I would to do an alpha release before the end of the week if possible 🙂

curquiza

perfecto !!

MarinPostma · 2021-03-31T19:45:38Z

bors merge

113: snapshots r=MarinPostma a=MarinPostma This pr adds support for snapshoting. The snapshoting process for an index requires that no other update is processing at the same time. A mutex lock has been added to prevent a snapshot from occuring at the same time as an update, while still premitting updates to be pushed. The list of the indexes to snapshot is first retrieved from the `UuidResolver` which also performs its snapshot. This list is passed to the update store, which attempts to acquire a lock on the update store while it snaphots itself and it's associated index store. This means that a snapshot can only be completed once all indexes have finished their ongoing update. This pr also adds refactoring of the code to allow unit testing and mocking, and unit test the snapshot creation. Co-authored-by: mpostma <postma.marin@protonmail.com> Co-authored-by: tamo <irevoire@protonmail.ch> Co-authored-by: marin <postma.marin@protonmail.com> Co-authored-by: Marin Postma <postma.marin@protonmail.com>

bors · 2021-03-31T19:54:22Z

This PR was included in a batch that successfully built, but then failed to merge into main (it was a non-fast-forward update). It will be automatically retried.

bors · 2021-03-31T19:54:35Z

Merge conflict.

MarinPostma · 2021-04-01T13:15:46Z

bors merge

bors · 2021-04-01T13:24:36Z

Build succeeded:

tests

MarinPostma added 20 commits March 24, 2021 12:38

implement snapshot scheduler

ee838be

use options to schedule snapshot

c966b1d

snapshot indexes

35a7b80

sequential index snapshot

520f7c0

add lock to prevent snapshot during update

7f6a54c

restore snapshots

4847884

fix snapshot creation

a85e7ab

clean snapshot creation

44dcfe2

remove from snapshot

d73fbde

fix snapshot bugs

e9da191

create indexes snapshots concurrently

1f51fc8

refactor for tests

3cc3637

add tests and mocks

4629354

load snapshot

eb53ed4

fix snapshot dir already exists

48d5f88

remove prints

06f9dae

integration test snapshot

1f16c8d

format code

4041d9d

remove bad file name test

83ffdc8

fix clippy

d892a26

MarinPostma force-pushed the snapshots branch from 4c3f2be to d892a26 Compare March 24, 2021 11:43

MarinPostma marked this pull request as ready for review March 24, 2021 11:43

MarinPostma requested review from curquiza and irevoire March 24, 2021 11:44

irevoire previously approved these changes Mar 24, 2021

View reviewed changes

curquiza suggested changes Mar 24, 2021

View reviewed changes

MarinPostma added 2 commits March 25, 2021 09:35

perform snapshot on startup

79d0970

fix snapshot path

d029464

MarinPostma dismissed curquiza’s stale review via 7050236 March 26, 2021 16:54

curquiza previously approved these changes Mar 28, 2021

View reviewed changes

bors bot added a commit that referenced this pull request Mar 29, 2021

Try #113:

84678e8

ignore snapshot test (#130)

d8af4a7

MarinPostma dismissed curquiza’s stale review via d8af4a7 March 31, 2021 18:08

curquiza previously approved these changes Mar 31, 2021

View reviewed changes

Merge remote-tracking branch 'origin/main' into snapshots

248e9b3

MarinPostma dismissed curquiza’s stale review via 248e9b3 April 1, 2021 13:11

curquiza approved these changes Apr 1, 2021

View reviewed changes

bors bot merged commit 89e05fc into main Apr 1, 2021

bors bot deleted the snapshots branch April 1, 2021 13:24

MarinPostma mentioned this pull request Jun 30, 2021

Bug in snapshot test meilisearch/meilisearch#1434

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

snapshots #113

snapshots #113

MarinPostma commented Mar 20, 2021 •

edited

Loading

irevoire Mar 24, 2021

MarinPostma Mar 24, 2021

shekhirin Apr 2, 2021

curquiza left a comment

curquiza Mar 24, 2021

curquiza commented Mar 24, 2021 •

edited

Loading

MarinPostma commented Mar 24, 2021

MarinPostma commented Mar 29, 2021

bors bot commented Mar 29, 2021

curquiza commented Mar 29, 2021

MarinPostma commented Mar 29, 2021

bors bot commented Mar 29, 2021

MarinPostma commented Mar 30, 2021

bors bot commented Mar 30, 2021

curquiza commented Mar 31, 2021

MarinPostma commented Mar 31, 2021

curquiza commented Mar 31, 2021

curquiza left a comment

MarinPostma commented Mar 31, 2021

bors bot commented Mar 31, 2021

bors bot commented Mar 31, 2021

MarinPostma commented Apr 1, 2021

bors bot commented Apr 1, 2021

	"Snashot scheduled every {}s.",
	"Snapshot scheduled every {}s.",

snapshots #113

snapshots #113

Conversation

MarinPostma commented Mar 20, 2021 • edited Loading

irevoire Mar 24, 2021

Choose a reason for hiding this comment

MarinPostma Mar 24, 2021

Choose a reason for hiding this comment

shekhirin Apr 2, 2021

Choose a reason for hiding this comment

curquiza left a comment

Choose a reason for hiding this comment

curquiza Mar 24, 2021

Choose a reason for hiding this comment

curquiza commented Mar 24, 2021 • edited Loading

MarinPostma commented Mar 24, 2021

MarinPostma commented Mar 29, 2021

bors bot commented Mar 29, 2021

curquiza commented Mar 29, 2021

MarinPostma commented Mar 29, 2021

bors bot commented Mar 29, 2021

try

MarinPostma commented Mar 30, 2021

bors bot commented Mar 30, 2021

curquiza commented Mar 31, 2021

MarinPostma commented Mar 31, 2021

curquiza commented Mar 31, 2021

curquiza left a comment

Choose a reason for hiding this comment

MarinPostma commented Mar 31, 2021

bors bot commented Mar 31, 2021

bors bot commented Mar 31, 2021

MarinPostma commented Apr 1, 2021

bors bot commented Apr 1, 2021

MarinPostma commented Mar 20, 2021 •

edited

Loading

curquiza commented Mar 24, 2021 •

edited

Loading