refactor: store large checkpoints as multipart upload

[sergiupopescu199]

Description of change

Add the ability to store large checkpoints as multipart upload on a remote store (e.g AWS S3).

Links to any relevant issues

fixes #4983

Type of change

Choose a type of change, and delete any options that are not relevant.

• Enhancement (a non-breaking change which adds functionality)

How the change has been tested

• Followed the docker/iota-data-ingestion/README.md instructions of setting up localstack and all the necessary components to test locally
• updated the BlobWorker to accommodate as checkpoint 0 with size around 13GB

#[async_trait]
impl Worker for BlobWorker {
    async fn process_checkpoint(&self, checkpoint: CheckpointData) -> Result<()> {
        let bytes = Blob::encode(&checkpoint, BlobEncoding::Bcs)?.to_bytes();
        let location = Path::from(format!(
            "{}.chk",
            checkpoint.checkpoint_summary.sequence_number
        ));

        let bytes = if checkpoint.checkpoint_summary.sequence_number == 0 {
            vec![0; 13000 * 1024 * 1024] // Around 13GB worth of zeros
        } else {
            bytes
        };

        self.upload_blob(
            bytes,
            checkpoint.checkpoint_summary.sequence_number,
            location,
        )
        .await?;

        Ok(())
    }
}

• Ran the local node and the data-ingestion-core bin
• Checked if any timeouts were encountered

Change checklist

Tick the boxes that are relevant to your changes, and delete any items that are not.

• I have followed the contribution guidelines for this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
apps-backend	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 30, 2025 1:49pm
apps-ui-kit	✅ Ready (Inspect)	Visit Preview		Jan 30, 2025 1:49pm
rebased-explorer	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 30, 2025 1:49pm
wallet-dashboard	✅ Ready (Inspect)	Visit Preview		Jan 30, 2025 1:49pm

[kodemartin]

Hey @sergiupopescu199, it looks good. Left a couple of suggestion to consider/discuss.

[kodemartin]

looks good with a comment that needs addressing and one more optional nitpick.

[samuel-rufi]

Nice PR @sergiupopescu199 ! From my side in addition only some refinements on documentation.

[tomxey]

lgtm 🌍

[kodemartin]

lgtm 🎈 great job to the author, and the reviewers!