Signals

Signals provides parallel and streaming upload and download of S3 objects.

Features

• Streaming: Parallel and streaming upload and download for efficient operations.
• Integrity checks: Integrity checks are done during multi-part upload.
• Retries: Every call to s3 are retried according to Amazon S3 recommendations.
• Memory conscious: Signals tries to make a conscious usage of memory during upload and download.

Installation

The package is available in Github, using Gradle:

implementation("com.lapanthere:signals:0.4.0")

Usage

File("data.txt").inputStream().use { file -> 
    S3OutputStream(bucket = "bucket", key = "data.txt").use { s3 ->
        file.copyTo(s3)
    }
}

File("data.txt").outputStream().use { file -> 
    S3InputStream(bucket = "bucket", key = "data.txt").use { s3 -> 
        s3.copyTo(file)
    }
}

Part size calculations

The size of each part follow the same logic for both upload and download. It starts at 5MB, and grow up to 3.6GB to allow to upload up to 5TB, to match with AWS S3 limits.

Part size growth

Total size growth

See also

• aws/s3