Implement Backup Compression and Decompression

[denisakp]

Overview

To improve the manageability and efficiency of handling backup files, we want to implement a feature to compress backups into tar.gz format before uploading them to storage backends. This enhancement applies to backups for:

• PostgreSQL (both plain and directory formats)
• MySQL (plain SQL)
• MongoDB (directory format backups)

The compressed backups will simplify storage and retrieval processes, reduce upload times, and optimize space usage on storage backends. Additionally, we require a robust decompression function to handle previously compressed files, ensuring fault tolerance and efficient resource utilization.

Why This Change is Necessary

• Reduced Storage and Bandwidth Usage: Compressing backups reduces file size, minimizing storage costs and speeding up uploads/downloads.
• Simplified Management: By compressing MongoDB directory backups and PostgreSQL directory format backups into single files, we streamline handling and retrieval.
• Scalability: Optimizing compression and decompression using Go routines ensures that the solution remains performant even with large datasets.
• Fault Tolerance: Backups should be reliable even in the event of failures during compression or decompression. Implementing a robust system ensures data integrity.

Acceptance Criteria

• Compress PostgreSQL plain and directory backups into tar.gz format before upload
• Compress MySQL plain backups into tar.gz format
• Compress MongoDB directory backups into tar.gz format to create a single compressed file
• Implement decompression functionality for tar.gz files that were previously compressed using this system
• Ensure that both compression and decompression processes are fault-tolerant and optimized for concurrency using Go routines

Technical Details

1. Compression Functionality

• Implement a CompressBackup function that compresses backups into tar.gz format.
• Utilize Go routines for parallel compression to optimize performance, especially for large databases.
• Handle errors gracefully, ensuring partial failures do not leave corrupted archives

2. Decompression Functionality

• Implement a DecompressBackup function to handle extracting tar.gz files
• Ensure fault tolerance with retry mechanisms and optimized Go routines
• Validate the integrity of decompressed files before considering the process successful

Tasks

• Implement CompressBackup function with fault tolerance and concurrency
• Implement DecompressBackup function with the same reliability features
• Integrate compression into the backup process before uploading to storage backends