An all-in-one mirror utility.
For more information about our mirror service, refer to https://github.com/sjtug/mirror-docker-unified/wiki For legacy version, refer to legacy branch.
mirror-clone is originally built for setting up a mirror repo on SJTU S3 service, and now being made into an all-in-one mirror tool. The most notable feature of mirror-clone is to synchronize files from Rsync endpoint to S3.
mirror-clone revolves around abstractions like source, target, snapshot, and transfer. By assembling them, we can get a task that transfers one repo to another.
RUST_LOG=info ./mirror-clone --progress --target-type s3 --s3-prefix homebrew-bottles --s3-buffer-path /srv/disk1/mirror-clone-cache homebrew
RUST_LOG=info ./mirror-clone --progress --target-type s3 --s3-prefix crates.io/crates --s3-buffer-path /srv/disk1/mirror-clone-cache crates-io
RUST_LOG=info ./mirror-clone --progress --target-type file --file-base-path ~/mirror-clone/crates.io --file-buffer-path ~/Work/intel_temp crates-io
When running on server, we recommend using RUST_LOG=info flag and remove --progress flag.
For more usage, refer to our infra wiki.
- Simple Diff Transfer, compares filename and transfer what's missing in target.
Refer to source code for more information.
Refer to source code for more information.
- mirror-intel, sends HEAD request to mirror-intel endpoint, so as to fill the mirror-intel cache.
- S3
- File system
Use ./mirror-clone --help to view current commands and their meaning.
- More pipes. For example, snapshot cache pipe, etc.
- More sources.
- More transfers.