Spot launch doesn't respect .gitignore

[keirp]

When launching an instance with sky launch, files included in the my .gitignore file are ignored during the rsync. However, when I launch a job with sky spot launch, it seems as though the files are included. Is this intended behavior?

[Michaelvll]

Thanks for reporting the issue @keirp! This has been an issue for a while. The inconsistency comes from the cloud bucket’s rsync interface misalign with the system’s rsync. As we are using cloud bucket as the backend for the files uploading to the spot job, instead of using the system’s rsync for normal jobs, the problem occurs. cc @romilb, maybe we can manually list all the files to be uploaded using rsync’s —dryrun and call the cloud bucket cli with a list of files?

…

________________________________ From: keirp ***@***.***> Sent: Tuesday, April 25, 2023 1:00:17 PM To: skypilot-org/skypilot ***@***.***> Cc: Subscribed ***@***.***> Subject: [skypilot-org/skypilot] Spot launch doesn't respect `.gitignore` (Issue #1898) When launching an instance with sky launch, files included in the my .gitignore file are ignored during the rsync. However, when I launch a job with sky spot launch, it seems as though the files are included. Is this intended behavior? — Reply to this email directly, view it on GitHub<#1898>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABTQXJM3B7EHZDFZWHX7BHTXDAUNDANCNFSM6AAAAAAXLPCJG4>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[concretevitamin]

cc @romilbhardwaj

Another idea is to read .gitignore and pass each line as an --exclude arg to the aws s3 sync command: https://github.com/skypilot-org/skypilot/blob/master/sky/data/storage.py#L1711

[concretevitamin]

Hit this myself. Current workdir has a bunch of "results" non-code files that should not be uploaded. Now every spot launch is taking quite a while just for uploading these. An important UX issue.

[landscapepainter]

I'll take on this one.

[landscapepainter]

@concretevitamin Would you mind elaborating on what you meant by 'bunch of "results" non-code files'? I haven't encountered anything odd that got uploaded to the spot instance while running spot launch

[romilbhardwaj]

I think he was just providing an example - he has a some non-code files and dirs (e.g., experiment results) that are large and included in his gitignore, but they still get uploaded when running sky spot launch. If we parse .gitignore correctly when uploading storage objects, these files should not get uploaded.