feat: add optional check for existence of source directories

[diivi]

Resolves #501.
Will add tests once I know if this is correct.

[diivi]

@witten
This was the error when I tried to add a missing directory before:

Now:

Not sure if this was what was expected so I marked the PR as draft.

[witten]

So I think you found a legitimate bug in the current code! It looks like when database hooks are enabled and a non-existent directory is given in source_directories, then borgmatic gives the weird error message above (in your first screenshot) and exits. That's not intentional; non-existent source directories aren't supposed to cause errors (without your new option being set). When database hooks are not enabled and a non-existent directory is given in source_directories, Borg issues a warning but then continues on without any error, like this:

Processing files ...
/root/noexist: [Errno 2] No such file or directory: 'noexist'

Anyway, I think the bug you've uncovered occurs in collect_special_file_paths() in borgmatic/borg/create.py, which is triggered when a database hook is configured (or read_special is set to true). Specifically, the call to execute_command_and_capture_output() there blows up when one of the directories doesn't exist. At least, I think that's what's going on. If you want to try to fix the bug as part of this ticket it, you are welcome to do so. Otherwise, feel free to file it or just let me know and I'd be happy to have a look.

[witten]

The changes you have so far look good!

[diivi]

Interesting, I'll work on that bug next, just to do one thing per PR.

[witten]

This is great. Thank you!

[diivi]

Specifically, the call to execute_command_and_capture_output() there blows up

Yes, I just checked, and this is what's happening. I don't understand why Borg raises an error when I have database hooks configured and pass a non-existent directory in this command though:

borg create --one-file-system --read-special --list --filter AMEx- /home/divyansh/Desktop/bkt2::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /home/divyansh/Desktop/os12 /root/.borgmatic --dry-run --list

As you said, even when the source directory does not exist, it should just give a warning and continue.

Also, this (similar) command works (when I don't have database hooks configured):

borg create --list --filter AMEx- /home/divyansh/Desktop/bkt2::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f} /home/divyansh/Desktop/os12 /root/.borgmatic --stats --debug --show-rc

[witten]

I'm gonna move this discuss over to https://projects.torsion.org/borgmatic-collective/borgmatic/issues/655! I'll respond there.