Show backup size with excludes applied

[freder]

since I want this feature and I believe it has been requested elsewhere already, I had a first stab at it, as a proof-of-concept.

I'm currently not sure if the globmatch package handles patterns the same way borg does.

[m3nu]

Nice take on a hard problem. 👍

If this works, just put the new, more accurate size. No need to add a new column (except for debugging).

I just dont think we can add this new globmatch dependency. Only 6 stars, 3 files with a few lines each. Would be too risky and hard to package.

And it’s not really needed: we mainly need shell patterns and maybe fn. See here for the full list: https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns

Shell patterns are translated into Python regex by Borg. We can borrow from there: https://github.com/borgbackup/borg/blob/c88a37eea430d7ec2e5da1ae503e43519ee90cb1/src/borg/shellpattern.py

[m3nu]

Their code is already very similar to the one in Borg. They both say to have copied it from Python’s fnmatch to a degree.

https://github.com/vidartf/globmatch/blob/master/globmatch/translation.py#L91

[freder]

(how) would it work if we simply copied https://github.com/borgbackup/borg/blob/master/src/borg/shellpattern.py into this repo? would we only have to add their license to the one respective file and that's it?

or what if we imported the file directly from the local borg installation? (I little hacky, I suppose)

[m3nu]

We cant import it because Borg is only called via their CLI. It may be installed in a different Python env.

Copying the file will work, since they copied it from fnmatch anyways. Or adjust and integrate the relevant function only.

[jose1711]

Why not to call borg with dry_run and parse the output? I am not a developer so please don't judge me - this is only for a demonstration purposes.

Example: show all excluded files in /etc and /tmp matching the **/r* exclude pattern:

from borg import archiver as borgArchiver
from borg.helpers import Location
from borg.logger import create_logger, setup_logging
from borg.patterns import CmdTuple, FnmatchPattern, IECommand
from tempfile import NamedTemporaryFile
import io
import logging
import argparse

paths = ['/etc', '/tmp']
patterns = [CmdTuple(val=FnmatchPattern('**/r*'), cmd=IECommand.Exclude)]

logfile = io.StringIO()
setup_logging(logfile)

tmp_file = NamedTemporaryFile().name

args = argparse.Namespace()

archiver = borgArchiver.Archiver()
archiver.log_json = False
location = Location(tmp_file)
args.location = location
args.encryption = 'none'
archiver.do_init(args)

args.dry_run = True
args.output_list = True
args.paths = paths
args.patterns = patterns

args.output_filter = None
args.exclude_nodump = None
args.one_file_system = None
args.exclude_caches = None
args.exclude_if_present = None
args.read_special = None
args.keep_exclude_tags = None

archiver.do_create(args)

logfile.seek(0)
print('Files matched by the exclude pattern:')
print([x.strip() for x in logfile.readlines() if x.startswith('x ')])

List of the included files could be parsed from the same logfile + stat call to get the sizes.

[m3nu]

Vorta doesn't interface with Borg via Python, but via the CLI. So we can't use private Borg functions and classes. This is because there are different ways to install Borg and it may not be in the same Python environment.

[jose1711]

Vorta doesn't interface with Borg via Python, but via the CLI. So we can't use private Borg functions and classes. This is because there are different ways to install Borg and it may not be in the same Python environment.

Understood, but in that case same can be achieved via Popen, right?

[m3nu]

If borg create --dry-run gives the required data, then yes.

[jose1711]

If borg create --dry-run gives the required data, then yes.

It is actually recommended in borg-patterns man page:

[freder]

borg create \
	--list \
	--dry-run \
	--exclude='...' \
	--exclude='...' \
	/path/to/archive::DRYRUN \
	/path/to/path \
	2>&1 \
		| perl -lne 'if (/- (.*)/) { print $1 }' \
		| while read p ; do \
			[ -f "$p" ] && echo "$p" ; \
		done

prints the paths of all files that are going to be included in the backup. unfortunately --stats does not work with --dry-run.

[freder]

• removed https://github.com/vidartf/globmatch dependency in favor of https://github.com/borgbackup/borg/blob/master/src/borg/shellpattern.py
• added new column showing number of (filtered) files

[freder]

what do you guys think?

[m3nu]

Could be a good addition. Is it ready to merge?

[freder]

I rebased and cleaned up. should be good to go now.

[m3nu]

Good stuff. And timely to add while v0.8 is under heavy development.

2 thoughts:

• Is translate the best function name, since we already use it to mean translations of language.
• There is a good bit of sizing-related code by now. Does it make sense to move it to a new module/file?

[freder]

2 thoughts:
Is translate the best function name, since we already use it to mean translations of language.

translate is very generic, indeed. I renamed it to pattern_to_regex.

There is a good bit of sizing-related code by now. Does it make sense to move it to a new module/file?

it's probably a good idea to refactor in a separate PR, or not?

[m3nu]

it's probably a good idea to refactor in a separate PR, or not?

True. Thanks for the other change. Will locally test and merge it tomorrow morning.

[m3nu]

Thanks for the valuable addition! Now merged.

[freder]

👍