diff: KeyError in get_dict_from_list when diffing archives with a removed folder that now has .nobackup

[rxgh99]

Describe the bug
A folder was removed as a source using the source window in Vorta. The file .nobackup was touched in the removed folder and a subsequent backup was run.

When diffing the .nobackup archive run with the previous archive (which included the folder), the following error occurred (note: from Vorta logs). In this example, the folder "Music" was removed and .nobackup placed in the folder.

2021-04-06 23:25:36,589 - vorta.borg.borg_thread - INFO - Running command /Applications/Vorta.app/Contents/Resources/borg-dir/borg.exe diff --info --log-json xxxxxx@xxxxxx.repo.borgbase.com:repo::hostname-2021-04-06-185815 hostname-2021-04-06-190533
2021-04-06 23:25:42,110 - root - CRITICAL - Uncaught exception, file a report at https://github.com/borgbase/vorta/issues/new
Traceback (most recent call last):
File "views/archive_tab.py", line 520, in list_diff_result
File "views/diff_result.py", line 22, in init
File "views/diff_result.py", line 94, in parse_diff_lines
File "views/diff_result.py", line 87, in parse_line
File "utils.py", line 112, in get_dict_from_list
KeyError: 'Music'

Note: an additional backup was run and a diff on the two most recent archives (those with the .nobackup exclude) was successful. The failure occurs diffing the transition to .nobackup.

To Reproduce
Steps to reproduce the behavior:

1. Include a folder as a source
2. Run a backup
3. Touch .nobackup in the folder
4. Remove the folder from the source list
5. Run a backup
6. Diff the two backups

Environment (please complete the following information):

• OS: macOS 11.2
• Vorta version: 0.7.5
• Installed from: vorta.borgbase.com

[m3nu]

The diff feature was completely rewritten for the upcoming 0.7.6 version. So the issue you are describing is probably already resolved in the current master branch. Can you test if the issue still occurs there?

[rxgh99]

Thanks. I would like to test against master branch - are there any quick pointers on how to pull and install? I originally installed Vorta from binary; I also have brew installation. Thanks.

[m3nu]

https://vorta.borgbase.com/contributing/#testing-the-current-development-version

[rxgh99]

Thank you. I managed to get it to install, and run, but still appears to be pulling 0.7.5 and was getting same error on running this version of vorta. (sorry, no logs from installation.) Attempting to debug.

[m3nu]

To use the latest json-based parsing it also needs Borg 1.1.16.

But if list-based parsing still has an issue, we should address that too.

If you mean the version number inside the app, it’s only updated before a release.

[rxgh99]

Thanks. I'm reinstalling brew and will install Borg and Vorta from there and then test.

[rxgh99]

Very similar error is still returned. Running Borg 1.1.16 from brew and latest Vorta installed via pip3.

Log of error:
2021-04-07 - vorta.borg.borg_thread - INFO - Running command /opt/homebrew/bin/borg diff --info --log-json --json-lines xxxxxx@xxxxxx.repo.borgbase.com:repo::hostname.home-2021-04-06-185815 hostname-2021-04-06-190533
2021-04-07 - root - CRITICAL - Uncaught exception, file a report at https://github.com/borgbase/vorta/issues/new
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.9/site-packages/vorta/views/archive_tab.py", line 518, in list_diff_result
window = DiffResult(result['data'], archive_newer, archive_older, result['params']['json_lines'])
File "/opt/homebrew/lib/python3.9/site-packages/vorta/views/diff_result.py", line 32, in init
files_with_attributes, nested_file_list = parse_diff_json_lines(lines)
File "/opt/homebrew/lib/python3.9/site-packages/vorta/views/diff_result.py", line 59, in parse_diff_json_lines
d = get_dict_from_list(nested_file_list, dirpath.split('/'))
File "/opt/homebrew/lib/python3.9/site-packages/vorta/utils.py", line 112, in get_dict_from_list
return reduce(operator.getitem, mapList, dataDict)
KeyError: 'Music'

However, the command is successful and returns a list of changes when run at the command line in Terminal.

[m3nu]

Tagging @rblenis , who is the expert on json-diff.

[rblenis]

I will take a look at it this weekend.

[m3nu]

Steps to reproduce the behavior:

1. Include a folder as a source
2. Run a backup
3. Touch .nobackup in the folder
4. Remove the folder from the source list
5. Run a backup
6. Diff the two backups

I couldn't reproduce this in the current master. And noticed that step 3 shouldn't have any effect.

We did get a similar issue report #650 last year, so maybe there is something going on.

[rxgh99]

I should provide more detail on the use case. I have tried again but fail to reproduce with current 0.7.5 binary.

More detail: 
# I currently have bar1 and bar2 folders explicitly defined in Sources
- Example paths for first backup:    /Volumes/Main/foo/bar1,  /Volumes/Main/foo/bar2

# I no longer want to backup /Volumes/Main/foo
- Remove foo/bar1 and foo/bar2 from the Sources
- Add .nobackup to foo
- Add /Volumes/Main to sources

I have the original archives that failed the diff and the only difference I can see between the various archives is the order of the folders, when looking through the extract window (one is the reverse order to the other). But diffs to other archives with the same opposite folder order do not force an exception.

[m3nu]

Those steps worked to trigger the exception, @rxgh99. Thanks for that!

@rblenis already found the issue and added a PR in #947

[m3nu]

Fixed via #947 by @rblenis.

The fix is already merged into the current master branch. You can test it as described here. If the issue still occurs in the current master branch, please let us know.