load_repository() does not load correctly Targets objects of delegated roles

[sechkova]

Description of issue:
When a repository is loaded from the filesystem by repository_tool.load_repository(), delegated roles' objects are missing from the delegating role.

Steps to reproduce:

Create a longer chain of delegations, for example delegate to hashed bins (the same issue applies to any other delegation):

repository.targets.delegate('unclaimed', [public_unclaimed_key], [])
repository.targets('unclaimed').delegate_hashed_bins([], [public_bins_key], 32)
repository.mark_dirty(['00-07', '08-0f', '10-17', '18-1f', '20-27', '28-2f', 
   ...: ...   '30-37', '38-3f', '40-47', '48-4f', '50-57', '58-5f', '60-67', '68-6f', 
   ...: ...   '70-77', '78-7f', '80-87', '88-8f', '90-97', '98-9f', 'a0-a7', 'a8-af', 
   ...: ...   'b0-b7', 'b8-bf', 'c0-c7', 'c8-cf', 'd0-d7', 'd8-df', 'e0-e7', 'e8-ef', 
   ...: ...   'f0-f7', 'f8-ff', 'root', 'targets', 'snapshot', 'timestamp', 'unclaimed'])  
repository.writeall()

Check the length of the delegations:

In [6]: len(repository.targets._delegated_roles)                                                  
Out[6]: 33
In [7]: len(repository.targets('unclaimed')._delegated_roles)                                     
Out[7]: 32

Load the created repository from disk:

repository = load_repository('repository')
In [5]: len(repository.targets._delegated_roles)                                                  
Out[5]: 33

In [6]: len(repository.targets('unclaimed')._delegated_roles)  
Out[6]: 0

repository.targets('unclaimed').add_target_to_bin('t1', 32)
Error: unclaimed does not have a delegated role 60-67

Current behavior:
Loaded delegated roles' objects are missing from the delegating object if the role is not the top-level 'targets' role.

Expected behavior:
After loading from filesystem, the delegated roles are added to the delegating role's _delegated_roles list.

[lukpueh]

Excellent work, @sechkova! Out of curiosity I just compared tuf.roledb._roledb_dict before and after reloading using below snippet. Looks like the issue you describe above only exists in the repository_tool metadata model and not in the roledb metadata model.
However, there seem to be other minor inconsistencies. :/ I'll post more about it later.

# following repository.writeall() from above snippet  (probably taken from the tutorial) ...
import tuf.roledb
import copy, pprint, dictdiffer, operator # NOTE: dictdiffer is 3rd-party
from tuf.repository_tool import *
# Backup roledb (state after constructing metadata)
roledb_dict_backup = copy.deepcopy(tuf.roledb._roledb_dict)
tuf.roledb.clear_roledb()

# Re-populate roledb from files on disk
repository = load_repository(repository._repository_directory)

# Diff them using 3rd-party dictdiffer tool and pretty print
diff_results = dictdiffer.diff(roledb_dict_backup, tuf.roledb._roledb_dict)
pprint.pprint(sorted(diff_results, key=operator.itemgetter(0)))

# Check https://dictdiffer.readthedocs.io/ for docs about the format of the result

[sechkova]

Out of curiosity I just compared tuf.roledb._roledb_dict before and after reloading using below snippet. Looks like the issue you describe above only exists in the repository_tool metadata model and not in the roledb metadata model.

Yes, exactly, what I described concerns the "class-based" metadata representation. Roledb metadata is loaded (mostly) correctly in this case.

[joshuagl]

This was resolved in #1052