[SCHEMATIC-1] BugFix: manifest submission when data previously annotated without manifest upload

[GiaJordan]

Issue

Resolves SCHEMATIC-1 which occurs when users:

• Have a dataset/top level folder on synapse with 50 or more files
• generate a manifest with schematic/DCA and fill it out
  • possibly validate the manifest with schematic
• Use the manifest to annotate the data with a client other than schematic ie. R
• Try to generate a new manifest with schematic using the existing annotations from the data on synapse

Changes

• when a temporary file view is created with DatasetFileView and the resulting view is "tidied", any existing eTag columns will be removed before the new values are stored in that column.
• Test resources on synapse have been created to replicate the issue and an integration test has been added. It was not necessary to include 50+ files as we're testing DatasetFileView directly instead of through manifest generation

Notes

• Technically using --use_annotations to generate a manifest for a dataset where there isn't already a manifest present is not supported, but this provides a way for users to get back on the recommended data flow path instead of having to start all over.
• DatasetFileView is only used in this one case, gathering annotations for files in a dataset with more than 50 files. With the ability we have now to scope fileview queries in the SynapseStorage object and get the annotations for all files in a dataset we should consider whether we should deprecate the DatasetFileView class.
• We should also consider whether there's anything else preventing us from officially supporting using the --use_annotations feature when there is no manifest in the dataset since this change enables it in at least this case.

[thomasyu888]

Great work @GiaJordan , I'll defer to @BryanFauble and/or @andrewelamb for final review

[BryanFauble]

LGTM!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
33.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud