feat: set annotations in an async way when submitting a manifest

[linglp]

Context

related to: FDS-2113 Store annotations in an async way when submitting manifests
The speed of submitting manifest when storing annotation in a synchronous way is like ~10min

Major changes

• Per communication with Bryan, I updated the code to get annotations and store annotations in an async way
• update synapseclient to 4.3.0 to avoid timeout error in 4.2.0

Benchmarking

With the help of tracing, I was able to benchmark manifest submission with the new change.

For submitting a manifest locally with file_only option and file_annotations_upload=True option for 3000 files:

In my local environment, with async:

• the speed is around 5min 12s, the longest latency was 5min 39 second, and the shortest latency was 4min 38 seconds.

Using aws-dev environment to test, with async:

• the speed is around 4min 40s, the longest latency was 4min 50 seconds, and the shortest latency is 4min 30 seconds.

[BryanFauble]

I like the direction of your changes here. Great work!

[BryanFauble]

Just a few comments around areas to give some more thought. Otherwise this is great!

[sonarqubecloud]

Quality Gate passed

Issues
25 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud