(Chore): Migrate to poetry

[BinamB]

Description about what this pull request does.

Please make sure to follow the DEV guidelines before asking for review.

New Features

• Migrated dependency handling to poetry
• Add option to manually add a guid to urls mapping json file to validation script

Breaking Changes

Bug Fixes

• Fix getting file name from the GDC manifest.

Improvements

• Added support for Glacier Instant Retreival
• Add more logs

Dependency updates

• Update black on pre-commit
• Update python image on Dockerfile to python3.9-buster-2.0.0
• Update dependencies in toml

Deployment changes

NOTES: changed scripts folder to dcfdataservice. I did not pick dcf-dataservice over dcfdataservice because poetry's toml file does not like special characters like - or _

[jacob50231]

Code looks good assuming we can get this to pass jenkins.

[jacob50231]

Is there a reason we're still tracking requirements.txt after migrating to poetry?

[MaribelleHGomez]

I believe that is still needed for the Google dataflow job

[BinamB]

Yeah its a different requirements file thats supposed to provide the dependencies for the google dataflow job. The poetry toml installs the dependencies on the kubernetes job and the kubernetes job initiates a google job which uses the requriements.txt

[MaribelleHGomez]

lgtm

[jacob50231]

I was able to fix the snyk issue and we were able to get it tested on a dev environment. Everything is looking good!

Premature still more tests to run