Research and Prototype Earth Data CMR Requests for Granule Verification Logic - for DMS DAG

[krisstanton]

Research and Prototype Earth Data CMR Requests for Granule Verification Logic - for DMS DAG

Adding a bit of context around the larger task (beyond the scope of this ticket):
In order to get automated deletes working, here are the components we need

• Automated Inventory Processing into the DMS DB -- Dependent on DMS
  • This would be the restore/conversion process and then reporting it on DMS.
• Verification of granules (our checkboxes) -- Some of this is related to CNN and some is Dependent on DMS
  • This ticket (The one you are reading right now!) covers the logic for one of these checkboxes (for the MAXAR Delivery Bucket)
• Integrate the verification scripts and Automated Deletes scripts into a DAG -- this should be reusable for all vendors

Background.
This a component is for the continuous deletion of files from the MCP Maxar Delivery Bucket.
This ticket encapsulates the task of checking if a Granules is properly published within the searchable Earth Data system.

• Find out specifically which Earth Data CMR API Requests we can make in order to verify
  • Individual Granules
  • A Batch of Granules
  • Note, There is a normal API which we have to throttle requests, find out if there is an alternate one we can use, or at least what the limits are on the main one we can use.
    • Verify and document the CMR API Requests limits (for the normal and alternative method - if it exists)
• Create a prototype which can make these requests and parse through the returned data
  • Use code I've previously written as a starting point
  • Ensure that each Granule we examine is actually stored on CBA PROD and NOT OLD NGAP (check the S3 Directory Path in the detailed metadata that comes back with the request)
  • Check with the DMS team to make sure this will work within the Archive DAG framework
  • Assist with integration of this code with the Archive DAG where needed.