Dataverse: script to verify WorldMap links are valid

[raprasad]

Dataset metadata will contain links to WorldMap layers.

Verify that the WorldMap links are valid--e.g., that the layers still exist.

Run this check via GeoConnect:

• Get all map layer metadata links via API call
  • Add Dataverse API call to return list of MapLayerMetadata objects. Started in f0cec55.
• Check against WorldMap, delete non-existent links
• Run reverse check. For Layers with DataverseInfo objects, make sure Dataverse still has the file/dataset. Delete the WorldMap DataverseInfo object but not the Layer itself.
  • should rarely happen

---

Notes

• New API endpoint needed in Dataverse codebase
• Originally geoconnect issue: Dataverse: script to verify WorldMap links are valid geoconnect#12

[djbrooke]

We discussed this in the Sprint Planning meeting yesterday. We decided on the following approach:

• We'll add an API endpoint to Dataverse
• We'll add a script to run in Geoconnect to verify the mapping in Dataverse (using the new endpoint) and the the endpoint in Worldmap (existing endpoint)
• We'll Update the Geoconnect documentation to inform users what happens when a mapping is deleted

[raprasad]

Question in gist for tomorrow: https://gist.github.com/raprasad/b69dcd0549e78837cb29b90e07244b7b

[pdurbin]

I'd like to point out that from the WorldMap side, when you delete a map (WorldMap calls it a "layer"), you get no indication that you're leaving cruft behind in Dataverse. This is what the WorldMap user sees:

The cruft left behind in Dataverse is the "Explore" button. When anyone comes along and clicks the Explore button...

... they see "Page Not Found" on the WorldMap side (Safari hides the URL but it's http://worldmap.harvard.edu/data/geonode:starbucks_u_gq ):

This issue is about deleting the cruft from Dataverse, specifically the MapLayerMetadata entry in the Dataverse database, so that the "Explore" button goes away when it has been detected (nightly or whatever) that the map/layer in WorldMap no longer exists.

[pdurbin]

I just opened cga-harvard/geonode#214 asking for the WorldMap "delete" interface to say something like, "this layer is in use by a dataset with a published DOI."

[pdurbin]

@kcondon also, please note that scripts/database/upgrades/upgrade_v4.6.1_to_v4.6.2.sql has been updated

[kcondon]

After deleting map on Worldmap side, there is stale data on Dataverse, ie. the explore button, but also stale data on GeoConnect. This ticket does address the Dataverse stale data but trying to remap the deleted map from the Dataverse side fails for shape files and has a workaround for tabular data.

[raprasad]

@kcondon I put in a fix for stale tabular file data. The updated geoconnect code is available via dvn-build. note: they're in the process of restarting worldmap which is down right now. (May look at shapefiles depending on worldmap availability and time left with other issues)

[raprasad]

Geoconnect pull request: IQSS/geoconnect#125

[pdurbin]

I just reviewed IQSS/geoconnect#125 and it looks good so I'm sending this issue to QA at https://waffle.io/IQSS/dataverse

@kcondon I'd say you can go ahead and merge pull request #3721 into the Dataverse code base because the branch (3480-worldmap-links) hasn't changed since you tested it yesterday and @raprasad and I are not planning on touching it. Basically, further improvements are being worked out in the geoconnect code base at this point.

[kcondon]

@pdurbin I'd still like to test it again, thanks for the heads up.