duplicate dataset comes back after dedupe

[FuhuXia]

After dedupe, a duplicate of https://catalog.data.gov/dataset/national-settlement-service-data keeps coming back after next harvest job.

How to reproduce

1. harvest https://catalog.data.gov/harvest/federal-reserve
2. Run dedupe script on org https://catalog.data.gov/organization/board-of-governors-of-the-federal-reserve-system
3. reharvest

Expected behavior

The dataset number should not change.

Actual behavior

One duplicate created for dataset https://catalog.data.gov/dataset/national-settlement-service-data

Sketch

Three approaches to fix the issue

1. Clear the harvest source then reharvest. This will lose tracking stats for all datasets in this source.
2. Exam the state of the affected dataset in DB and SOLR and figure out why duplicate occurs. Could be a new bug in ckanext-datajson.
3. Could be a bug in the dedupe process that an edge case is not handled well.

[FuhuXia]

Did a harvest source clear and reharvest. The issue is back but the duplicate dataset changed to
international-summary-statistics
international-summary-statistics-2ddc8

Trying to replicate it in other environments.

[FuhuXia]

Could not replicate on develop or staging.

[FuhuXia]

The root cause of the issue is that package international-summary-statistics has a harvest object associated with it on the UI and SOLR but that harvest object has no package_id in the DB. This discrepancy makes duplicate dataset keeps coming back after dedupe process.

Manually api calls have fixed the packages for the the harvest source federal-reserve.

To address the issue, we need to improve dedupe process to detect and delete packages associated with harvest objects that has no pacakge_ids.