Use Case: UI report of migration to show all items have been migrated

[DonRichards]

A 1-to-1 GUI report that shows a comparison of objects ingested into CLAW vs objects found in the original site.

Using a map of the original PID URL to it's new counterpart on the CLAW instance. As a count is referred as second step for the object should be checked for validity (ie. identify on images) and check the MODS values are in the expected RDF field.

• Writing this while at icampCA 18

[mjordan]

@DonRichards this is a great idea. I think it's important that users who don't have access to the Islandora command line are able to confirm that the migrations were successful, since doing so may help them through a potentially traumatic experience 😃.

It would be very useful to also get a report of which objects didn't migrate. I think the Migrate framework stores failures or errors, so it might be possible to get a report from that data.

This probably should be implemented as a separate, optional module, if possible, focusing only on reporting.

[mjordan]

Possibly related issue: #822.

[mjordan]

@DonRichards since the iCamp I've put together a basic web crawler at https://github.com/mjordan/islandora_crawler that gets the object URLs from a 7.x repo by scanning collection browse pages. I note it here because one way to compare what is in the source repo with what was migrated to the target repo would be to do a similar crawl on both and compare the numbers. There are a lot of other ways to do it, but I though it might be useful to note that method here.

In this case, we could provide a GUI tool to do the crawling/comparison.

[DonRichards]

A second part of tool could be a module on Islandora 7 that creates a list of what CLAW should expect to import like an API call.

[DonRichards]

@mjordan I like the idea of a gui. Any chance that got some traction sense November? I think that would be really useful at Islandoracon.

[mjordan]

@DonRichards I put some time into #1021 but got tangled up in how to handle assigning taxonomies to the test "nodes" and never returned to that effort. It would be cool to have something like your graph above to show at the Conference. What would it take to do that?

[DonRichards]

@mjordan
I think it could be pretty simple. Get a list of PIDs of collections and a list of object PIDs with their hashes and hash types by object and a blank column for the 8.x UUID.

Create a db table for scraped 7x info

• Get a list of all of the 7.x PIDs by collection. Expanding on the comment #1021 add an additional blank column for the new 8.x "identifier" and also going to need the hash types/values for each PID.

+-----------------+--------------------------------------+---------------+------------------------------------------+----------+
|     7.x PID     |               8.x UUID               | 7.x hash type |                 7.x hash                 | Verified |
+-----------------+--------------------------------------+---------------+------------------------------------------+----------+
| volvoices:23023 | e6fbe890-5c89-11e9-8647-d663bd873d93 | MD5           | ff40834aae1ebe8658e6978af6de9c07         | y        |
| volvoices:23023 | e6fbe890-5c89-11e9-8647-d663bd873d93 | MD5           | d5a3320d2b097c4320f6109aea6f8b4d         | y        |
| volvoices:23023 | e6fbe890-5c89-11e9-8647-d663bd873d93 | SHA1          | cf23df2207d99a74fbe169e3eba035e633b65d94 | y        |
| volvoices:23025 |                                      | MD5           | bf40834aae1ebe8658e6978af6de9c07 n       |          |
| volvoices:23025 |                                      | MD5           | j5a3320d2b097c4320f6109aea6f8b4d         | n        |
| volvoices:23025 |                                      | MD5           | lba3320g2b092c4230f6109aea6f8b7f         | n        |
+-----------------+--------------------------------------+---------------+------------------------------------------+----------+

• After the object is created and the datastreams are added the final step is to append the new 8.x identifier to the table.

Looking at the image I originally posted.

• Grey background of the bar (for blue) is total number of objects expected to be ingested from 7.x.
• blue is completed ingested into 8.x compared to the total expected to be ingested.
• Grey background of the bar (for red) is total number of objects already ingested as part of the migration.
• red is verified total number of objects in 8.x that have a db entry for a 7.x migration.
  • "verified" means the equivalent hash value (MD5, sha1, etc) is checked against the hash value originally harvested from 7.x. If it's a match, update the "hash verified" to "y".
  • This bar represents only what is currently in 8.x (values with both a 7.x PID & a 8.x UUID.
    • Specifically the Total with both minus the total with both and verified not set to "y"

The last step would require a process to run either after each ingest or independently to check the hash values match and update the table.

After everything is complete (100%) change the bars in a dramatic way to show everything is complete. Like showing total number on both side boldly. Perhaps adding instructions on how to independently verify.

[DonRichards]

@mjordan Sorry it took so long to respond. Got sidetracked.