Run third test migration of production Hydrus content to H2 stage #1226
Assignees
Comments
Skips due to item already having catkey (1,297)Skips due to item being in graveyard APO (4) |
|
The above items skipped to already having a catkey are from the following collections: |
|
I don't see anything here that gives me pause. @hannahfrost Do you think we need to check any individual items identified above from any of these collections? druid:by600sn5927 - Electronic Acquisitions |
|
@amyehodge Should I wait on this to begin importing on the H2 side? |
|
@mjgiarlo Nope. Go ahead! We can make changes next time if @hannahfrost sees anything in here she thinks should still be migrated despite having a catkey. But I think that's highly unlikely. |
|
@mjgiarlo This list includes some things that I wasn't expecting. I am going to reach out to some of those collections owners about whether they need those items in H2 or not. |
|
Please make sure that hydrus collections with the review workflow turned on continue to have the review workflow turned on when migrated to H2. |
New issue created for this: #1240 |
39 contributor-related failures |
|
@amyehodge all 3,825 works migrated just fine (with many of the same metadata warnings and placeholder behaviors as before re: contact emails and work types and abstracts). The above 39 items still have contributor-related problems, which caused them to have works without any work versions. This would be worth doing more digging into. |
|
Closing for now since the run is done and the analysis has been written up. |
|
The above contributor issues are all from the following three collections:
I looked at a few of these and am not immediately seeing anything. |
|
@amyehodge OK, I think the problem is a failure to validate departments as contributors. I thought we handled this in #1129 but our implementation there doesn't permit us to have department contributors. @jcoyne can you say more about the implementation in 1129? |
This was referenced Jun 23, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Prerequisites
sul-h2-stage:/data/h2-fileshas ~5.6T free storage space (for storing H2 uploaded files)sul-h2-stage:/data/hydrus-filesis mounted (with Hydrus production uploaded files)migrationbranch of Hydrus onmainso that it may be safely deployed to prod for the exportmigrations2branch of H2 onmainso it may be deployed cleanlySubtasks
migrationbranch of Hydrus to productioncollection_druids.txtwith collection druids on Hydrus prod (or create it locally and scp it, whatever)screensession)RAILS_ENV=production bin/export-collections > collections.jsonl~/collections.jsonlscreensession)RAILS_ENV=production bin/export-items > items.jsonl 2> item-errors.log~/items.jsonlmigrations2branch of H2 to stageRAILS_ENV=production bin/rails r 'Work.update_all(head_id: nil)' && RAILS_ENV=production bin/rails r 'Collection.update_all(head_id: nil)' && RAILS_ENV=production bin/rails r 'Collection.destroy_all' && RAILS_ENV=production bin/rails r 'ContactEmail.destroy_all' && RAILS_ENV=production bin/rails r 'Event.destroy_all' && RAILS_ENV=production bin/rails r 'ActiveStorage::Blob.destroy_all'[AbstractContributor.count, RelatedWork.count, RelatedLink.count, WorkVersion.count, ActiveStorage::Attachment.count, User.count, ActiveStorage::Blob.count, Collection.count, AttachedFile.count, Event.count, ContactEmail.count, Keyword.count, Work.count, CollectionVersion.count])[12687, 681, 1181, 4794, 12201, 1736, 12198, 177, 12201, 33314, 4967, 25462, 4844, 177][0, 0, 0, 0, 0, 1736, 0, 0, 0, 0, 0, 0, 0, 0]RAILS_ENV=production bin/rails r 'puts "Users: #{User.count}\nContactEmails: #{ContactEmail.count}\nEvents: #{Event.count}\nRelatedLinks: #{RelatedLink.count}\nCollections: #{Collection.count}\nCollectionVersions: #{CollectionVersion.count}"' | tee -a migrate-collections.logRAILS_ENV=production bin/migrate-collections ~/collections.jsonl 2>&1 | tee -a migrate-collections.logRAILS_ENV=production bin/rails r 'puts "Users: #{User.count}\nContactEmails: #{ContactEmail.count}\nEvents: #{Event.count}\nRelatedLinks: #{RelatedLink.count}\nCollections: #{Collection.count}\nCollectionVersions: #{CollectionVersion.count}"' | tee -a migrate-collections.logRAILS_ENV=production bin/rails r 'puts "Users: #{User.count}\nAuthors: #{Author.count}\nContributors: #{Contributor.count}\nContactEmails: #{ContactEmail.count}\nEvents: #{Event.count}\nKeywords: #{Keyword.count}\nRelatedLinks: #{RelatedLink.count}\nRelatedWorks: #{RelatedWork.count}\nWorks: #{Work.count}\nWorkVersions: #{WorkVersion.count}"' | tee -a migrate-items.logRAILS_ENV=production bin/migrate-items ~/items.jsonl 0 /data/hydrus-files migration_item_type_updates.csv 2>&1 | tee -a migrate-items.logRAILS_ENV=production bin/rails r 'puts "Users: #{User.count}\nContributors: #{Contributor.count}\nContactEmails: #{ContactEmail.count}\nEvents: #{Event.count}\nKeywords: #{Keyword.count}\nRelatedLinks: #{RelatedLink.count}\nRelatedWorks: #{RelatedWork.count}\nWorks: #{Work.count}\nWorkVersions: #{WorkVersion.count}"' | tee -a migrate-items.logThe text was updated successfully, but these errors were encountered: