Migration Improvements #10
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…metadata and linked agents from MODS.
|
@dannylamb testing this now and getting some strange results. I ran the migration at the command line and it failed, giving this error: Here's a screenshot of the "Messages" list for the files migration:
The weird thing is that the failed object, In case it helps, here's the RELS-EXT for 'ir:1': As you can see, this object is embargoed, but I'm running the migration as fedoraAdmin, which should be able to view the object. And I'm attaching my .yml files. Also, looks like none of the media came over. I assume that's because the OBJ migration failed and it never got to the OBJ Media migration. This 7.x repo is accessible to the outside world, let me know if you want access. |
This was referenced Dec 5, 2018
Closed
|
This supercedes #11 |
|
OK, I rolled in all the AUDIT changes into this, because I would've had to rewrite all the xpaths and I already did a ton of work converting that stuff to solr here. Let me cook up better testing instructions. |
|
Will do. |
|
e.g. update the readme |
|
@mjordan Should work for you now. I think it may even still be pointed at your Islandora 7.x 😁 I'll need to change that too when updating the README. |
|
@mjordan You'll have to pull down Islandora/documentation#105 and https://github.com/Islandora-CLAW/islandora_demo/pull/9, re-import those features, and then disable the derivative contexts so they don't fire during the migration. Then you should get everything in your basic image collection, including the collection itself. It'll even work with the test pdf you shoved in there. |
|
Which contexts are derivative contexts other than "Image Service File"? |
|
@mjordan You should see two: |
First pass at DSID -> Term mapping
…7x_claw into dannylamb-linked-agents
|
Ok, so from a fresh box, you'll need set up your git:
Then use
Then pull down all three PRs. Due to weird permissions issues I had to stash before
Then import the config
I ran the migrations through drush with I did not disable any contexts, and although I got a bunch of these everything was fine. Those should go away (and you'll have to disable the two contexts that generate derivatives) when Islandora/documentation#106 lands. I got 22 items pulled over with 152 files and media. |
|
Spinning up a fresh box now. |
|
@dannylamb Yes! All worked as far as I can tell. Here's one of the resulting "Media" lists:
I also got a bunch of Nice work! |
|
Should we be persisting the source object's PID during the migration, as per Islandora/documentation#822? |
|
@mjordan yeah i'd say so. let me see if there's a good place to stuff it in the metadata profile. |
|
@mjordan Also, i'll make a ticket for those |
|
|
|
@mjordan Ok, if you pull in these changes, you can get your PIDS as
|
|
Very cool will test. My only observation about using |
|
For the record:
|
|
@mjordan Yes, in its current state, the code will populate the field with both the PID and a MODS local identifier if present. If we want to reserve one just for PIDs from 7.x and make it single valued, we can do that, np. It would certainly make it easier to do |
|
@mjordan Ok, I updated the Pull down changes to both PRs, and then
And then
|
|
👍 The new PID field is not showing up for me on the View tab, but it is present, and populated, in the Edit tab:
I won't be at tomorrow's CLAW call, but it might be good to show this to the group to make sure everyone's OK with having a separate field. Thanks, this is super impressive. |
|
LOL it also swaps out Preservation Master for Original File, so there's a lot going on. I'll be happy to demo it. |
|
Tested this for @dannylamb, and only item different from his instructions above is that we figured out (via chatting with each other), we need to add Steps: (Local machine)
(Virtual machine)
Derivatives failed
Success!
|
Not sure if all these are meaningful. Not 100% sure how things work from being away. ...and those two final error messages. Not sure if they are giant red flags or not. |
|
Well when you put it like that @ruebot I could at least squash that warning from flysystem with a The ERROR isn't a big deal because you're getting all the datastreams over, but it shouldn't be happening. It definitely comes down to how we've got an action and context configured. I'd have to pull the thread a little further to pinpoint exactly where. |
|
@ruebot @mjordan So that wall of warnings is actually handled if I pull in some upstream changes from the flysystem stream wrapper: twistor/flysystem-stream-wrapper#9 I'll see what it takes to get those packages in properly. I don't think it's pulled in by us directly, but through the flysystem module. |
|
So this works (sort of) and we can merge it in before it gets any bigger, but we need to
|
|
@whikloj @ruebot I've asked @twistor to slice a new release of I was getting the ERRORs other folks are seeing, but now can't seem to recreate them. I'll have to blow away my environment and start from scratch. I noticed that I hadn't disabled the As for those negative unprocessed numbers, I tried getting the same info from the command line to validate it and got root@claw:/var/www/html/drupal/web/modules/contrib# drush migrate:status --group islandora_7x
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
[warning] array_diff(): Argument #1 is not an array Islandora.php:221
---------------------------------- ----------------------- -------- ------- ---------- ------------- ---------------------
Group Migration ID Status Total Imported Unprocessed Last Imported
---------------------------------- ----------------------- -------- ------- ---------- ------------- ---------------------
Islandora Objects (islandora_7x) islandora_7x_tags Idle 6 6 0 2018-12-20 13:07:49
Islandora Objects (islandora_7x) islandora_audit_file Idle 22 22 0 2018-12-20 13:08:01
Islandora Objects (islandora_7x) islandora_person Idle 22 0 22 2018-12-20 13:08:05
Islandora Objects (islandora_7x) islandora_corporate Idle 22 0 22 2018-12-20 13:08:08
Islandora Objects (islandora_7x) islandora_subject Idle 22 38 -16 2018-12-20 13:08:21
Islandora Objects (islandora_7x) islandora_geographic Idle 22 18 4 2018-12-20 13:08:33
Islandora Objects (islandora_7x) islandora_objects Idle 22 22 0 2018-12-20 13:08:42
Islandora Objects (islandora_7x) islandora_audit_media Idle 35 35 0 2018-12-20 13:08:59
Islandora Objects (islandora_7x) islandora_files Idle 0 152 -152 2018-12-20 13:10:23
Islandora Objects (islandora_7x) islandora_media Idle 0 152 -152 2018-12-20 13:11:58
---------------------------------- ----------------------- -------- ------- ---------- ------------- ---------------------
Those warnings point to https://github.com/dannylamb/migrate_7x_claw/blob/linked-agents/src/Plugin/migrate/source/Islandora.php#L221. I'll revisit that logic. |
|
@dannylamb of course. So the This should probably default to But in the case of Media and Files we should define it as Neither of those two options are defined, which is why it is failing to perform a count. |
|
So is everyone okay if I merge this and then we work to fix the underlying Islandora Source Plugin problems? |
whikloj
approved these changes
Dec 21, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves Islandora/documentation#905
See README in this PR for installation and testing instructions. There's a zip batch included if you need sample objects for 7.x