Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement recast service #44

Merged
merged 8 commits into from
Feb 21, 2019
Merged

Implement recast service #44

merged 8 commits into from
Feb 21, 2019

Conversation

whikloj
Copy link
Member

@whikloj whikloj commented Mar 14, 2018

GitHub Issue: Islandora/documentation/issues/800

Depends on: Islandora/Crayfish-Commons#16

What does this Pull Request do?

This PR implements a new API-X extension service inside of Crayfish. The service binds to resources of rdf:type ldp:RdfSource and rewrites the request to add Fedora URIs for Drupal URIs it finds in the Gemini DB.

What's new?

  1. Moves the GeminiClient out of Milliner and into Crayfish-Commons to use it in Recast as well.
  2. Adds a new findUrls service to UrlMapper in Gemini.
  3. Adds a new /by_uri endpoint with a getByUri function which uses the above search.
  4. Adds the new Recast service, the service is normally accessible at the svc:recast uri via API-X
  5. Adds/updates tests as necessary
  • Does this change require documentation to be updated? maybe
  • Does this change add any new dependencies? no
  • Does this change require any other modifications to be made to the repository (ie. Regeneration activity, etc.)? no
  • Could this change impact execution of existing code? no

How should this be tested?

As API-X with Fedora 5 is still broken these are new testing instructions.

Pull down a claw-playbook (or delete the contents of your roles/external directory on an existing one) and edit the requirements.yml to make the following change:

> git diff requirements.yml 
diff --git a/requirements.yml b/requirements.yml
index e685417..ee777d8 100644
--- a/requirements.yml
+++ b/requirements.yml
@@ -57,9 +57,9 @@
   name: Islandora-Devops.cantaloupe
   version: 0.0.1
 
-- src: https://github.com/Islandora-Devops/ansible-role-crayfish
+- src: https://github.com/whikloj/ansible-role-crayfish
   name: Islandora-Devops.crayfish
-  version: 0.0.8
+  version: issue-800
 
 - src: https://github.com/Islandora-Devops/ansible-role-drupal-openseadragon
   name: Islandora-Devops.drupal-openseadragon

Then vagrant up

Once done you should have a completely functioning Claw instance.

Create a new repository object as a collection, then create a new repository object (as whatever) and make it part of the first collection.

Find this second object's Fedora URL. I normally just dig through all the URIs in Fedora but you can also look in the Gemini DB if you want.

Once you have your Fedora URI

> curl -H"Authorization: Bearer islandora" http://localhost:8080/fcrepo/rest/41/73/e4/5e/4173e45e-6cf8-4e83-8189-788152aafd15
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix fedora:  <http://fedora.info/definitions/v4/repository#> .
@prefix ldp:  <http://www.w3.org/ns/ldp#> .
@prefix dcterms:  <http://purl.org/dc/terms/> .

<http://localhost:8080/fcrepo/rest/41/73/e4/5e/4173e45e-6cf8-4e83-8189-788152aafd15>
        rdf:type                    fedora:Container ;
        rdf:type                    fedora:Resource ;
        rdf:type                    <http://pcdm.org/models#Object> ;
        fedora:lastModifiedBy       "bypassAdmin" ;
        <http://schema.org/dateModified>  "2019-02-04T17:29:28+00:00"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        <http://schema.org/author>  <http://localhost:8000/user/1?_format=jsonld> ;
        <http://schema.org/sameAs>  "http://localhost:8000/node/2?_format=jsonld" ;
        <http://schema.org/dateCreated>  "2019-02-04T17:29:15+00:00"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        dcterms:extent              "1 item" ;
        fedora:createdBy            "bypassAdmin" ;
        fedora:lastModified         "2019-02-04T17:29:28.963Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        fedora:created              "2019-02-04T17:29:28.963Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        <http://pcdm.org/models#memberOf>  <http://localhost:8000/node/1?_format=jsonld> ;
        dcterms:title               "A basic image"@en ;
        rdf:type                    ldp:RDFSource ;
        rdf:type                    ldp:Container .

Then using that URI as your Apix-Ldp-Resource call the Recast service and note the change in the memberOf predicate values.

> curl -H"Authorization: Bearer islandora" -H"Apix-Ldp-Resource: http://localhost:8080/fcrepo/rest/41/73/e4/5e/4173e45e-6cf8-4e83-8189-788152aafd15" http://localhost:8000/recast/ 
@prefix ns0: <http://fedora.info/definitions/v4/repository#> .
@prefix schema: <http://schema.org/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix dc: <http://purl.org/dc/terms/> .
@prefix ns1: <http://pcdm.org/models#> .

<http://localhost:8080/fcrepo/rest/41/73/e4/5e/4173e45e-6cf8-4e83-8189-788152aafd15>
  a <http://fedora.info/definitions/v4/repository#Container>, <http://fedora.info/definitions/v4/repository#Resource>, <http://pcdm.org/models#Object>, <http://www.w3.org/ns/ldp#RDFSource>, <http://www.w3.org/ns/ldp#Container> ;
  ns0:lastModifiedBy "bypassAdmin" ;
  schema:dateModified "2019-02-04T17:29:28+00:00"^^xsd:dateTime ;
  schema:author <http://localhost:8000/user/1?_format=jsonld> ;
  schema:sameAs "http://localhost:8000/node/2?_format=jsonld" ;
  schema:dateCreated "2019-02-04T17:29:15+00:00"^^xsd:dateTime ;
  dc:extent "1 item" ;
  ns0:createdBy "bypassAdmin" ;
  ns0:lastModified "2019-02-04T17:29:28.963Z"^^xsd:dateTime ;
  ns0:created "2019-02-04T17:29:28.963Z"^^xsd:dateTime ;
  ns1:memberOf <http://localhost:8000/node/1?_format=jsonld>, <http://localhost:8080/fcrepo/rest/73/ac/45/73/73ac4573-91b7-4ba5-8ed6-9376d6e2e8ca> ;
  dc:title "A basic image"@en .

http://localhost:8000/recast/ calls the default action of add and is the same as calling http://localhost:8000/recast/add. Try http://localhost:8000/recast/replace to remove the corresponding Drupal uris

> curl -H"Authorization: Bearer islandora" -H"Apix-Ldp-Resource: http://localhost:8080/fcrepo/rest/41/73/e4/5e/4173e45e-6cf8-4e83-8189-788152aafd15" http://localhost:8000/recast/replace
@prefix ns0: <http://fedora.info/definitions/v4/repository#> .
@prefix schema: <http://schema.org/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix dc: <http://purl.org/dc/terms/> .
@prefix ns1: <http://pcdm.org/models#> .

<http://localhost:8080/fcrepo/rest/41/73/e4/5e/4173e45e-6cf8-4e83-8189-788152aafd15>
  a <http://fedora.info/definitions/v4/repository#Container>, <http://fedora.info/definitions/v4/repository#Resource>, <http://pcdm.org/models#Object>, <http://www.w3.org/ns/ldp#RDFSource>, <http://www.w3.org/ns/ldp#Container> ;
  ns0:lastModifiedBy "bypassAdmin" ;
  schema:dateModified "2019-02-04T17:29:28+00:00"^^xsd:dateTime ;
  schema:author <http://localhost:8000/user/1?_format=jsonld> ;
  schema:sameAs "http://localhost:8000/node/2?_format=jsonld" ;
  schema:dateCreated "2019-02-04T17:29:15+00:00"^^xsd:dateTime ;
  dc:extent "1 item" ;
  ns0:createdBy "bypassAdmin" ;
  ns0:lastModified "2019-02-04T17:29:28.963Z"^^xsd:dateTime ;
  ns0:created "2019-02-04T17:29:28.963Z"^^xsd:dateTime ;
  ns1:memberOf <http://localhost:8080/fcrepo/rest/73/ac/45/73/73ac4573-91b7-4ba5-8ed6-9376d6e2e8ca> ;
  dc:title "A basic image"@en .

Additional Notes:

Interested parties

@Islandora-CLAW/committers

@dannylamb
Copy link
Contributor

@whikloj Just a guess from Travis's output, but maybe try generating a composer.json using PHP 5.6? #42 (comment)

@whikloj
Copy link
Member Author

whikloj commented Mar 14, 2018

@dannylamb This should "work" now, but the Gemini and Recast tests will fail as they rely on the Crayfish-Commons PR. So once that is merged, then these tests could be restarted.

@Natkeeran
Copy link
Contributor

@whikloj

Ran the script and getting an exception thrown. The service is not being registered.

phpunit/phpunit suggests installing ext-xdebug (*)
Writing lock file
Generating autoload files
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   252  100   252    0     0    806      0 --:--:-- --:--:-- --:--:--   805
 * Reloading Apache httpd web server apache2                                                                                                                                           * 
Sleep for 5 while apix-loader reloads
HTTP/1.1 500 Server Error
Content-Type: text/plain; charset=ISO-8859-1
Transfer-Encoding: chunked

org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://localhost:8000/recast/ with statusCode: 500
	at org.apache.camel.component.http4.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:312)
	at org.apache.camel.component.http4.HttpProducer.process(HttpProducer.java:207)
	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
	at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
	at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:117)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
	at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
	at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:117)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
	at org.apache.camel.component.jetty.CamelContinuationServlet.doService(CamelContinuationServlet.java:206)
	at org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:74)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
	at org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:146)
	at org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:43)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:748)

@dannylamb
Copy link
Contributor

@whikloj can you hit the button on this one? i'd like to test this out.

@dannylamb
Copy link
Contributor

i am resisting the urge to hit it myself and am trying not to implicate myself in the pull 😁

@whikloj
Copy link
Member Author

whikloj commented Feb 2, 2019

@dannylamb so in the Crayfish config.yaml files ../syn-settings.xml means move up from the cfg directory to the microservice root (ie. Houdini) so you need to go ../../syn-settings.xml to have it in the Crayfish directory 🤦‍♂️

Couple changes to match up with the changes in Crayfish but this still works, though for those things not in Gemini I can't do anything so we might want to consider whether more things should be mapped using it.

@whikloj
Copy link
Member Author

whikloj commented Feb 3, 2019

well this has become completely screwed. I'll see if I can get all the pieces together again, but don't bother trying this one.

Move Milliner to use GeminiClient in Crayfish-Commons
@whikloj whikloj closed this Feb 3, 2019
@whikloj whikloj reopened this Feb 4, 2019
@codecov
Copy link

codecov bot commented Feb 4, 2019

Codecov Report

Merging #44 into master will decrease coverage by 2.89%.
The diff coverage is 83.01%.

Impacted file tree graph

@@            Coverage Diff             @@
##             master     #44     +/-   ##
==========================================
- Coverage       100%   97.1%   -2.9%     
- Complexity      122     140     +18     
==========================================
  Files             9       9             
  Lines           570     621     +51     
==========================================
+ Hits            570     603     +33     
- Misses            0      18     +18
Impacted Files Coverage Δ Complexity Δ
Milliner/src/Service/MillinerService.php 100% <ø> (ø) 48 <0> (ø) ⬇️
Gemini/src/UrlMapper/UrlMapper.php 100% <100%> (ø) 8 <1> (+1) ⬆️
Recast/src/Controller/RecastController.php 81.11% <81.11%> (ø) 24 <24> (?)
Gemini/src/Controller/GeminiController.php 98.21% <90.9%> (-1.79%) 19 <5> (+5)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d81724e...3080da2. Read the comment docs.

@whikloj
Copy link
Member Author

whikloj commented Feb 4, 2019

@dannylamb Codecov is kicking me for boosting it to 100% previously 🤦‍♂️ but otherwise this is good. I updated the testing instructions above.

@dannylamb
Copy link
Contributor

You flew too close to the 🌞 @whikloj

I can give this a whirl then 👍

@dannylamb
Copy link
Contributor

@whikloj Both add and replace work out just fine. JSONLD looks good:

[
   {
      "@id":"http:\/\/localhost:8080\/fcrepo\/rest\/47\/f9\/8e\/82\/47f98e82-6b49-49d9-9d5e-1574a12e6ab6",
      "@type":[
         "http:\/\/fedora.info\/definitions\/v4\/repository#Container",
         "http:\/\/pcdm.org\/models#File",
         "http:\/\/fedora.info\/definitions\/v4\/repository#Resource",
         "http:\/\/www.w3.org\/ns\/ldp#RDFSource",
         "http:\/\/www.w3.org\/ns\/ldp#Container"
      ],
      "http:\/\/fedora.info\/definitions\/v4\/repository#lastModifiedBy":[
         {
            "@value":"bypassAdmin"
         }
      ],
      "http:\/\/schema.org\/dateModified":[
         {
            "@value":"2019-02-08T14:33:46+00:00",
            "@type":"http:\/\/www.w3.org\/2001\/XMLSchema#dateTime"
         }
      ],
      "http:\/\/schema.org\/author":[
         {
            "@id":"http:\/\/localhost:8000\/user\/1?_format=jsonld"
         }
      ],
      "http:\/\/schema.org\/sameAs":[
         {
            "@value":"http:\/\/localhost:8000\/sites\/default\/files\/2019-02\/2-Service%20File.jpg"
         }
      ],
      "http:\/\/schema.org\/dateCreated":[
         {
            "@value":"2019-02-08T14:33:46+00:00",
            "@type":"http:\/\/www.w3.org\/2001\/XMLSchema#dateTime"
         }
      ],
      "http:\/\/www.ebu.ch\/metadata\/ontologies\/ebucore\/ebucore#width":[
         {
            "@value":"1494",
            "@type":"http:\/\/www.w3.org\/2001\/XMLSchema#int"
         }
      ],
      "http:\/\/www.ebu.ch\/metadata\/ontologies\/ebucore\/ebucore#hasMimeType":[
         {
            "@value":"image\/jpeg"
         }
      ],
      "http:\/\/fedora.info\/definitions\/v4\/repository#createdBy":[
         {
            "@value":"bypassAdmin"
         }
      ],
      "http:\/\/fedora.info\/definitions\/v4\/repository#lastModified":[
         {
            "@value":"2019-02-08T14:33:49.092Z",
            "@type":"http:\/\/www.w3.org\/2001\/XMLSchema#dateTime"
         }
      ],
      "http:\/\/fedora.info\/definitions\/v4\/repository#created":[
         {
            "@value":"2019-02-08T14:33:49.092Z",
            "@type":"http:\/\/www.w3.org\/2001\/XMLSchema#dateTime"
         }
      ],
      "http:\/\/www.ebu.ch\/metadata\/ontologies\/ebucore\/ebucore#height":[
         {
            "@value":"1074",
            "@type":"http:\/\/www.w3.org\/2001\/XMLSchema#int"
         }
      ],
      "http:\/\/www.w3.org\/1999\/02\/22-rdf-syntax-ns#label":[
         {
            "@value":"2-Service File.jpg",
            "@language":"en"
         }
      ],
      "http:\/\/pcdm.org\/models#fileOf":[
         {
            "@id":"http:\/\/localhost:8000\/node\/2?_format=jsonld"
         },
         {
            "@id":"http:\/\/localhost:8080\/fcrepo\/rest\/e5\/e2\/05\/1a\/e5e2051a-4df7-4dfb-bd5f-499089dea7b3"
         }
      ],
      "http:\/\/purl.org\/dc\/terms\/title":[
         {
            "@value":"2-Service File.jpg",
            "@language":"en"
         }
      ]
   }
]

But turtle is re-writing namespaces to ns01, ns02, etc... Since it's the default return format, it'd be nice to sort that out. EasyRDF is always a pain in the arse, but I'm guessing you could stash the namespaces from what you receive and re-use them when serializing out. If that turns out to be an ordeal we can spool off another ticket, because what you have works just fine with jsonld.

@whikloj
Copy link
Member Author

whikloj commented Feb 21, 2019

@dannylamb could we put this in and then I'll work on the serialization issue after. Just to get this in and available.

Copy link
Contributor

@dannylamb dannylamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, Let's do this 🚀

@dannylamb dannylamb merged commit 6ab0da6 into Islandora:master Feb 21, 2019
@whikloj whikloj deleted the issue-800 branch February 21, 2019 17:26
@whikloj whikloj mentioned this pull request Mar 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants