WIP: jsonld ver. 1.1 exports #140

dbkeator · 2019-07-01T23:27:57Z

No description provided.

dbkeator · 2019-07-01T23:50:49Z

@cmaumet This is the branch I'm working with for the JSON-LD 1.1 exports. Essentially there are 2 parts to it, a call to Core.createDefaultJSONLDcontext() which currently only adds the namespaces from Constants.py and NIDM experiment terms (it doesn't yet get all the terms from the NIDM-E OWL files like it should) and then it adds any additional prefixes that are in the current NIDM-E graph in the event someone loaded a NIDM-E graph which includes terms not in the Contants.py namespaces and/or (eventually) the NIDM-E OWL files (see: https://github.com/incf-nidash/PyNIDM/blob/master/nidm/experiment/Core.py#L433).

The test NIDM TTL files I'm using are here (for example): https://github.com/dbkeator/simple2_NIDM_examples/blob/master/datasets.datalad.org/abide/RawDataBIDS/CMU_a/nidm.ttl

Note, my current NIDM-E reader doesn't read in the brain volume data from the above NIDM TTL file it only reads in the NIDM-E specific project->session->acquisition hierarchy. That's a WIP.

The utility I use to test converting the NIDM TTL files -> JSON-LD is here: https://github.com/incf-nidash/PyNIDM/blob/master/nidm/experiment/tools/nidm_utils.py

You simply run nidm_utils.py jsonld -nl [path to NIDM.ttl file] and it will save out a file of the same name with the json extention.

The error I'm getting when running a conversion is shown below. This started when I moved to using pyld instead of rdflib-jsonld:

  File "/anaconda3/lib/python3.7/site-packages/pyld/jsonld.py", line 677, in compact
    expanded = self.expand(input_, options)
  File "/anaconda3/lib/python3.7/site-packages/pyld/jsonld.py", line 835, in expand
    expanded = self._expand(active_ctx, None, document, options, False)
  File "/anaconda3/lib/python3.7/site-packages/pyld/jsonld.py", line 2090, in _expand
    active_ctx, element['@context'], options)
  File "/anaconda3/lib/python3.7/site-packages/pyld/jsonld.py", line 2762, in _process_context
    code='invalid @version value')
pyld.jsonld.JsonLdError: ('Unsupported JSON-LD version: 1.1',)
Type: jsonld.UnsupportedVersion
Code: invalid @version value
Details: {'context': {'@version': '1.1', 'Acquisition': 'http://purl.org/nidash/nidm#Acquisition', 'AcquisitionObject': 'http://purl.org/nidash/nidm#AcquisitionObject', 'Age': 'http://ncicb.nci.nih.gov/xml/owl/EVS/Thesaurus.owl#Age', 'Anatomical': 'http://purl.org/nidash/nidm#Anatomical', 'DemographicsInstrument': 'http://purl.org/nidash/nidm#DemographicsInstrument', 'Diagnosis': 'http://ncitt.ncit.nih.gov/Diagnosis', 'DiffusionTensor': 'http://purl.org/nidash/nidm#DiffusionTensor', 'DiffusionWeighted': 'http://purl.org/nidash/nidm#DiffusionWeighted', 'FlowWeighted': 'http://purl.org/nidash/nidm#FlowWeighted', 'Functional': 'http://purl.org/nidash/nidm#Functional', 'Identifier': 'http://semanticscience.org/ontology/sio.owl#Identifier', 'MagneticResonanceImaging': 'http://purl.org/nidash/nidm#MagneticResonanceImaging', 'PhenotypicSex': 'http://purl.obolibrary.org/obo/pato#PhenotypicSex', 'Project': 'http://purl.org/nidash/nidm#Project', 'Session': 'http://purl.org/nidash/nidm#Session', 'StimulusResponseFile': 'http://purl.org/nidash/nidm#StimulusResponseFile', 'Structural': 'http://purl.org/nidash/nidm#Structural', 'Subject': 'http://semanticscience.org/ontology/sio.owl#Subject', 'T1Weighted': 'http://purl.org/nidash/nidm#T1Weighted', 'T2StarWeighted': 'http://purl.org/nidash/nidm#T2StarWeighted', 'T2Weighted': 'http://purl.org/nidash/nidm#T2Weighted', 'Task': 'http://purl.org/nidash/nidm#Task', 'URL': 'http://semanticscience.org/ontology/sio.owl#URL', 'activity': {'@id': 'prov:activity', '@type': '@id'}, 'adhd_index': 'https://github.com/dbkeator/nidm-local-terms/issues/57', 'afni': 'http://purl.org/nidash/afni#', 'agent': {'@id': 'prov:agent', '@type': '@id'}, 'ants': 'http://stnava.github.io/ANTs/', 'assessment-instrument': 'http://neurolog.unice.fr/ontoneurolog/v3.0/instrument.owl#assessment-instrument', 'association': {'@id': 'prov:qualifiedAssociation', '@type': '@id'}, 'author': 'http://www.w3.org/ns/dcat#author', 'b-value': 'http://purl.org/nidash/nidm#b-value', 'b-vector': 'http://purl.org/nidash/nidm#b-vector', 'bids': 'http://bids.neuroimaging.io/', 'birnlex': 'http://bioontology.org/projects/ontologies/birnlex/', 'birnlex_2152': 'http://bioontology.org/projects/ontologies/birnlex/birnlex_2152', 'birnlex_2199': 'http://bioontology.org/projects/ontologies/birnlex/birnlex_2199', 'creator': 'http://www.w3.org/ns/dcat#creator', 'crypto': 'http://id.loc.gov/vocabulary/preservation/cryptographicHashFunctions#', 'dc': 'http://purl.org/dc/elements/1.1/', 'dcat': 'http://www.w3.org/ns/dcat#', 'dct': 'http://purl.org/dc/terms/', 'dctypes': 'http://purl.org/dc/dcmitype/', 'description': 'http://purl.org/dc/terms/description', 'dicom': 'http://neurolex.org/wiki/Category:DICOM_term/', 'dx': 'http://ncitt.ncit.nih.gov/Diagnosis', 'endedAtTime': {'@id': 'prov:endedAtTime', '@type': 'xsd:dateTime'}, 'entity': {'@id': 'prov:entity', '@type': '@id'}, 'ethnicity': 'http://semanticscience.org/ontology/sio.owl#ethnicity', 'familyName': 'http://xmlns.com/foaf/0.1/familyName', 'file': 'http://semanticscience.org/ontology/sio.owl#file', 'filename': 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#filename', 'foaf': 'http://xmlns.com/foaf/0.1/', 'freesurfer': 'https://surfer.nmr.mgh.harvard.edu/', 'fsl': 'http://purl.org/nidash/fsl#', 'full4_iq': 'https://github.com/dbkeator/nidm-local-terms/issues/2', 'gender': 'https://ndar.nih.gov/api/datadictionary/v2/dataelement/gender', 'generation': {'@id': 'prov:qualifiedGeneration', '@type': '@id'}, 'givenName': 'http://xmlns.com/foaf/0.1/givenName', 'hadAcquisitionModality': 'http://purl.org/nidash/nidm#hadAcquisitionModality', 'hadImageContrastType': 'http://purl.org/nidash/nidm#hadImageContrastType', 'hadImageUsageType': 'http://purl.org/nidash/nidm#hadImageUsageType', 'hadPlan': {'@id': 'prov:hadPlan', '@type': '@id'}, 'hadRole': {'@id': 'prov:hadRole', '@type': '@id'}, 'handedness': 'http://purl.obolibrary.org/obo/handedness', 'hyper_impulsive': 'https://github.com/dbkeator/nidm-local-terms/issues/59', 'inattentive': 'https://github.com/dbkeator/nidm-local-terms/issues/58', 'instrument-based-assessment': 'http://neurolog.unice.fr/ontoneurolog/v3.0/instrument.owl#instrument-based-assessment', 'iq_measure': 'https://github.com/dbkeator/nidm-local-terms/issues/60', 'license': 'http://purl.org/dc/terms/license', 'ncicb': 'http://ncicb.nci.nih.gov/xml/owl/EVS/Thesaurus.owl#', 'ncit': 'http://ncitt.ncit.nih.gov/', 'ndar': 'https://ndar.nih.gov/api/datadictionary/v2/dataelement/', 'nfo': 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#', 'nidm': 'http://purl.org/nidash/nidm#', 'niiri': 'http://iri.nidash.org/', 'nlx': 'http://uri.neuinfo.org/nif/nifstd/', 'obo': 'http://purl.obolibrary.org/obo/', 'onli': 'http://neurolog.unice.fr/ontoneurolog/v3.0/instrument.owl#', 'owl': 'http://www.w3.org/2002/07/owl#', 'pato': 'http://purl.obolibrary.org/obo/pato#', 'performance_iq': 'https://github.com/dbkeator/nidm-local-terms/issues/4', 'prov': 'http://www.w3.org/ns/prov#', 'qc_anatomical_1': 'https://github.com/dbkeator/nidm-local-terms/issues/63', 'qc_rest_2': 'https://github.com/dbkeator/nidm-local-terms/issues/62', 'race': 'http://semanticscience.org/ontology/sio.owl#race', 'rdf': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#', 'rdfs': 'http://www.w3.org/2000/01/rdf-schema#', 'records': {'@container': '@type', '@id': '@graph'}, 'scr': 'http://scicrunch.org/resolver/', 'sio': 'http://semanticscience.org/ontology/sio.owl#', 'spm': 'http://purl.org/nidash/spm#', 'src_subject_id': 'https://ndar.nih.gov/api/datadictionary/v2/dataelement/src_subject_id', 'startedAtTime': {'@id': 'prov:startedAtTime', '@type': 'xsd:dateTime'}, 'title': 'http://purl.org/dc/dcmitype/title', 'usage': {'@id': 'prov:qualifiedUsage', '@type': '@id'}, 'vc': 'http://www.w3.org/2006/vcard/ns#', 'verbal_iq': 'https://github.com/dbkeator/nidm-local-terms/issues/3', 'wasAttributedTo': {'@id': 'prov:wasAttributedTo', '@type': '@id'}, 'xml': 'http://www.w3.org/XML/1998/namespace', 'xsd': 'http://www.w3.org/2001/XMLSchema#', 'xsi': 'http://www.w3.org/2001/XMLSchema-instance'}}

satra · 2019-07-02T01:04:50Z

@dbkeator - which version of pyld do you have?

cmaumet · 2019-07-02T07:04:19Z

Thank you @dbkeator! I'm pingging @natachaperez who will work on this with me.

dbkeator · 2019-07-02T16:11:26Z

@satra ver 1.0.5

satra · 2019-07-02T20:41:38Z

can you serialize your graph to turtle and the context to json and share it with me?

dbkeator · 2019-07-02T22:14:44Z

@satra Below a TTL file serialization and the corresponding JSON-LD serialization done with rdflib-jsonld:

TTL
JSON-LD

These files were created with the code in Core.serializeJSONLD() function except that instead of returning the rdflib-jsonld serialization I did the pyld trick you mentioned at OHBM (or at least I thought I did)...see the old serialization code commented out below with the return statement replaced with the following 3 lines of code (or in full fancy color: PR140)...which generates the pyld error with context...

#return rdf_graph_parse.serialize(format='json-ld', context=context, indent=4).decode('ASCII') g=rdf_graph_parse.serialize(format='json-ld', context=context, indent=4).decode('ASCII') import pyld as ld return ld.jsonld.compact(json.loads(g), context)

satra · 2019-07-03T02:24:49Z

nidm/experiment/Core.py

        #WIP: LOOK AT https://github.com/satra/nidm-jsonld
-        return rdf_graph_parse.serialize(format='json-ld', context=context, indent=4).decode('ASCII')
+        #return rdf_graph_parse.serialize(format='json-ld', context=context, indent=4).decode('ASCII')
+        g=rdf_graph_parse.serialize(format='json-ld', context=context, indent=4).decode('ASCII')


Suggested change

g=rdf_graph_parse.serialize(format='json-ld', context=context, indent=4).decode('ASCII')

g=rdf_graph_parse.serialize(format='json-ld', indent=4).decode('ASCII')

satra · 2019-07-03T02:25:20Z

nidm/experiment/Core.py

+        #return rdf_graph_parse.serialize(format='json-ld', context=context, indent=4).decode('ASCII')
+        g=rdf_graph_parse.serialize(format='json-ld', context=context, indent=4).decode('ASCII')
+        import pyld as ld
+        return ld.jsonld.compact(json.loads(g), context)


and make sure that:

context['@version'] = 1.1

So this worked. My main problem appeared to be that I was setting context['@Version'] to the string 1.1 not the floating point number. Anyway it serializes now...but my tests still fail because pyld needs to be added as a required package.

satra · 2019-07-03T02:26:13Z

left some comments.

satra · 2019-07-09T00:25:46Z

nidm/version.py

@@ -58,5 +58,5 @@
 #REQUIRES = ["prov", "rdflib", "graphviz", "pydotplus", "pydot", "validators", "requests", "fuzzywuzzy", "pygithub",
 #            "pandas", "pybids", "duecredit", "pytest", "graphviz", "click", "ontquery"]
 INSTALL_REQUIRES = ["prov", "rdflib", "graphviz", "pydotplus", "pydot", "validators", "requests", "fuzzywuzzy", "pygithub",
-            "pandas", "pybids", "duecredit", "pytest", "graphviz", "click", "ontquery","rdflib-jsonld"]
+            "pandas", "pybids", "duecredit", "pytest", "graphviz", "click", "ontquery","rdflib-jsonld","pyld"]


Suggested change

"pandas", "pybids", "duecredit", "pytest", "graphviz", "click", "ontquery","rdflib-jsonld","pyld"]

"pandas", "pybids", "duecredit", "pytest", "graphviz", "click", "ontquery","rdflib-jsonld","pyld", "neurdflib"]

just ensuring that the @tgbugs version of rdflib is installed

WIP: jsonld ver. 1.1 exports

68b7ff7

dbkeator assigned cmaumet Jul 1, 2019

satra reviewed Jul 3, 2019

View reviewed changes

dbkeator added 2 commits July 8, 2019 17:17

fixed problem with pyld json-ld 1.1 serialization

7d5828f

added pyld as a required package to nidm/version.py

3cf6887

dbkeator merged commit f741d2c into incf-nidash:master Jul 9, 2019

satra reviewed Jul 9, 2019

View reviewed changes

cmaumet mentioned this pull request Jul 23, 2019

WIP: To enrich NIDM-E with attributes from T1w.json and task-rest_bold.json files dbkeator/PyNIDM#4

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: jsonld ver. 1.1 exports #140

WIP: jsonld ver. 1.1 exports #140

dbkeator commented Jul 1, 2019

dbkeator commented Jul 1, 2019 •

edited by cmaumet

Loading

satra commented Jul 2, 2019

cmaumet commented Jul 2, 2019

dbkeator commented Jul 2, 2019

satra commented Jul 2, 2019

dbkeator commented Jul 2, 2019

satra Jul 3, 2019

satra Jul 3, 2019

dbkeator Jul 9, 2019

satra commented Jul 3, 2019

satra Jul 9, 2019

satra Jul 9, 2019

	g=rdf_graph_parse.serialize(format='json-ld', context=context, indent=4).decode('ASCII')
	g=rdf_graph_parse.serialize(format='json-ld', indent=4).decode('ASCII')

	"pandas", "pybids", "duecredit", "pytest", "graphviz", "click", "ontquery","rdflib-jsonld","pyld"]
	"pandas", "pybids", "duecredit", "pytest", "graphviz", "click", "ontquery","rdflib-jsonld","pyld", "neurdflib"]

WIP: jsonld ver. 1.1 exports #140

WIP: jsonld ver. 1.1 exports #140

Conversation

dbkeator commented Jul 1, 2019

dbkeator commented Jul 1, 2019 • edited by cmaumet Loading

satra commented Jul 2, 2019

cmaumet commented Jul 2, 2019

dbkeator commented Jul 2, 2019

satra commented Jul 2, 2019

dbkeator commented Jul 2, 2019

satra Jul 3, 2019

Choose a reason for hiding this comment

satra Jul 3, 2019

Choose a reason for hiding this comment

dbkeator Jul 9, 2019

Choose a reason for hiding this comment

satra commented Jul 3, 2019

satra Jul 9, 2019

Choose a reason for hiding this comment

satra Jul 9, 2019

Choose a reason for hiding this comment

dbkeator commented Jul 1, 2019 •

edited by cmaumet

Loading