Remap oai_dc fields dc:type and dc:date

[pdurbin]

What this PR does / why we need it:

The oai_dc export and harvesting format has had the following fields remapped:

• dc:type was mapped to the field "Kind of Data". Now it is hard-coded to the word "Dataset".
• dc:date was mapped to the field "Production Date" when available and otherwise to "Publication Date". Now it is mapped the field “Publication Date” or the field used for the citation date, if set (see Set Citation Date Field Type for a Dataset).
• dc:rights was not mapped to anything. Now it is mapped (when available) to terms of use, restrictions, and license. Deferred until this issue:
  • Access Rights metadata in OpenAIRE metadata export is being misapplied #5920

As these are backward incompatible changes, they have been emphasized in the release note snippet.

Which issue(s) this PR closes:

• Closes Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting #8129

Special notes for your reviewer:

Should these backward-incompatible changes be hidden behind a feature flag?

Suggestions on how to test this:

See rules above under "what this PR does". Also, below are some examples of before and after.

Before

• dc:date is mapped to the field "Production Date" when available and otherwise to "Publication Date".
• We see "survey" under dc:type because it was entered in the "Kind of Data" field. dc:type will be absent if "Kind of Data" isn't filled in.

<oai_dc:dc xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
  <dc:title>Darwin's Finches</dc:title>
  <dc:identifier>https://doi.org/10.5072/FK2/QHIUBQ</dc:identifier>
  <dc:creator>Finch, Fiona</dc:creator>
  <dc:publisher>Root</dc:publisher>
  <dc:description>Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds.</dc:description>
  <dc:subject>Medicine, Health and Life Sciences</dc:subject>
  <dc:date>2024-09-10</dc:date>
  <dc:type>survey</dc:type>
</oai_dc:dc>

After

• dc:date is mapped the field "Publication Date" or the field used for the citation date, if set (see Set Citation Date Field Type for a Dataset).
• dc:type is hard-coded to "Dataset".

<oai_dc:dc xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
  <dc:title>Darwin's Finches</dc:title>
  <dc:identifier>https://doi.org/10.5072/FK2/QHIUBQ</dc:identifier>
  <dc:creator>Finch, Fiona</dc:creator>
  <dc:publisher>Root</dc:publisher>
  <dc:description>Darwin's finches (also known as the Galápagos finches) are a group of about fifteen species of passerine birds.</dc:description>
  <dc:subject>Medicine, Health and Life Sciences</dc:subject>
  <dc:date>2024-09-10</dc:date>
  <dc:type>Dataset</dc:type>
</oai_dc:dc>

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No.

Is there a release notes update needed for this change?:

Yes, included.

Additional documentation:

None.

[pdurbin]

@jggautier heads up that this relates to this issue in that we are now adding "dc:rights" to the oai_dc export/harvesting format:

• Feature Request/Idea: Make sure that for datasets published with CC0 waiver or standard license, metadata exports include the waiver or license #8798

[pdurbin]

@tcoupin I'm requesting a review from you because you modified the dc:date login in the following pull request and I changed it (as explained above):

• oai_dc export: use publication date if production date is not set #8733

[qqmyers]

Re: dc:date - should it be mapped to the same field as https://guides.dataverse.org/en/latest/api/native-api.html#set-citation-date-field-type-for-a-dataset ? That is publicationDate by default.

[coveralls]

coverage: 20.735%. remained the same
when pulling 01e266c on 8129-harvesting
into 4143031 on develop.

[pdurbin]

@qqmyers well, Publication Date is what @philippconzett asked for in the issue (#8129).

[qqmyers]

@philippconzett 's notes also point out that this date potentially going to be interpreted as the citation date. Since we allow configuring that in the local installation, it seems like it could be confusing to hardcode it for harvesting. If the harvester used the field from that setting, citations would be consistent in the local display and harvesting sites, and it would default to publicationDate as requested in the issue.

[pdurbin]

I don't have a strong opinion about it.

[philippconzett]

I think @qqmyers's suggestion for dc:date makes sense.

[plecor]

I've taken @tcoupin's role on Dataverse issues, so I am looking at this for him.

Part of the context for the change he implemented (mapping dc:date to Publication Date if Production Date is empty) was that when Dataverse harvests another OAI-PMH repo, dc:date is mapped to productionDate and this production date is then used in the citation of the harvested dataset. #8733 and #8732 were both part of an effort to guarantee the coherence between citation dates when harvesting another Dataverse.

So I agree with @qqmyers's suggestion on dc:date.

Hardcoding dc:type to Dataset would certainly simplify things. In practice, I know some Dataverse instances allow for kindOfData values that are not synonyms of 'Dataset'. For instance : https://entrepot.recherche.data.gouv.fr/oai?verb=GetRecord&metadataPrefix=oai_dc&identifier=doi:10.57745/DZIM2L

There might be an alternative solution where there is always at least a dc:type tag with value Dataset and the list of kindOfData values (making sure that Dataset occurs only once)? That would however means that not all dc:type values come from a controlled vocabulary.

[pdurbin]

@plecor thanks. One thing to consider with "dc:type" is that types other than datasets (like software and workflows) are coming...

• dataset types (software, workflow, etc.) - initial support #10694

... so maybe we can revisit "dc:type" once that pull request is merged.

To all, I pushed some tests to exercise export and setting the citation date.

Now I'm trying to see if there's a small change I can make to DublinCoreExportUtil to get the citation date out.

I can get just the year (YYYY) with code like this...

String citation = version.getCitation();
// We're looking for ", YYYY, " in a citation like this:
// Finch, Fiona, 1999, "Darwin's Finches", https://doi.org/10.5072/FK2/WSSYBE, Root, V1
Pattern pattern = Pattern.compile(", (\\d{4}), ");
Matcher matcher = pattern.matcher(citation);
matcher.find();
String yearInCitation = matcher.group(1);
writeFullElement(xmlw, dcFlavor+":"+"date", yearInCitation);

... but I need the full YYYY-MM-DD version to put in the the oai_dc output. 🤔 If you have any ideas for me, please let me know.

[pdurbin]

I dug a little more and our citation code is focused on returning just a 4 digit year for the date. This would be a change from what we do now (YYYY-MM-DD)

@philippconzett @plecor @qqmyers what do you think? Should we change oai_dc to YYYY for dc:date?

The spec Philipp found seems to say it's ok. Check out the year 1650 as an example at https://www.base-search.net/about/en/faq_oai.php#dc-date

[qqmyers]

Since the citationDateFieldType is part of the Dataset, I'd think at some point it could/should be part of the DatasetDTO and JSON export, thereby being available to other exporters (will the SPA or other client need this info (in the JSON returned from the dataset api) at some point?).

If that's too much for now, I think the idea of parsing it from the citation as YYYY makes sense, assuming that's sufficient for how people want to use that field. Alternately, I think you could 'go around' the exporter SPI interface and get the full value directly pretty easily as well, e.g. with something like:

        DatasetFieldType citationDataType = jakarta.enterprise.inject.spi.CDI.current().select(DatasetServiceBean.class).get().findByGlobalId(globalId.asString()).getCitationDateDatasetFieldType();
        if(citationDataType!= null) {
            date = dto2Primitive(version, citationDataType.getName());
        } else {
            date = datasetDto.getPublicationDate();
        }

This would not be the only current exporter doing that (e.g. the DDI exporter grabs the ExportInstallationAsDistributorOnlyWhenNotSet Setting it needs).

[pdurbin]

Alternately, I think you could 'go around' the exporter SPI interface and get the full value directly pretty easily as well, e.g. with something like...

@qqmyers I gave this a try but citationDataType.getName() yields a four digit year (YYYY) so it's no better than getting (just) the year from the citation with the regex I showed above.

[qqmyers]

Are you thinking of

dataverse/src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java

Line 1468 in ff26ae8

public Date getCitationDate() {

? citationDataType.getName() should get the name of the field type and then dto2Primitive should get the full value from the field itself in the metadatablock, which I think should always be the full date (unless there's a field that is YYYY only?).

[philippconzett]

@jggautier Thanks for clarifying. Is there a way to figure out what information we deliver to harvesters like BASE Bielefeld? See the initial post in #8129.

[jggautier]

@philippconzett, do you mean if there's a way to figure out what we currently deliver to harvesters like BASE Bielefeld or what we should deliver? Sorry if that sounds like a dumb question lol. I've been focused a lot on how decisions are made as well as what decisions are made, which maybe led to me overthink your question!

If it's currently, you wrote in #8129 about what we currently deliver when it comes to rights metadata, which is that we don't deliver anything since Dataverse doesn't provide the field dc:rights. I can say that nothing's changed about this since you opened #8129 a few years ago. So when I asked if it's appropriate that this PR being merged will close #8129, and that there's discussion in that GitHub issue that isn't addressed, I was mostly thinking about your comments related to dc:rights and degrees of open access.

For what we should start delivering to harvesters like BASE Bielefeld, you mentioned the guidance at https://www.base-search.net/about/en/faq_oai.php#dc-rights, which recommends the two vocabularies you pointed to earlier today: the info-eu-repo-Access-Rights vocabulary and the COAR-Access-Rights vocabulary.

I think it'll be helpful to consider what I wrote in #5920, where I wrote about what we've learned and challenges about how the info-eu-repo-Access-Rights vocabulary is being included in the OpenAIRE exports that Dataverse creates.

This might be a matter of scoping and timing, too, right? We could create a new GitHub issue specifically about the use of the info-eu-repo-Access-Rights and COAR-Access-Rights vocabularies, that mentions what's discussed in #8129. So when #8129 is closed because this PR is merged, the unaddressed goals you mentioned in #8129 aren't lost and there's a place where the community can focus on how to address those goals.

And if we can learn how @pdurbin and others made the decisions in this PR about what goes into dc:rights, it'll be easier to think about how effective those decisions are.

[philippconzett]

Hi @jggautier! Thanks for disentangling this! I wasn't aware of #5920, but have read up about it now and added some comments there. I don't really know what this means for #8129. Maybe a temporary solution to make dataset metadata from Dataverse more visible in OpenAIRE could be a kind of "inverted" and slightly adapted version of what you described in #5920:

• openAccess: If any files are set to non-restricted, the metadata export uses "openAccess".
• restrictedAccess: If all of the files in the dataset are set to restricted and the option to request. access is enabled (people are allowed to request access using Dataverse's request access feature), the metadata export uses "restrictedAccess".
• closedAccess: If all of the files in the dataset are set to restricted and the option to request access is disabled, the metadata export uses "closedAccess".
• embargoedAccess: If all of the files in the dataset are set to embargoed, the metadata export uses "embargoedAccess".

[pdurbin]

could you write about how these mapping decisions were made?

@jggautier back in 59850ce these lines were added as part of PR #3308:

writeFullElement(xmlw, "dcterms:license", version.getLicense());
writeFullElement(xmlw, "dcterms:rights", version.getTermsOfUse());
writeFullElement(xmlw, "dcterms:rights", version.getRestrictions());

This was for Dataverse 4.5 ( https://github.com/IQSS/dataverse/releases/tag/v4.5 ) when harvesting was first introduced (in 4.x). It looks like the code was mostly worked on by @landreev and @sekmiller. In short, the logic has been here for 8 years and I can't find any comment on why we do it this way. @jggautier what you're showing in that screenshot is a reflection of the same, unchanged logic.

The code above is for the "dcterms" flavor of Dublin Core. In this current PR, I copied the logic above for the "dc" flavor. I hope this helps!

[jggautier]

Ah, thanks @pdurbin! I updated the Dataverse crosswalk to reflect this, specifically the part of the crosswalk showing that what's entered in the Restrictions field is included in the DC Terms export, as dcterms:rights like you wrote. The crosswalk used to indicate this, and for some reason I don't remember now, in 2022 I edited it to read that it was "(Not mapped)".

To learn more about why the predefined license, Terms of Use, and Restrictions are included in the DC Terms export, I tried to find the Functional Requirements Document mentioned in that PR you linked to. The Functional Requirements Document folder in our Google Drive has some info, but I haven't seen any FRDs so far that go into enough detail.

I think we should:

• Remove from this PR decisions about what's added to dc:rights and make sure that this PR does not close the issue at Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting #8129 or
• Agree that what this PR does is okay for now only because it matches what's included in the DC Terms metadata exports, make sure that this PR as designed right now does not close the issue at Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting #8129, and update Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting #8129 to reflect what's been decided in this PR or
• Move this PR from "In Review" to "In Progress" so that research can be done to determine how information about the licenses and terms metadata of deposits should be included in the Dublin Core metadata that Dataverse exports, including tackling what's discussed in Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting #8129

[pdurbin]

As discussed in Slack and elsewhere:

• I backed out any changes having to do with dc:rights and added a large font note to the top of Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting #8129 (which this PR closes) to look to Access Rights metadata in OpenAIRE metadata export is being misapplied #5920 instead. @cmbz @scolapasta I'm not sure where you want Access Rights metadata in OpenAIRE metadata export is being misapplied #5920 in terms of priorities.
• I made some doc changes suggested by @landreev

@landreev I'm going to unassign myself but please let me know if you'd like me to jump back on this branch and do any additional coding or testing!

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:8129-harvesting

ghcr.io/gdcc/configbaker:8129-harvesting

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[landreev]

@pdurbin
Just for clarity, could you please attach an actual example - 2 exported oai_dc fragments, before and after, to illustrate the final result of the changes made in the PR. (For the benefit of somebody reading the PR in the future; the discussion above is quite extensive and potentially confusing)

[pdurbin]

@landreev sure, I added two XML examples, before and after, to the description of this PR.

[landreev]

I have confirmed that the change does not affect harvesting. In normal practice, this should not even be a concern, as two Dataverses should never use the oai_dc as the format for harvesting from each other. But it's not entirely impossible that it will be a practical use case for somebody. Plus it would simply feel wrong, for Dataverse not to be able to import its own metadata exports. So, happy to report that 2 Dataverses can still harvest from each other using the format.