Users can download Schema.org metadata from Export Metadata pulldown

[jggautier]

Users should be able to download Schema.org metadata from the Export Metadata pulldown in each dataset Metadata tab:

This would be the same metadata being added to the header of each dataset landing page in schema.org markup (#2243)

The additional metadata export should be added to the list of exports in the Admin Guides' Automatic Export section.

[pdurbin]

I just implemented this in pull request #4302

Here's how it looks in the GUI:

I set isHarvestable to false because this issue was estimated as a "2" and it's something we can implement in another small chunk in the future. We can wait until there is demand. (Coincidentally, on the dataverse-community list someone just asked three hours ago if harvesting is possible via a format we don't support (DataCite): https://groups.google.com/d/msg/dataverse-community/55cSbjBi10o/zLFO41__AQAJ )

Moving to code review at https://waffle.io/IQSS/dataverse

[jggautier]

Could schema.org be capitalized? "Schema.org JSON-LD"

I know the schema.org website has it all lowercase in their site header. It's not so clear throughout the rest of the website. DataCite capitalizes it

and it looks better in the list.

[pdurbin]

@jggautier sure, capitalized in b00d4d6

[pdurbin]

@jggautier I'm looking at https://schema.org/Dataset for guidance on how to support multiple descriptions (something @kcondon asked about). It says keywords are delimited by commas. Am I suppose to do the same with description? What if the description itself has commas? Or am I supposed to put each descriptions inside an array? That doesn't validate:

Please advise. Thanks. Also, I'm not sure what's going on with that warning about affiliation above.

[jggautier]

I separated each description with a comma in the example json doc, like with the keywords. I thought that because each value is enclosed in quotation marks, there wouldn't be problems with commas in the values and using commas as delimiters.

(The tool's asking for an @type, and the @type for description is "Text", but then we get a validation error because the @type for "name" is "Thing". Since we're not including anything else with the description, like the description date, it seems easier to exclude name.)

We decided to ignore that author affiliation warning. When there's no @type, the default @type "Thing" is used. Google's validation tool doesn't like "Thing" for author affiliation, but we can't use the @types it does like: "Person" and "Organization". We agreed it's okay to ignore the warning. The less-preferred alternatives are (1) using the @type "Author" or "Organization", which says that every author is a person or organization, which isn't true, and Dataverse has no way of knowing which author is a person and which is an organization or (2) not including an affiliation.

[pdurbin]

@jggautier ok so you want something like this (as shown in your screenshot above):

  "description": [
      "description1",
      "description2"
  ],

Makes sense, thanks. I'll see what I can do. In the code there are a lot of places where we grab just the first description of a dataset so I'm not sure why it's so important to expose multiple descriptions here but I assume you and @kcondon have your reasons.

I'm still confused about how to interpret https://schema.org/Dataset in the sense that I'm surprised that a field like "description" can be either a simple string...

  "description": "description1",

... or an array...

  "description": [
      "description1",
      "description2"
  ],

All "description" says is "Text" and "A description of the item." And if you look at https://schema.org/Text it doesn't say something like "this can be a string or an array" from what I can tell. Are there validators for schema.org apart from Google's online tool? Is the format expressed in JSON Schema ( http://json-schema.org ) or similar? Ideally I'd add some unit tests to the code to make sure Dataverse is emitting schema.org JSON that is valid according to the specification.

Thanks also for letting me know that "affiliation" is expected to throw a warning. This feels like a bug to me. I can take out affiliation if we would prefer there to be no warnings. At the very least I'll add javadoc indicating that these warnings are expected.

[scolapasta]

@landreev @pdurbin @jggautier Now that we have this in export, should we use this cached version for the JSON-LD on the page? (as a performance enhancement)

[pdurbin]

@scolapasta let's have a face to face conversation with @djbrooke and others to define the definition of done for this issue. It was estimated as a "2" and it's already in QA. I think @kcondon was close to merging it until he had a question about multiple descriptions.

Speaking of multiple descriptions, @jggautier and I talked this out this morning and we are not going to support multiple descriptions after all. All of the examples at https://schema.org/description show a single description, like this:

When folks like @sfarnel and @johnhuck come along and investigate how we've implemented schema.org support ( see ualbertalib/metadata#154 ), I assume they'll expect a description as in the screenshot above. That is to say, we've implemented "description" as a string rather than an array of strings. I still find it very odd that Google's testing tool is happy with both. Again, I'd love to see some JSON Schema or similar for this specification. I'd love to be able to write unit tests to validate the JSON that Dataverse is emitting.

@landreev would be in a better position than I am to know how much work it would be to make the performance enhancement you're talking about. The only change I was thinking of making to the code is adding some javadoc so that developers aren't surprised (as I was) that the JSON-LD we emit shows a warning in https://search.google.com/structured-data/testing-tool for "affiliation". @jggautier mentioned that someone at Google has approved ignoring the warning.

[scolapasta]

Sure; could be we consider in scope, or we could open up as a separate issue (or decide it's not necessary). Just wanted to make sure we considered it.

I think the change would just be in the method that gets the JSON-LD to look for a file instead of generating, but yes @landreev will be able to better estimate.

[pdurbin]

Ok, all morning I've been discussing this issue with @scolapasta @jggautier @kcondon @landreev @pameyer and probably some others. I just made some changes to the code, which I'll enumerate below, and pass this ticket to QA. If anyone wants to do some code review, please be my guest.

• 3cc02d0 For performance reasons, use the cached JSON-LD on disk (if available) when showing the JSON-LD in the <head> of the page, rather than going to the database.
• ba9c6bd Document in the API Guide the magic string you have to know to export the new JSON-LD format via API.
• e5c2528 For consistency, show "Schema.org JSON-LD" with capitalized "Schema.org" in the Admin Guide.
• 84224bd Guard against case where terms.getTermsOfUse is null. The dataset had been created with JSON via API using src/test/resources/json/dataset-finch1.json.
• 086824d More Javadoc for developers.

The format we are emitting is the same as when @kcondon last tested it yesterday. @jggautier plans to open an issue about how we display descriptions (especially if there are multiple) in the app.