Added Computational Workflow metadata and related documentation #8812

abujeda · 2022-06-23T17:26:19Z

What this PR does / why we need it:
Add Computational Workflow metadata to Dataverse and related documentation on how to use it.

Computational Workflow: A workflow that covers computational / data-driven workflows. They are designed to compose and execute a series of computational or data manipulation steps in a scientific application.

Which issue(s) this PR closes:
Closes #8639

Special notes for your reviewer:
None

Suggestions on how to test this:
~~For new installations, the metadata block will be added automatically to the database.~~ No longer the case. This is now an experimental feature.
Please add the new metadata block to the Dataverse installation:
curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @data/metadatablocks/computational_workflow.tsv -H "Content-type: text/tab-separated-values"

Create a new dataverse and add computational workflow metadata into it.
Create a dataset within this dataverse and populate the computational workflow metadata fields.

There are only 3 metadata fields. None of them are compulsory.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:
No

Is there a release notes update needed for this change?:
Yes. Added to the PR

Additional documentation:
None

pdurbin · 2022-07-18T21:00:38Z

@adaybujeda the Jenkins tests didn't run so I merged the latest from develop to kick off another build.

I'm still in the process of reviewing this PR but I thought I'd add a couple screenshots from a quick play:

showing new block

showing facet

One quick bit of feedback is that "External Code Repository URL" should be a clickable URL. Please stay tuned for more!

@poikilotherm we spoke briefly about this PR today. Would like like to add a review as well?

coveralls · 2022-07-18T21:08:23Z

Coverage remained the same at 19.756% when pulling b1bb1d3 on adaybujeda:8639-computational-workflow-metadata into ba0183d on IQSS:develop.

abujeda · 2022-07-19T07:23:15Z

Thanks @pdurbin.
Regarding the clickable URL for External Code Repository URL field, does this achieved by updating the display format to:
<a href="#VALUE" target="_blank" rel="noopener">#VALUE</a>, right?

poikilotherm

While I generally like the idea of having the option to filter by metadata type and workflows are unlikely to suffer as much from the mixed-content problems for a dataset, I would highly appreciate to make and declare this an experimental feature.

Also: can we cause a problem with same metadata fields in different schemas? (See comment inline)

poikilotherm · 2022-07-19T12:31:57Z

conf/solr/8.11.1/schema.xml

@@ -261,6 +261,9 @@
    <field name="cleaningOperations" type="text_en" multiValued="false" stored="true" indexed="true"/>
    <field name="collectionMode" type="text_en" multiValued="true" stored="true" indexed="true"/>
    <field name="collectorTraining" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="workflowType" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="codeRepository" type="text_en" multiValued="true" stored="true" indexed="true"/>


question: this is obviously coming from https://schema.org/codeRepository, getting used within the Workflow schema (below) and within to-be-done CodeMeta schema #7877. AFAIK this could be the first time ever we might add the same metadata field in two different schemas. Are we confident this will not cause troubles?

The name field in the tsv files must be globally unique in an instance, now enforced.

poikilotherm · 2022-07-19T12:46:45Z

doc/release-notes/8639-computational-workflow.md

+## Adding Computational Workflow Metadata
+The new Computational Workflow metadata block will allow depositors to effectively tag datasets as computational workflows.
+
+To add the new metadata block, follow the instructions in the user guide: <https://guides.dataverse.org/en/latest/admin/metadatacustomization.html>
+
+The location of the new metadata block tsv file is: `dataverse/scripts/api/data/metadatablocks/computational_workflow.tsv`


suggestion: May I request we declare this an experimental thing as usually done with new things like these?

I am not sure it should be. Is this something you can make a decision on @jggautier?

Or @scolapasta / @pdurbin , should this feature be labeled as experimental?

@poikilotherm, the effect of labelling this experimental would be to let installations know that it's likely that it will change (more likely than other changes in the update) as more work is done to support the publication of workflows and software. Is that the reasoning?

Since this is considered an MVP, I agree, and the group that worked on this metadatablock share the same concerns you raise in your comments in this PR (and in your talks during the Dataverse Community Meeting). I would confirm with Mahmood that this be labelled as experimental. Does anyone know Mahmood's GitHub username?

It seems to be @mmshad.

As @scolapasta and @jggautier know, we just talked about this at standup. That is, should this feature be enabled for all new installations or not? We're going to talk about this a bit more during our sprint planning meeting in a few hours.

As Gustavo pointed out, for embargo (which isn't on by default) we turned it on for the demo server to play around with it. (I'm not even sure if embargo is enabled in Harvard Dataverse or not.) So we could try this... enable the workflow block on the demo server first.

If we do flag the workflow block as experimental and not enabled by default we should probably remove it from the appendix of the User Guide or at least have a note saying that your admin need to manually enable it.

@mmshad if you happen to know any Harvard Dataverse users that would like to try out this workflow block, please let us know.

@pdurbin , I can ask a few labs to start uploading their workflow. My team can use it as well.

poikilotherm · 2022-07-19T12:48:01Z

doc/sphinx-guides/source/user/dataset-management.rst

+BagIt Support
+-------------
+
+BagIt is a set of hierarchical file system conventions designed to support disk-based storage and network transfer of arbitrary digital content. It offers several benefits such as integration with digital libraries, easy implementation, and transfer validation. See `the Wikipedia article <https://en.wikipedia.org/wiki/BagIt>`__ for more information.
+
+If the repository you are using has enabled BagIt file handling, when uploading BagIt files the repository will validate the checksum values listed in each BagIt’s manifest file against the uploaded files and generate errors about any mismatches. The repository will identify a certain number of errors, such as the first five errors in each BagIt file, before reporting the errors.
+
+|bagit-image1|
+
+You can fix the errors and reupload the BagIt files.
+
+For information on how to enable and configure the BagIt file handler see the :ref:`installation guide <BagIt File Handler>`
+


question: Why is this here? This PR should be about workflow support. Is this some rogue commit that landed in the PR?

For the objective 2 in the DataCommons project, the project manager wanted to bundle all the changes into a single unit to do user testing. When discussing with @scolapasta what would be the best approach to integrate all of the objective 2 changes into Dataverse, we decided to split into a set of PRs.

Documentation as a whole for objective 2 was bundle into this change, this is why BagIt appears here.

@pdurbin is it OK to leave the BagIt documentation in this PR or do you want me to move it to a new one?
As far as objective 2 is concerned, we need all the changes.

poikilotherm · 2022-07-19T13:37:17Z

scripts/api/setup-datasetfields.sh

@@ -7,3 +7,5 @@ curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @da
 curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @data/metadatablocks/astrophysics.tsv -H "Content-type: text/tab-separated-values"
 curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @data/metadatablocks/biomedical.tsv -H "Content-type: text/tab-separated-values"
 curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @data/metadatablocks/journals.tsv -H "Content-type: text/tab-separated-values"
+curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @data/metadatablocks/computational_workflow.tsv -H "Content-type: text/tab-separated-values"


suggestion: Can we please make this not a default for new installations, instead adding at as an experimental thing?

There is lots of more discussion coming down about how data, code and workflows should be handled and the results might change/replace/alter the approach outlined here. Would be nice to not have new installations add this by default then.

I wasn't sure myself when creating the PR so I asked @scolapasta about it. If I recall correctly, he said it was OK to add it in the default setup for new installations as you still need to add it to your collection manually.

pdurbin · 2022-07-19T14:03:09Z

updating the display format to:
<a href="#VALUE" target="_blank" rel="noopener">#VALUE</a>, right?

@adaybujeda yes exactly. You can look at keywordVocabularyURI as an example. It looks like you already have the type (url) and watermark set correctly.

I'm not actually sure what "termURI" should be. You have "computationalworkflow" for all three fields but https://guides.dataverse.org/en/5.11/admin/metadatacustomization.html suggests this could be a schema.org or purl.org URL. @qqmyers and @jggautier know more about this.

jggautier · 2022-07-19T14:10:55Z

In the TSV file at https://github.com/IQSS/dataverse/blob/8b467f710051c26950f38cdc9bf5eaffb5f76d0f/scripts/api/data/metadatablocks/computational_workflow.tsv, I don't see any URIs in the termURI field. I see "computationalworkflow" in the metadatablock_id column:

pdurbin · 2022-07-20T19:31:55Z

We just had a sprint planning meeting where a bunch of us discussed this issue and PR: @lenwiz @scolapasta @mreekie @landreev @sekmiller @kcondon @qqmyers @donsizemore @siacus

Here are the decisions we made:

Overall, we'd like to treat this Computational Workflow metadata block as experimental. This will prevent this metadata block from getting into common usage, at least for now. This means (@adaybujeda, I can make these changes):
- In the appendix, we'll carve out an area to list experimental blocks
- In the installer, we won't load this block.
- In the release notes, we'll explain how to load it.
In terms of deployment, post release (probably 5.12):
- We'll deploy the block to https://demo.dataverse.org and possibly ask @mmshad to ask a few labs to try it. (We understand that people may have already given input. Also, we'd need to make sure they understand that it's a test server without real DOIs or persistence.)
- Assuming we're happy with demo, we'll deploy the block to Harvard Dataverse.
Going forward, we can probably advertise other experimental blocks in the appendix:
- CodeMeta
- 3D Object
- DarwinCore
- ... ?
We should continue to work on the process for how experimental blocks become real.

pdurbin · 2022-07-20T21:10:46Z

@adaybujeda I just made some suggestions in a pull request: abujeda#1

I'm happy to discuss! Or feel free to merge if you like the direction! Thanks!

abujeda · 2022-07-20T21:42:03Z

@pdurbin I have merged the PR.

@jggautier @mmshad, Philip made some changes to the documentation, see previous comments for more background information regarding the changes.

This commit has a summary of the changes in case you want to review:
6d75a63

jggautier · 2022-07-21T16:15:25Z

Hi all. I'd like to move the "(see .tsv version)" link closer to what it's describing. So instead of:

Computational Workflow Metadata: adapted from Bioschemas Computational Workflow Profile, version 1.0 and Codemeta (see .tsv version).

It would be:

Computational Workflow Metadata (see .tsv version): adapted from Bioschemas Computational Workflow Profile, version 1.0 and Codemeta.

Otherwise, I think it looks like the link is pointing to the TSV version of Codemeta. The same is true for the other metadatablocks described in the section.

Or maybe an alternative is to make the "See .tsv version" link its own sentence:

Computational Workflow Metadata: adapted from Bioschemas Computational Workflow Profile, version 1.0 and Codemeta. See .tsv version.

What do you think?

abujeda · 2022-07-22T08:04:37Z

The 2 options look good @jggautier.

Let me know if you want me to implement one to make the changes.

jggautier · 2022-07-22T14:37:04Z

Thanks for considering, @adaybujeda. I just made and committed the changes. They're relatively small but I figured it would be better to make them now instead of me opening a new GitHub issue about it.

pdurbin

Now that the "Computational Workflow" block is flagged as experimental and not loaded by default, I'm ready for this to go to QA. Approved.

Other small changes: - Carve out space for advertising experimental metadata blocks. - CodePlex is extinct. Remove and reorder source hosting. - Tweaks to BagIt docs.

Moved "see .tsv version" link closer to the name of the metadatablock it's describing

abujeda · 2022-07-25T14:15:43Z

Thanks @pdurbin. I have rebased from develop. Ready for QA

philippconzett · 2022-07-26T04:49:48Z

If file-level DOI is activated, would workflows get the value "Workflow" in the field resourceTypeGeneral of the DataCite Metadata Schema? See Table 7: Description of resourceTypeGeneral on page 48 of the DataCite Metadata Schema 4.4, available at https://schema.datacite.org/meta/kernel-4.4/doc/DataCite-MetadataKernel_v4.4.pdf.

abujeda · 2022-07-26T12:15:02Z

As far as I know, no, they wouldn't. We haven't updated DataCite to understand the new metadata block.

abujeda mentioned this pull request Jun 23, 2022

Feature Request/Idea: Add Computational Workflow Metadata and Related Docs #8639

Closed

pdurbin self-assigned this Jul 18, 2022

poikilotherm reviewed Jul 19, 2022

View reviewed changes

abujeda force-pushed the 8639-computational-workflow-metadata branch from 8b467f7 to 715ca69 Compare July 19, 2022 16:20

pdurbin mentioned this pull request Jul 19, 2022

get codemeta.tsv to load (displayOrder, mostly) #7844 poikilotherm/dataverse#553

Closed

poikilotherm mentioned this pull request Jul 22, 2022

7844 codemeta schema #7877

Merged

3 tasks

pdurbin approved these changes Jul 25, 2022

View reviewed changes

pdurbin removed their assignment Jul 25, 2022

abujeda and others added 4 commits July 25, 2022 15:13

Added Computational Workflow metadata and related documentation

f6b2c17

Updated computation workflow fields to more specific names

9ac011a

make comp workflow block experimental IQSS#8639

4a187e2

Other small changes: - Carve out space for advertising experimental metadata blocks. - CodePlex is extinct. Remove and reorder source hosting. - Tweaks to BagIt docs.

Moved "see .tsv version" link

b1bb1d3

Moved "see .tsv version" link closer to the name of the metadatablock it's describing

abujeda force-pushed the 8639-computational-workflow-metadata branch from 29adad1 to b1bb1d3 Compare July 25, 2022 14:13

kcondon self-assigned this Jul 29, 2022

kcondon merged commit 4605ecf into IQSS:develop Jul 29, 2022

scolapasta added HDC: 2 Harvard Data Commons Obj. 2 HDC Harvard Data Commons labels Aug 1, 2022

pdurbin added this to the 5.12 milestone Aug 2, 2022

poikilotherm mentioned this pull request Dec 13, 2022

9224 - revert workflow metadata block in Solr schema #9225

Merged

mreekie added the NIH OTA: 1.3.1 3 | 1.3.1 | Support software metadata | 5 prdOwnThis is an item synched from the product planning... label Dec 15, 2022

sync-by-unito bot mentioned this pull request Mar 3, 2023

3 | 1.3.1 & 1.3.2 | Support software metadata | 5 IQSS/dataverse-pm#11

Closed

3 tasks

mreekie added pm.GREI-d-1.3.1 NIH, yr1, aim3, task1: Support software metadata pm.GREI-d-1.3.2 NIH, yr1, aim3, task2: R & D phase biomedical workflows support labels Mar 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Computational Workflow metadata and related documentation #8812

Added Computational Workflow metadata and related documentation #8812

abujeda commented Jun 23, 2022 •

edited

Loading

pdurbin commented Jul 18, 2022

coveralls commented Jul 18, 2022 •

edited

Loading

abujeda commented Jul 19, 2022

poikilotherm left a comment

poikilotherm Jul 19, 2022

qqmyers Jul 19, 2022

poikilotherm Jul 19, 2022 •

edited

Loading

abujeda Jul 19, 2022

jggautier Jul 19, 2022

pdurbin Jul 20, 2022

mmshad Jul 20, 2022

poikilotherm Jul 19, 2022

abujeda Jul 19, 2022

poikilotherm Jul 19, 2022

abujeda Jul 19, 2022

pdurbin commented Jul 19, 2022

jggautier commented Jul 19, 2022

pdurbin commented Jul 20, 2022

pdurbin commented Jul 20, 2022

abujeda commented Jul 20, 2022 •

edited

Loading

jggautier commented Jul 21, 2022 •

edited

Loading

abujeda commented Jul 22, 2022

jggautier commented Jul 22, 2022

pdurbin left a comment

abujeda commented Jul 25, 2022

philippconzett commented Jul 26, 2022

abujeda commented Jul 26, 2022

Added Computational Workflow metadata and related documentation #8812

Added Computational Workflow metadata and related documentation #8812

Conversation

abujeda commented Jun 23, 2022 • edited Loading

pdurbin commented Jul 18, 2022

showing new block

showing facet

coveralls commented Jul 18, 2022 • edited Loading

abujeda commented Jul 19, 2022

poikilotherm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

poikilotherm Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdurbin commented Jul 19, 2022

jggautier commented Jul 19, 2022

pdurbin commented Jul 20, 2022

pdurbin commented Jul 20, 2022

abujeda commented Jul 20, 2022 • edited Loading

jggautier commented Jul 21, 2022 • edited Loading

abujeda commented Jul 22, 2022

jggautier commented Jul 22, 2022

pdurbin left a comment

Choose a reason for hiding this comment

abujeda commented Jul 25, 2022

philippconzett commented Jul 26, 2022

abujeda commented Jul 26, 2022

abujeda commented Jun 23, 2022 •

edited

Loading

coveralls commented Jul 18, 2022 •

edited

Loading

poikilotherm Jul 19, 2022 •

edited

Loading

abujeda commented Jul 20, 2022 •

edited

Loading

jggautier commented Jul 21, 2022 •

edited

Loading