Merge branch 'develop' into 10688_whitespace_trimming IQSS#10688

Author: pdurbin

.github/ISSUE_TEMPLATE/bug_report.md

@@ -14,7 +14,7 @@ Thank you for contributing to the Dataverse Project through the creation of a bu
 WARNING: If this is a security issue it should be reported privately to security@dataverse.org
 
 More information on bug issues and contributions can be found in the "Contributing to Dataverse" page:
-https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md#bug-reportsissues
+https://guides.dataverse.org/en/latest/contributor/index.html
 
 Please fill out as much of the template as you can.
 Start below this comment section.
@@ -44,7 +44,6 @@ Start below this comment section.
 **Any related open or closed issues to this bug report?**
 
 
-
 **Screenshots:**
 
 No matter the issue, screenshots are always welcome.
@@ -53,3 +52,7 @@ To add a screenshot, please use one of the following formats and/or methods desc
 
 * https://help.github.com/en/articles/file-attachments-on-issues-and-pull-requests
 *
+
+
+**Are you thinking about creating a pull request for this issue?**
+Help is always welcome, is this bug something you or your organization plan to fix?

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,7 +1,7 @@
 ---
 name: Feature request
 about: Suggest an idea or new feature for the Dataverse software!
-title: 'Feature Request/Idea:'
+title: 'Feature Request:'
 labels: 'Type: Feature'
 assignees: ''
 
@@ -11,7 +11,7 @@ assignees: ''
 Thank you for contributing to the Dataverse Project through the creation of a feature request!
 
 More information on ideas/feature requests and contributions can be found in the "Contributing to Dataverse" page:
-https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md#ideasfeature-requests
+https://guides.dataverse.org/en/latest/contributor/index.html
 
 Please fill out as much of the template as you can.
 Start below this comment section.
@@ -34,3 +34,6 @@ Start below this comment section.
 
 
 **Any open or closed issues related to this feature request?**
+
+**Are you thinking about creating a pull request for this feature?**  
+Help is always welcome, is this feature something you or your organization plan to implement?

.github/ISSUE_TEMPLATE/idea_proposal.md

@@ -0,0 +1,40 @@
+---
+name: Idea proposal
+about: Propose a new idea for discussion to improve the Dataverse software!
+title: 'Suggestion:'
+labels: 'Type: Suggestion'
+assignees: ''
+
+---
+
+<!--
+Thank you for contributing to the Dataverse Project through the creation of a feature request!
+
+More information on ideas/feature requests and contributions can be found in the "Contributing to Dataverse" page:
+https://guides.dataverse.org/en/latest/contributor/index.html
+
+Please fill out as much of the template as you can.
+Start below this comment section.
+-->
+
+**Overview of the Suggestion**
+
+
+**What kind of user is the suggestion intended for?**
+(Example users roles: API User, Curator, Depositor, Guest, Superuser, Sysadmin)
+
+
+**What inspired this idea?**
+
+
+**What existing behavior do you want changed?**
+
+
+**Any brand new behavior do you want to add to Dataverse?**
+
+
+**Any open or closed issues related to this suggestion?**
+
+
+**Are you thinking about creating a pull request for this issue?**
+Help is always welcome, is this idea something you or your organization plan to implement?

.github/workflows/check_property_files.yml

@@ -0,0 +1,32 @@
+name: "Properties Check"
+on:
+    pull_request:
+        paths:
+            - "src/**/*.properties"
+            - "scripts/api/data/metadatablocks/*"
+jobs:
+    duplicate_keys:
+        name: Duplicate Keys
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Run duplicates detection script
+              shell: bash
+              run: tests/check_duplicate_properties.sh
+
+    metadata_blocks_properties:
+        name: Metadata Blocks Properties
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Setup GraalVM + Native Image
+              uses: graalvm/setup-graalvm@v1
+              with:
+                github-token: ${{ secrets.GITHUB_TOKEN }}
+                java-version: '21'
+                distribution: 'graalvm-community'
+            - name: Setup JBang
+              uses: jbangdev/setup-jbang@main
+            - name: Run metadata block properties verification script
+              shell: bash
+              run: tests/verify_mdb_properties.sh

.github/workflows/shellspec.yml

@@ -24,28 +24,11 @@ jobs:
               run: |
                   cd tests/shell
                   shellspec
-    shellspec-centos7:
-        name: "CentOS 7"
+    shellspec-rocky9:
+        name: "RockyLinux 9"
         runs-on: ubuntu-latest
         container:
-            image: centos:7
-        steps:
-            - uses: actions/checkout@v2
-            - name: Install shellspec
-              run: |
-                  curl -fsSL https://github.com/shellspec/shellspec/releases/download/${{ env.SHELLSPEC_VERSION }}/shellspec-dist.tar.gz | tar -xz -C /usr/share
-                  ln -s /usr/share/shellspec/shellspec /usr/bin/shellspec
-            - name: Install dependencies
-              run: yum install -y ed
-            - name: Run shellspec
-              run: |
-                  cd tests/shell
-                  shellspec
-    shellspec-rocky8:
-        name: "RockyLinux 8"
-        runs-on: ubuntu-latest
-        container:
-            image: rockylinux/rockylinux:8
+            image: rockylinux/rockylinux:9
         steps:
             - uses: actions/checkout@v2
             - name: Install shellspec

conf/solr/schema.xml

@@ -142,6 +142,7 @@
 
     <field name="dvName" type="text_en" stored="true" indexed="true" multiValued="false"/>
     <field name="dvAlias" type="text_en" stored="true" indexed="true" multiValued="false"/>
+    <field name="dvParentAlias" type="text_en" stored="true" indexed="true" multiValued="false"/>
     <field name="dvAffiliation" type="text_en" stored="true" indexed="true" multiValued="false"/>
     <field name="dvDescription" type="text_en" stored="true" indexed="true" multiValued="false"/>
 
@@ -205,6 +206,7 @@
     <field name="entityId" type="plong" stored="true" indexed="true" multiValued="false"/>
 
     <field name="datasetVersionId" type="plong" stored="true" indexed="true" multiValued="false"/>
+    <field name="datasetType" type="string" stored="true" indexed="true" multiValued="false"/>
 
     <!-- Added for Dataverse 4.0 alpha 1 to sort by name  -->
     <!-- https://redmine.hmdc.harvard.edu/issues/3482 -->
@@ -426,6 +428,7 @@
     <copyField source="dvAlias" dest="_text_" maxChars="3000"/>
     <copyField source="dvAffiliation" dest="_text_" maxChars="3000"/>
     <copyField source="dsPersistentId" dest="_text_" maxChars="3000"/>
+    <copyField source="datasetType" dest="_text_" maxChars="3000"/>
     <!-- copyField commands copy one field to another at the time a document
         is added to the index.  It's used either to index the same field differently,
         or to add multiple fields to the same field for easier/faster searching.  -->

doc/release-notes/10169-JSON-schema-validation.md

@@ -0,0 +1,3 @@
+### Improved JSON Schema validation for datasets
+
+Enhanced JSON schema validation with checks for required and allowed child objects, type checking for field types including `primitive`, `compound` and `controlledVocabulary`. More user-friendly error messages to help pinpoint the issues in the dataset JSON. See [Retrieve a Dataset JSON Schema for a Collection](https://guides.dataverse.org/en/6.3/api/native-api.html#retrieve-a-dataset-json-schema-for-a-collection) in the API Guide and PR #10543.

doc/release-notes/10341-croissant.md

@@ -0,0 +1,9 @@
+A new metadata export format called Croissant is now available as an external metadata exporter. It is oriented toward making datasets consumable by machine learning.
+
+When enabled, Croissant replaces the Schema.org JSON-LD format in the `<head>` of dataset landing pages. For details, see the [Schema.org JSON-LD/Croissant Metadata](https://dataverse-guide--10533.org.readthedocs.build/en/10533/admin/discoverability.html#schema-org-head) under the discoverability section of the Admin Guide.
+
+For more about the Croissant exporter, see https://github.com/gdcc/exporter-croissant
+
+For installation instructions, see [Enabling External Exporters](https://dataverse-guide--10533.org.readthedocs.build/en/10533/installation/advanced.html#enabling-external-exporters) in the Installation Guide.
+
+See also Issue #10341 and PR #10533.

doc/release-notes/10433-add-thumbnail-for-featured-dataverses.md

@@ -0,0 +1,5 @@
+Add the ability to configure a thumbnail logo that is displayed for a collection when the collection is configured as a featured collection. If present, this thumbnail logo is shown. Otherwise, the collection logo is shown. Configuration is done under the "Theme" for a collection.
+
+The HTML preview of the documentation can be found [here](https://dataverse-guide--10433.org.readthedocs.build/en/10433/user/dataverse-management.html#theme).
+
+For more information, see [#10291](https://github.com/IQSS/dataverse/issues/10291).

doc/release-notes/10508-base-image-fixes.md

@@ -0,0 +1,12 @@
+# Security and Compatibility Fixes to the Container Base Image
+
+- Switch "wait-for" to "wait4x", aligned with the Configbaker Image
+- Update "jattach" to v2.2
+- Install AMD64 / ARM64 versions of tools as necessary
+- Run base image as unprivileged user by default instead of `root` - this was an oversight from OpenShift changes
+- Linux User, Payara Admin and Domain Master passwords:
+  - Print hints about default, public knowledge passwords in place for
+  - Enable replacing these passwords at container boot time
+- Enable building with updates Temurin JRE image based on Ubuntu 24.04 LTS
+- Fix entrypoint script troubles with pre- and postboot script files
+- Unify location of files at CONFIG_DIR=/opt/payara/config, avoid writing to other places

doc/release-notes/10517-datasetType.md

@@ -0,0 +1,10 @@
+### Initial Support for Dataset Types
+
+Out of the box, all datasets have the type "dataset" but superusers can add additional types. At this time the type can only be set at creation time via API. The types "dataset", "software", and "workflow" will be sent to DataCite when the dataset is published.
+
+For details see <https://dataverse-guide--10694.org.readthedocs.build/en/10694/user/dataset-management.html#dataset-types> and #10517. Please note that this feature is highly experimental and is expected to evolve.
+
+Upgrade instructions
+--------------------
+
+Update your Solr schema.xml file to pick up the "datasetType" additions and do a full reindex.

doc/release-notes/10583-dataset-unlink-functionality-same-permission-as-link.md

@@ -0,0 +1,2 @@
+New "Unlink Dataset" button has been added to the Dataset Page to allow a user to unlink a dataset from a collection that was previously linked with the "Link Dataset" button. The user must possess the same permissions needed to unlink the Dataset as they would to link the Dataset.
+The [existing API](https://guides.dataverse.org/en/6.3/admin/dataverses-datasets.html#unlink-a-dataset) for unlinking datasets has been updated to no longer require superuser access. The "Publish Dataset" permission is now enough.

doc/release-notes/10633-add-dataverse-api-extension.md

@@ -0,0 +1 @@
+The addDataverse (/api/dataverses/{identifier}) API endpoint has been extended to allow adding metadata blocks, input levels and facet ids at creation time, as the Dataverse page in create mode does in JSF.

doc/release-notes/10726-dataverse-facets-api-extension.md

@@ -0,0 +1,3 @@
+New optional query parameter "returnDetails" added to "dataverses/{identifier}/facets/" endpoint to include detailed information of each DataverseFacet.
+
+New endpoint "datasetfields/facetables" that lists all facetable dataset fields defined in the installation.

doc/release-notes/10733-add-publication-status-to-search-api-results.md

@@ -0,0 +1,14 @@
+Search API (/api/search) response will now include publicationStatuses in the Json response as long as the list is not empty
+
+Example:
+```javascript
+"items": [
+    {
+        "name": "Darwin's Finches",
+        ...
+        "publicationStatuses": [
+            "Unpublished",
+            "Draft"
+        ],
+(etc, etc)
+```

doc/release-notes/10741-list-metadatablocks-display-on-create-fix.md

@@ -0,0 +1 @@
+Fixed dataverses/{identifier}/metadatablocks endpoint to not return fields marked as displayOnCreate=true if there is an input level with include=false, when query parameters returnDatasetFieldTypes=true and onlyDisplayedOnCreate=true are set.

doc/release-notes/10744-ro-crate-docs.md

@@ -0,0 +1,3 @@
+## RO-Crate Support (Metadata Export)
+
+Dataverse now supports [RO-Crate](https://www.researchobject.org/ro-crate/) in the sense that dataset metadata can be exported in that format. This functionality is not available out of the box but you can enable one or more RO-Crate exporters from the [list of external exporters](https://preview.guides.gdcc.io/en/develop/installation/advanced.html#inventory-of-external-exporters). See also #10744.

doc/release-notes/10749-dataverse-user-permissions-api-extension.md

@@ -0,0 +1 @@
+New API endpoint "dataverses/{identifier}/userPermissions" for obtaining the user permissions on a dataverse.

doc/release-notes/10758-rust-client.md

@@ -0,0 +1,3 @@
+### Rust API client library
+
+An API client library for the Rust programming language is now available at https://github.com/gdcc/rust-dataverse and has been added to the [list of client libraries](https://dataverse-guide--10758.org.readthedocs.build/en/10758/api/client-libraries.html) in the API Guide. See also #10758.

doc/release-notes/10797-update-current-version-bug-fix.md

@@ -0,0 +1,11 @@
+A significant bug in the superuser-only "Update-Current-Version" publication was found and fixed in this release. If the Update-Current-Version option was used when changes were made to the dataset Terms (rather than to dataset metadata), or if the PID provider service was down/returned an error, the update would fail and render the dataset unusable and require restoration from a backup. The fix in this release allows the update to succeed in both of these cases and redesigns the functionality such that any unknown issues should not make the dataset unusable (i.e. the error would be reported and the dataset would remain in its current state with the last-published version as it was and changes still in the draft version.)
+
+Users of earlier Dataverse releases are encouraged to alert their superusers to this issue. Those who wish to disable this functionality have two options:
+* Change the dataset.updateRelease entry in the Bundle.properties file (or local language version) to "Do Not Use" or similar (doesn't disable but alerts superusers to the issue), or
+* Edit the dataset.xhtml file to remove the lines
+
+<c:if test="#{dataverseSession.user.isSuperuser()}">
+  <f:selectItem rendered="#" itemLabel="#{bundle['dataset.updateRelease']}" itemValue="3" />
+</c:if>
+
+, delete the contents of the generated and osgi-cache directories in the Dataverse Payara domain, and restart the Payara server.

doc/release-notes/10800-add-dataverse-request-json-fix.md

@@ -0,0 +1 @@
+Fixed the "addDataverse" API endpoint (/dataverses/{id} POST) expected request JSON structure to parse facetIds as described in the docs.

doc/release-notes/10810-search-api-payload-extensions.md

@@ -0,0 +1,52 @@
+Search API (/api/search) response will now include new fields for the different entities.
+
+For Dataverse:
+
+- "affiliation" 
+- "parentDataverseName"
+- "parentDataverseIdentifier"
+- "image_url" (optional)
+
+```javascript
+"items": [
+    {
+        "name": "Darwin's Finches",
+        ...
+        "affiliation": "Dataverse.org",
+        "parentDataverseName": "Root",
+        "parentDataverseIdentifier": "root",
+        "image_url":"data:image/png;base64,iVBORw0..."
+(etc, etc)
+```
+
+For DataFile:
+
+- "releaseOrCreateDate"
+- "image_url" (optional)
+
+```javascript
+"items": [
+    {
+        "name": "test.txt",
+        ...
+        "releaseOrCreateDate": "2016-05-10T12:53:39Z",
+        "image_url":"data:image/png;base64,iVBORw0..."
+(etc, etc)
+```
+
+For Dataset:
+
+- "image_url" (optional)
+
+```javascript
+"items": [
+    {
+        ...
+        "image_url": "http://localhost:8080/api/datasets/2/logo"
+        ...
+(etc, etc)
+```
+
+The image_url field was already part of the SolrSearchResult JSON (and incorrectly appeared in Search API documentation), but it wasn’t returned by the API because it was appended only after the Solr query was executed in the SearchIncludeFragment of JSF. Now, the field is set in SearchServiceBean, ensuring it is always returned by the API when an image is available.
+
+The schema.xml file for Solr has been updated to include a new field called dvParentAlias for supporting the new response field "parentDataverseIdentifier". So for the next Dataverse released version, a Solr reindex will be necessary to apply the new schema.xml version.

doc/release-notes/10819-publish-thumbnail-bug.md

@@ -0,0 +1,6 @@
+The initial release of the Dataverse v6.3 introduced a bug where publishing would break the dataset thumbnail, which in turn broke the rendering of the parent Collection ("dataverse") page. This problem was fixed in the PR 10820.
+
+This bug fix will prevent this from happening in the future, but does not fix any existing broken links. To restore any broken thumbnails caused by this bug, you can call the http://localhost:8080/api/admin/clearThumbnailFailureFlag API, which will attempt to clear the flag on all files (regardless of whether caused by this bug or some other problem with the file) or the http://localhost:8080/api/admin/clearThumbnailFailureFlag/id to clear the flag for individual files. Calling the former, batch API is recommended. 
+
+Additionally, the same PR made it possible to turn off the feature that automatically selects of one of the image datafiles to serve as the thumbnail of the parent dataset. An admin can turn it off by raising the feature flag `<jvm-options>-Ddataverse.feature.disable-dataset-thumbnail-autoselect=true</jvm-options>`. When the feature is disabled, a user can still manually pick a thumbnail image, or upload a dedicated thumbnail image.
+

doc/release-notes/7068-reserve-file-pids.md

@@ -0,0 +1,9 @@
+## Release Highlights
+
+### Pre-Publish File DOI Reservation with DataCite
+
+Dataverse installations using DataCite (or other persistent identifier (PID) Providers that support reserving PIDs) will be able to reserve PIDs for files when they are uploaded (rather than at publication time). Note that reserving file DOIs can slow uploads with large numbers of files so administrators may need to adjust timeouts (specifically any Apache "``ProxyPass / ajp://localhost:8009/ timeout=``" setting in the recommended Dataverse configuration).
+
+## Major Use Cases
+
+- Users will have DOIs/PIDs reserved for their files as part of file upload instead of at publication time. (Issue #7068, PR #7334)

doc/release-notes/8129-harvesting.md

@@ -0,0 +1,18 @@
+### Remap oai_dc export and harvesting format fields: dc:type and dc:date
+
+The `oai_dc` export and harvesting format has had the following fields remapped:
+
+- dc:type was mapped to the field "Kind of Data". Now it is hard-coded to the word "Dataset".
+- dc:date was mapped to the field "Production Date" when available and otherwise to "Publication Date". Now it is mapped the field "Publication Date" or the field used for the citation date, if set (see [Set Citation Date Field Type for a Dataset](https://guides.dataverse.org/en/6.3/api/native-api.html#set-citation-date-field-type-for-a-dataset)).
+
+In order for these changes to be reflected in existing datasets, a [reexport all](https://guides.dataverse.org/en/6.3/admin/metadataexport.html#batch-exports-through-the-api) should be run.
+
+For more information, please see #8129 and #10737.
+
+### Backward incompatible changes 
+
+See the "Remap oai_dc export" section above.
+
+### Upgrade instructions
+
+In order for changes to the `oai_dc` metadata export format to be reflected in existing datasets, a [reexport all](https://guides.dataverse.org/en/6.3/admin/metadataexport.html#batch-exports-through-the-api) should be run.