Merge branch 'develop' into 8808-10575-update-global-role

.gitignore

@@ -34,6 +34,7 @@ oauth-credentials.md
 /src/main/webapp/oauth2/newAccount.html
 scripts/api/setup-all.sh*
 scripts/api/setup-all.*.log
+src/main/resources/edu/harvard/iq/dataverse/openapi/
 
 # ctags generated tag file
 tags

README.md

@@ -7,7 +7,7 @@ Dataverse is an [open source][] software platform for sharing, finding, citing,
 
 We maintain a demo site at [demo.dataverse.org][] which you are welcome to use for testing and evaluating Dataverse.
 
-To install Dataverse, please see our [Installation Guide][] which will prompt you to download our [latest release][].
+To install Dataverse, please see our [Installation Guide][] which will prompt you to download our [latest release][]. Docker users should consult the [Container Guide][].
 
 To discuss Dataverse with the community, please join our [mailing list][], participate in a [community call][], chat with us at [chat.dataverse.org][], or attend our annual [Dataverse Community Meeting][].
 
@@ -28,6 +28,7 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 [Dataverse community]: https://dataverse.org/developers
 [Installation Guide]: https://guides.dataverse.org/en/latest/installation/index.html
 [latest release]: https://github.com/IQSS/dataverse/releases
+[Container Guide]: https://guides.dataverse.org/en/latest/container/index.html
 [features]: https://dataverse.org/software-features
 [project board]: https://github.com/orgs/IQSS/projects/34
 [roadmap]: https://www.iq.harvard.edu/roadmap-dataverse-project

conf/solr/9.3.0/solrconfig.xml

@@ -290,7 +290,7 @@
          have some sort of hard autoCommit to limit the log size.
       -->
     <autoCommit>
-      <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
+      <maxTime>${solr.autoCommit.maxTime:30000}</maxTime>
       <openSearcher>false</openSearcher>
     </autoCommit>
 
@@ -301,7 +301,7 @@
       -->
 
     <autoSoftCommit>
-      <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
+      <maxTime>${solr.autoSoftCommit.maxTime:1000}</maxTime>
     </autoSoftCommit>
 
     <!-- Update Related Event Listeners

doc/release-notes/10236-openapi-definition-endpoint.md

@@ -0,0 +1,8 @@
+In Dataverse 6.0 Payara was updated, which caused the url `/openapi` to stop working:
+
+- https://github.com/IQSS/dataverse/issues/9981
+- https://github.com/payara/Payara/issues/6369
+
+When it worked in Dataverse 5.x, the `/openapi` output was generated automatically by Payara, but in this release we have switched to OpenAPI output produced by the [SmallRye OpenAPI plugin](https://github.com/smallrye/smallrye-open-api/tree/main/tools/maven-plugin). This gives us finer control over the output.
+
+For more information, see the section on  [OpenAPI](https://dataverse-guide--10328.org.readthedocs.build/en/10328/api/getting-started.html#openapi) in the API Guide.

doc/release-notes/10466-math-challenge-403-error-page.md

@@ -0,0 +1 @@
+On forbidden access error page, also know as 403 error page, the math challenge is now correctly display to submit the contact form.

doc/release-notes/10547-solr-updates.md

@@ -0,0 +1 @@
+Multiple improvements have ben made to they way Solr indexing and searching is done. Response times should be significantly improved. See the individual PRs in this release for details.

doc/release-notes/10561-3dviewer.md

@@ -0,0 +1 @@
+3DViewer by openforestdata.pl has been added to the list of external tools: https://preview.guides.gdcc.io/en/develop/admin/external-tools.html#inventory-of-external-tools

doc/release-notes/9729-release-notes.md

@@ -0,0 +1 @@
+An error is now correctly reported when an attempt is made to assign an identical role to the same collection, dataset, or file. #9729 #10465

doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv

@@ -7,3 +7,4 @@ Data Curation Tool	configure	file	"A GUI for curating data by adding labels, gro
 Ask the Data	query	file	Ask the Data is an experimental tool that allows you ask natural language questions about the data contained in Dataverse tables (tabular data). See the README.md file at https://github.com/IQSS/askdataverse/tree/main/askthedata for the instructions on adding Ask the Data to your Dataverse installation. 
 TurboCurator by ICPSR	configure	dataset	TurboCurator generates metadata improvements for title, description, and keywords. It relies on open AI's ChatGPT & ICPSR best practices. See the `TurboCurator Dataverse Administrator <https://turbocurator.icpsr.umich.edu/tc/adminabout/>`_ page for more details on how it works and adding TurboCurator to your Dataverse installation.
 JupyterHub	explore	file	The `Dataverse-to-JupyterHub Data Transfer Connector <https://forgemia.inra.fr/dipso/eosc-pillar/dataverse-jupyterhub-connector>`_ is a tool that simplifies the transfer of data between Dataverse repositories and the cloud-based platform JupyterHub. It is designed for researchers, scientists, and data analysts, facilitating collaboration on projects by seamlessly moving datasets and files. The tool is a lightweight client-side web application built using React and relies on the Dataverse External Tool feature, allowing for easy deployment on modern integration systems. Currently optimized for small to medium-sized files, future plans include extending support for larger files and signed Dataverse endpoints. For more details, you can refer to the external tool manifest: https://forgemia.inra.fr/dipso/eosc-pillar/dataverse-jupyterhub-connector/-/blob/master/externalTools.json
+3DViewer by openforestdata.pl	explore	file	The 3DViewer by openforestdata.pl can be used to explore 3D files (e.g. STL format). It was presented by Kamil Guryn during the 2020 community meeting (`slide deck <https://osf.io/aqgr5>`_, `video <https://youtu.be/YH4I_kldmGI?si=Y-9dZ3xPXeswchAl&t=2379>`_) and can be found at https://github.com/OpenForestData/open-forest-data-previewers

doc/sphinx-guides/source/api/changelog.rst

@@ -30,6 +30,7 @@ v6.0
 ----
 
 - **/api/access/datafile**: When a null or invalid API token is provided to download a public (non-restricted) file with this API call, it will result on a ``401`` error response. Previously, the download was allowed (``200`` response). Please note that we noticed this change sometime between 5.9 and 6.0. If you can help us pinpoint the exact version (or commit!), please get in touch. See :doc:`dataaccess`.
+- **/openapi**: This endpoint is currently broken. See https://github.com/IQSS/dataverse/issues/9981
 
 v5.6
 ----

doc/sphinx-guides/source/api/faq.rst

@@ -56,12 +56,12 @@ Where is the Comprehensive List of All API Functionality?
 
 There are so many Dataverse Software APIs that a single page in this guide would probably be overwhelming. See :ref:`list-of-dataverse-apis` for links to various pages.
 
-It is possible to get a complete list of API functionality in Swagger/OpenAPI format if you deploy Dataverse Software 5.x. For details, see https://github.com/IQSS/dataverse/issues/5794
+It is possible to get a complete list of API functionality in Swagger/OpenAPI format. See :ref:`openapi`.
 
 Is There a Changelog of API Functionality That Has Been Added Over Time?
 ------------------------------------------------------------------------
 
-No, but there probably should be. If you have suggestions for how it should look, please create an issue at https://github.com/IQSS/dataverse/issues
+Changes to the API that don't break anything can be found in the `release notes <https://github.com/IQSS/dataverse/releases>`_ of each release. Breaking changes are documented in :doc:`changelog`.
 
 .. _no-api:
 
@@ -89,6 +89,8 @@ Why Are the Return Values (HTTP Status Codes) Not Documented?
 
 They should be. Please consider making a pull request to help. The :doc:`/developers/documentation` section of the Developer Guide should help you get started. :ref:`create-dataverse-api` has an example you can follow or you can come up with a better way.
 
+Also, please note that we are starting to experiment with putting response codes in our OpenAPI document. See :ref:`openapi`.
+
 What If My Question Is Not Answered Here?
 -----------------------------------------
 

doc/sphinx-guides/source/api/getting-started.rst

@@ -154,6 +154,32 @@ Listing Permissions (Role Assignments)
 
 See :ref:`list-role-assignments-on-a-dataverse-api`.
 
+.. _openapi:
+
+Getting the OpenAPI Document
+----------------------------
+
+You can access our `OpenAPI document`_ using the ``/openapi`` endpoint. The default format is YAML if no parameter is provided, but you can also obtain the JSON version by either passing ``format=json`` as a query parameter or by sending ``Accept:application/json`` (case-sensitive) as a header.
+
+.. _OpenAPI document: https://spec.openapis.org/oas/latest.html#openapi-document
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export SERVER_URL=https://demo.dataverse.org
+  export FORMAT=json
+
+  curl "$SERVER_URL/openapi?format=$FORMAT"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl "https://demo.dataverse.org/openapi?format=json"
+
+We are aware that our OpenAPI document is not perfect. You can find more information about validating the document under :ref:`openapi-dev` in the Developer Guide.
+
 Beyond "Getting Started" Tasks
 ------------------------------
 

doc/sphinx-guides/source/developers/api-design.rst

@@ -7,6 +7,21 @@ API design is a large topic. We expect this page to grow over time.
 .. contents:: |toctitle|
 	:local:
 
+.. _openapi-dev:
+
+OpenAPI
+-------
+
+As you add API endpoints, please be conscious that we are exposing these endpoints as an OpenAPI document at ``/openapi`` (e.g. http://localhost:8080/openapi ). See :ref:`openapi` in the API Guide for the user-facing documentation.
+
+We've played around with validation tools such as https://quobix.com/vacuum/ and https://pb33f.io/doctor/ only to discover that our OpenAPI output is less than ideal, generating various warnings and errors.
+
+You can prevent additional problems in our OpenAPI document by observing the following practices:
+
+- When creating a method name within an API class, make it unique.
+
+If you are looking for a reference about the annotations used to generate the OpenAPI document, you can find it in the `MicroProfile OpenAPI Specification <https://download.eclipse.org/microprofile/microprofile-open-api-3.1/microprofile-openapi-spec-3.1.html#_detailed_usage_of_key_annotations>`_.
+
 Paths
 -----
 

docker/README.md

@@ -0,0 +1,5 @@
+# Dataverse in Docker
+
+Please see the [Container Guide][].
+
+[Container Guide]: https://guides.dataverse.org/en/latest/container/index.html

pom.xml

@@ -32,6 +32,12 @@
         <poi.version>5.2.1</poi.version>
         <tika.version>2.4.1</tika.version>
         <netcdf.version>5.5.3</netcdf.version>
+
+        <openapi.infoTitle>Dataverse API</openapi.infoTitle>
+        <openapi.infoVersion>${project.version}</openapi.infoVersion>
+        <openapi.infoDescription>Open source research data repository software.</openapi.infoDescription>
+        <!-- https://download.eclipse.org/microprofile/microprofile-open-api-3.1.1/microprofile-openapi-spec-3.1.1.html#_location_and_formats -->
+        <openapi.outputDirectory>${project.build.outputDirectory}/META-INF</openapi.outputDirectory>
     </properties>
 
     <!-- Versions of dependencies used both directly and transitive are managed here.
@@ -195,6 +201,11 @@
             <artifactId>microprofile-metrics-api</artifactId>
             <scope>provided</scope>
         </dependency>
+        <dependency>
+            <groupId>org.eclipse.microprofile.openapi</groupId>
+            <artifactId>microprofile-openapi-api</artifactId>
+            <scope>provided</scope>
+        </dependency>
         <dependency>
             <groupId>jakarta.platform</groupId>
             <artifactId>jakarta.jakartaee-api</artifactId>
@@ -930,6 +941,30 @@
                     <consoleOutput>true</consoleOutput>
                 </configuration>
             </plugin>
+            <plugin>
+                <groupId>io.smallrye</groupId>
+                <artifactId>smallrye-open-api-maven-plugin</artifactId>
+                <version>3.10.0</version>
+                <executions>
+                    <execution>
+                        <goals>
+                            <goal>generate-schema</goal>
+                        </goals>
+                        <!-- Plugin scans class files, not sources. Execute after compile phase but before package -->
+                        <phase>process-classes</phase>
+                        <configuration>
+                            <outputDirectory>${openapi.outputDirectory}</outputDirectory>
+                            <schemaFilename>openapi</schemaFilename>
+                            <infoTitle>${openapi.infoTitle}</infoTitle>
+                            <infoVersion>${openapi.infoVersion}</infoVersion>
+                            <infoDescription>${openapi.infoDescription}</infoDescription>
+                            <operationIdStrategy>CLASS_METHOD</operationIdStrategy>
+                            <scanPackages>edu.harvard.iq.dataverse</scanPackages>
+                            <scanDependenciesDisable>true</scanDependenciesDisable>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
         </plugins>
     </build>
     <profiles>
@@ -1087,4 +1122,4 @@
             </build>
         </profile>
     </profiles>
-</project>
+</project>

src/main/java/edu/harvard/iq/dataverse/SendFeedbackDialog.java

@@ -129,6 +129,10 @@ public void setUserSum(Long userSum) {
     }
 
     public String getMessageTo() {
+        if (op1 == null || op2 == null) {
+            // Fix for 403 error page: initUserInput method doesn't call before
+            initUserInput(null);
+        }
         if (feedbackTarget == null) {
             return BrandingUtil.getSupportTeamName(systemAddress);
         } else if (feedbackTarget.isInstanceofDataverse()) {

src/main/java/edu/harvard/iq/dataverse/api/Access.java

@@ -130,6 +130,14 @@
 import jakarta.ws.rs.core.MediaType;
 import static jakarta.ws.rs.core.Response.Status.FORBIDDEN;
 import static jakarta.ws.rs.core.Response.Status.UNAUTHORIZED;
+
+import org.eclipse.microprofile.openapi.annotations.Operation;
+import org.eclipse.microprofile.openapi.annotations.media.Content;
+import org.eclipse.microprofile.openapi.annotations.media.Schema;
+import org.eclipse.microprofile.openapi.annotations.parameters.RequestBody;
+import org.eclipse.microprofile.openapi.annotations.responses.APIResponse;
+import org.eclipse.microprofile.openapi.annotations.responses.APIResponses;
+import org.eclipse.microprofile.openapi.annotations.tags.Tag;
 import org.glassfish.jersey.media.multipart.FormDataBodyPart;
 import org.glassfish.jersey.media.multipart.FormDataParam;
 
@@ -1248,6 +1256,20 @@ private String getWebappImageResource(String imageName) {
     @AuthRequired
     @Path("datafile/{fileId}/auxiliary/{formatTag}/{formatVersion}")
     @Consumes(MediaType.MULTIPART_FORM_DATA)
+    @Produces("application/json")
+    @Operation(summary = "Save auxiliary file with version", 
+               description = "Saves an auxiliary file")
+    @APIResponses(value = {
+        @APIResponse(responseCode = "200",
+                    description = "File saved response"),
+        @APIResponse(responseCode = "403",
+                    description = "User not authorized to edit the dataset."),
+        @APIResponse(responseCode = "400",
+                    description = "File not found based on id.")
+    })
+    @Tag(name = "saveAuxiliaryFileWithVersion", 
+         description = "Save Auxiliary File With Version")
+    @RequestBody(content = @Content(mediaType = MediaType.MULTIPART_FORM_DATA))
     public Response saveAuxiliaryFileWithVersion(@Context ContainerRequestContext crc,
                                                  @PathParam("fileId") Long fileId,
                                                  @PathParam("formatTag") String formatTag,

src/main/java/edu/harvard/iq/dataverse/api/Admin.java

@@ -201,7 +201,7 @@ public Response putSetting(@PathParam("name") String name, String content) {
 
 	@Path("settings/{name}/lang/{lang}")
 	@PUT
-	public Response putSetting(@PathParam("name") String name, @PathParam("lang") String lang, String content) {
+	public Response putSettingLang(@PathParam("name") String name, @PathParam("lang") String lang, String content) {
 		Setting s = settingsSvc.set(name, lang, content);
 		return ok("Setting " + name + " - " + lang + " - added.");
 	}
@@ -224,7 +224,7 @@ public Response deleteSetting(@PathParam("name") String name) {
 
 	@Path("settings/{name}/lang/{lang}")
 	@DELETE
-	public Response deleteSetting(@PathParam("name") String name, @PathParam("lang") String lang) {
+	public Response deleteSettingLang(@PathParam("name") String name, @PathParam("lang") String lang) {
 		settingsSvc.delete(name, lang);
 		return ok("Setting " + name + " - " + lang + " deleted.");
 	}

src/main/java/edu/harvard/iq/dataverse/api/BuiltinUsers.java

@@ -119,7 +119,7 @@ public Response create(BuiltinUser user, @PathParam("password") String password,
      */
     @POST
     @Path("{password}/{key}/{sendEmailNotification}")
-    public Response create(BuiltinUser user, @PathParam("password") String password, @PathParam("key") String key, @PathParam("sendEmailNotification") Boolean sendEmailNotification) {
+    public Response createWithNotification(BuiltinUser user, @PathParam("password") String password, @PathParam("key") String key, @PathParam("sendEmailNotification") Boolean sendEmailNotification) {
         return internalSave(user, password, key, sendEmailNotification);
     }