Merge branch 'develop' into IQSS/7467 #7467 #7468

doc/release-notes/5.0-release-notes.md

@@ -245,9 +245,10 @@ The instructions below describe the upgrade procedure based on moving an existin
    `sudo /usr/local/glassfish4/bin/asadmin stop-domain`
    `sudo cp -ar /usr/local/glassfish4/glassfish/domains/domain1 /usr/local/payara5/glassfish/domains/`
 
-7. Remove the Glassfish cache directories
+7. Remove the cache directories
 
-   `sudo rm -rf /usr/local/payara5/glassfish/domains/domain1/generated/`  `sudo rm -rf /usr/local/payara5/glassfish/domains/domain1/osgi-cache/`
+   `sudo rm -rf /usr/local/payara5/glassfish/domains/domain1/generated/`  
+   `sudo rm -rf /usr/local/payara5/glassfish/domains/domain1/osgi-cache/`
 
 8. In domain.xml:
 
@@ -263,18 +264,26 @@ Add the below JVM options beneath the -Ddataverse settings:
    <jvm-options>-XX:+UseStringDeduplication</jvm-options>
    <jvm-options>-XX:+DisableExplicitGC</jvm-options>
 
+Also in domain.xml, replace the follow element:
+
+  `<jdbc-connection-pool datasource-classname="org.apache.derby.jdbc.EmbeddedXADataSource" name="__TimerPool" res-type="javax.sql.XADataSource"> <property name="databaseName" value="${com.sun.aas.instanceRoot}/lib/databases/ejbtimer"></property><property name="connectionAttributes" value=";create=true"></property></jdbc-connection-pool>`
+
+with
+
+  `<jdbc-connection-pool datasource-classname="org.h2.jdbcx.JdbcDataSource" name="__TimerPool" res-type="javax.sql.XADataSource"> <property name="URL" value="jdbc:h2:${com.sun.aas.instanceRoot}/lib/databases/ejbtimer;AUTO_SERVER=TRUE"></property> </jdbc-connection-pool>`
+
 9. Change any full pathnames /usr/local/glassfish4/... to /usr/local/payara5/... or whatever it is in your case. (Specifically check the -Ddataverse.files.directory and -Ddataverse.files.file.directory JVM options)
 
-10.  In domain1/config/jhove.conf, change the hard-coded /usr/local/glassfish4 path, as above.
+10.   In domain1/config/jhove.conf, change the hard-coded /usr/local/glassfish4 path, as above.
 
 (Optional): If you renamed your service account from glassfish to payara or appserver, update the ownership permissions. The Installation Guide recommends a service account of `dataverse`:
 
    `sudo chown -R dataverse /usr/local/payara5/glassfish/domains/domain1`
    `sudo chown -R dataverse /usr/local/payara5/glassfish/lib`
 
-11. You will also need to check that the service account has write permission on the files directory, if they are located outside the old Glassfish domain. And/or make sure the service account has the correct AWS credentials, if you are using S3 for storage.
+11.  You will also need to check that the service account has write permission on the files directory, if they are located outside the old Glassfish domain. And/or make sure the service account has the correct AWS credentials, if you are using S3 for storage.
 
-12. Finally, start Payara:
+12.  Finally, start Payara:
 
    `sudo -u dataverse /usr/local/payara5/bin/asadmin start-domain`
 

doc/release-notes/5.3-release-notes.md

@@ -0,0 +1,165 @@
+# Dataverse 5.3
+
+This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### Auxiliary Files (Experimental)
+
+Auxiliary files can now be added to datafiles and accessed using new experimental API endpoints. These endpoints allow additional, non-Dataverse generated metadata to be added alongside datafiles in dataverse.
+
+The support for auxiliary files in Dataverse is being driven by integration with the [Open Differential Privacy (DP) Project](https://opendp.io) and is designed to support the deposit and retrieval of differentially private metadata, but the endpoints are not specific to differential privacy use cases.
+
+### Additional Banner Functionality
+
+Banners in Dataverse can now be set to allow dismissal by a logged in user. Previously, banners would persist until they were removed by an administrator. This allows administrators to more easily communicate one-time messages to users.
+
+### File Tags Searchable from Advanced Search and Dataset Search
+
+File tags ("Documentation", "Data", "Code", etc.) now appear on the Advanced Search page.
+
+Performing a search for files on the dataset page now includes file tags. Previously, only file name and file description were searched.
+
+### Easier Configuration of Database Connections
+
+Previously, the configuration of the database connections has been quite static and not very easy to update. This has been an issue especially for cloud and container usage. Using new technologies provided by the move to Payara, you can now more easily configure the connection to your PostgreSQL DB.
+
+Using MicroProfile Config API (Issue #7000, Issue #7418), you can much more easily specify configuration
+details. For an overview of supported options, please see the
+[Installation Guide](https://guides.dataverse.org/en/5.3/installation/config.html#jvm-options).
+
+Note that some settings have been moved from domain.xml to code, such as min and max pool size.
+
+## Major Use Cases
+
+Newly-supported use cases in this release include:
+
+- Users can use an API to add auxiliary files to files in order to provide metadata representations for specific tools or integrations (Issue #7275, PR #7350)
+- Administrators can use a new API to manage banner messages and take advantage of new banner display options (Issue #7263, PR #7434)
+- Users replacing files will now have their files renamed when a file name conflict exists, making the behavior consistent with upload and edit (Issue #7335, PR #7336)
+- Users will now be able to search on file tags on the advanced search and dataset pages (Issue #7194, PR #7385)
+
+## Notes for Dataverse Installation Administrators
+
+### Payara 5.2020.6 (or Higher) Required
+
+Some changes in this release require an upgrade to Payara 5.2020.6 or higher.
+
+Instructions on how to update can be found in the
+[Payara documentation](https://docs.payara.fish/community/docs/5.2020.6/documentation/user-guides/upgrade-payara.html)
+
+### New Banner API, Obsolete DB Settings
+
+The functionality previously provided by the DB settings :StatusMessageHeader and ::StatusMessageText is no longer supported and is now provided through the Manage Banner Messages API. Learn more in the [API Guide](https://guides.dataverse.org/en/5.3/api/).
+
+### New Database Settings and JVM Options
+
+Several new JVM options have been added in this release:
+
+- dataverse.db.name
+- dataverse.db.user
+- dataverse.db.password
+- dataverse.db.host
+- dataverse.db.port
+
+For an overview of these new options, please see the
+[Installation Guide](https://guides.dataverse.org/en/5.3/installation/config.html#jvm-options)
+
+See above note about obsolete DB options.
+
+### Introducing MicroProfile Config API
+
+With this Dataverse release, Dataverse Administrators can start to make use of the MicroProfile Config API.
+
+This will benefit both developers and sysadmins, but the codebase will have to be refactored to make use of it. As this will take time, we will always provide a backward compatible way of using it.
+
+For more details about these new options, please see the Consuming Configuration section of the [Developer Guide](https://guides.dataverse.org/en/5.3/developers/).
+
+### Java Message System Configuration
+
+The Ingest process uses the Java Message System to create ingest tasks in a queue. That queue had been configured from command line or domain.xml before. This has now changed to being done
+in code.
+
+In the unlikely case you might want to change any of these settings, feel free to change and recompile or raise an issue on Github. See `IngestQueueProducer` for more details.
+
+If you want to clean up your existing installation, you can delete the old, unused queue like this:
+
+- `<payara install path>/bin/asadmin delete-connector-connection-pool --cascade=true jms IngestQueueConnectionFactoryPool`
+
+## Notes for Tool Developers and Integrators
+
+### Experimental Auxiliary File Support
+
+Experimental endpoints have been added to allow auxiliary files to be added to datafiles. These auxiliary files can be deposited and accessed via API. Later releases will include options for accessing these files through the UI. For more information, see the Auxiliary File Support section of the [Developer Guide](https://guides.dataverse.org/en/5.3/developers/).
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the [5.3 Milestone](https://github.com/IQSS/dataverse/milestone/93?closed=1) in Github.
+
+For help with upgrading, installing, or general questions please post to the [Dataverse Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org.
+
+## Installation
+
+If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.3/installation/).
+
+## Upgrade Instructions
+
+0\. These instructions assume that you've already successfully upgraded from Dataverse 4.x to Dataverse 5 following the instructions in the [Dataverse 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0).
+
+1\. Upgrade to Payara 5.2020.6 or higher.
+
+Instructions on how to update can be found in the
+[Payara documentation](https://docs.payara.fish/community/docs/5.2020.6/documentation/user-guides/upgrade-payara.html).
+
+It would likely be safer to upgrade Payara first, while still running Dataverse 5.2, and then proceed with the steps below. Upgrading from an earlier version of Payara should be a straightforward process: Undeploy Dataverse; stop Payara; move the current Payara directory out of the way; unzip the new Payara version in its place; replace **the brand new `payara/glassfish/domains/domain1` with your old, preserved `domain1`**; start Payara, deploy Dataverse 5.2. We still recommend that you read the detailed upgrade instructions above; and, if you run into any issues with this upgrade, it will help to be able to separate them from any problems with the upgrade of Dataverse proper. 
+If you are still using pre-5.0 version of Dataverse, and Glassfish version 4, please follow the upgrade instructions in the Dataverse 5.0 release notes; but use the latest version of Payara 5 (5.2020.7, as of this writing).
+
+2\. Undeploy the previous version.
+
+- `<payara install path>/bin/asadmin list-applications`
+- `<payara install path>/bin/asadmin undeploy dataverse<-version>`
+
+(where `<payara install path>` is where Payara 5 is installed, for example: `/usr/local/payara5`)
+
+3\. Update your database connection.
+
+Please configure your connection details, replacing all the `${DB_...}`.
+(If you are using a PostgreSQL server on `localhost:5432`, you can omit `dataverse.db.host` and `dataverse.db.port`.)
+
+- `<payara install path>/bin/asadmin create-system-properties "dataverse.db.user=${DB_USER}"`
+- `<payara install path>/bin/asadmin create-system-properties "dataverse.db.host=${DB_HOST}"`
+- `<payara install path>/bin/asadmin create-system-properties "dataverse.db.port=${DB_PORT}"`
+- `<payara install path>/bin/asadmin create-system-properties "dataverse.db.name=${DB_NAME}"`
+- `echo "AS_ADMIN_ALIASPASSWORD=${DB_PASS}" > /tmp/password.txt`
+
+- `<payara install path>/bin/asadmin create-password-alias --passwordfile /tmp/password.txt dataverse.db.password`
+- `rm /tmp/password.txt`
+
+4\. In domain.xml, verify that the __TimerPool jdbc-connection-pool is using the H2 database, as follows (if you have the old Derby version from Glassfish 4, replace it):
+
+  `<jdbc-connection-pool datasource-classname="org.h2.jdbcx.JdbcDataSource" name="__TimerPool" res-type="javax.sql.XADataSource"> <property name="URL" value="jdbc:h2:${com.sun.aas.instanceRoot}/lib/databases/ejbtimer;AUTO_SERVER=TRUE"></property> </jdbc-connection-pool>`
+
+5\. Reset the EJB timer database back to default:
+
+- `<payara install path>/bin/asadmin set configs.config.server-config.ejb-container.ejb-timer-service.timer-datasource=jdbc/__TimerPool`
+
+6\. Delete the old password alias and DB pool:
+
+- `<payara install path>/bin/asadmin delete-jdbc-connection-pool --cascade=true dvnDbPool`
+- `<payara install path>/bin/asadmin delete-password-alias db_password_alias`
+
+7\. Stop payara, remove the generated and ejbtimer database directories, then restart.
+
+- `service payara stop`
+- `rm -rf <payara install path>/glassfish/domains/domain1/generated`
+- `rm -rf <payara install path>/glassfish/domains/domain1/lib/databases/ejbtimer`
+- `service payara start`
+
+8\. Deploy this version.
+
+- `<payara install path>/bin/asadmin deploy dataverse-5.3.war`
+
+9\. Restart payara
+
+- `service payara stop`
+- `service payara start`

doc/sphinx-guides/source/api/faq.rst

@@ -35,7 +35,7 @@ SWORD only supports a dozen or so operations. The Native API supports many more.
 To Operate on a Dataset Should I Use Its DOI (or Handle) or Its Database ID?
 ----------------------------------------------------------------------------
 
-It is fine to target a datasets using either its Persistent ID (PID such as DOI or Handle) or its database id.
+It is fine to target a dataset using either its Persistent ID (PID such as DOI or Handle) or its database id.
 
 Here's an example from :ref:`publish-dataset-api` of targeting a dataset using its DOI:
 

doc/sphinx-guides/source/conf.py

@@ -65,9 +65,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '5.2'
+version = '5.3'
 # The full version, including alpha/beta/rc tags.
-release = '5.2'
+release = '5.3'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.