Merge branch 'develop' into 3438-email-verification

doc/release-notes/7409-remove-worldmap.md

@@ -0,0 +1,9 @@
+## Release Highlights
+
+### Worldmap/Geoconnect Integration Now Obsolete
+
+As of this release, the Geoconnect/Worldmap integration is no longer available. The Harvard University Worldmap is going through a migration process, and instead of updating this code to work with the new infrastructure, the decision was made to pursue future Geospatial exploration/analysis through other tools, following the External Tools Framework in the Dataverse Software.
+
+## Notes to Admins
+
+Take a backup of the Worldmap links, if any.

doc/release-notes/7502-more-mime-types.md

@@ -0,0 +1,7 @@
+## Upgrade Steps
+
+In addition to mapping friendly names to these file types, the types are further mapped to aggregate file types facets on the homepage. A full reindex is required for the facets to be refreshed.
+
+Kick off full reindex
+
+http://guides.dataverse.org/en/4.20/admin/solr-search-index.html

doc/sphinx-guides/source/admin/index.rst

@@ -21,7 +21,6 @@ This guide documents the functionality only available to superusers (such as "da
    timers
    make-data-count
    integrations
-   geoconnect-worldmap
    user-administration
    dataverses-datasets
    solr-search-index

doc/sphinx-guides/source/admin/integrations.rst

@@ -85,13 +85,6 @@ TwoRavens is a web application for tabular data exploration and statistical anal
 
 For installation instructions, see the :doc:`external-tools` section.
 
-WorldMap
-++++++++
-
-WorldMap helps researchers visualize and explore geospatial data by creating maps.
-
-For installation instructions, see :doc:`geoconnect-worldmap`.
-
 Compute Button
 ++++++++++++++
 

doc/sphinx-guides/source/api/apps.rst

@@ -65,13 +65,6 @@ OSF allows you to view, download, and upload files to and from a Dataverse datas
 
 https://github.com/CenterForOpenScience/osf.io/tree/develop/addons/dataverse
 
-Geoconnect
-~~~~~~~~~~
-
-Geoconnect allows Dataverse files to be visualized and explored on `WorldMap <http://worldmap.harvard.edu>`_. Read more about it in the :doc:`/user/data-exploration/worldmap` section of the User Guide.
-
-https://github.com/IQSS/geoconnect
-
 dataverse-metrics
 ~~~~~~~~~~~~~~~~~
 

doc/sphinx-guides/source/api/intro.rst

@@ -198,7 +198,6 @@ Please note that some APIs are only documented in other guides that are more sui
   - :doc:`/admin/metadatacustomization`  
   - :doc:`/admin/metadataexport`
   - :doc:`/admin/make-data-count`
-  - :doc:`/admin/geoconnect-worldmap`
   - :doc:`/admin/solr-search-index`
 
 - Installation Guide

doc/sphinx-guides/source/developers/geospatial.rst

@@ -5,19 +5,12 @@ Geospatial Data
 .. contents:: |toctitle|
 	:local:
 
-Geoconnect
-----------
-
-Geoconnect works as a middle layer, allowing geospatial data files in Dataverse to be visualized with Harvard WorldMap. To set up a Geoconnect development environment, you can follow the steps outlined in the `local_setup.md <https://github.com/IQSS/geoconnect/blob/master/local_setup.md>`_ guide. You will need Python and a few other prerequisites.
-
-As mentioned under the :ref:`architecture` section of Preparation in the Installation Guide, Geoconnect is an optional component of Dataverse, so this section is only necessary to follow it you are working on an issue related to this feature.
-
 How Dataverse Ingests Shapefiles
 --------------------------------
 
-A shapefile is a set of files, often uploaded/transferred in ``.zip`` format. This set may contain up to fifteen files. A minimum of three specific files (``.shp``, ``.shx``, ``.dbf``) are needed to be a valid shapefile and a fourth file (``.prj``) is required for WorldMap -- or any type of meaningful visualization.
+A shapefile is a set of files, often uploaded/transferred in ``.zip`` format. This set may contain up to fifteen files. A minimum of three specific files (``.shp``, ``.shx``, ``.dbf``) are needed to be a valid shapefile and a fourth file (``.prj``) is required for some applications -- or any type of meaningful visualization.
 
-For ingest and connecting to WorldMap, four files are the minimum required:
+For ingest, four files are the minimum required:
 
 - ``.shp`` - shape format; the feature geometry itself
 - ``.shx`` - shape index format; a positional index of the feature geometry to allow seeking forwards and backwards quickly
@@ -89,127 +82,6 @@ For two "final" shapefile sets, ``bicycles.zip`` and ``subway_line.zip``, a new
 - Mimetype: ``application/zipped-shapefile``
 - Mimetype Label: "Shapefile as ZIP Archive"
 
-WorldMap JoinTargets + API Endpoint
------------------------------------
-
-WorldMap supplies target layers -- or JoinTargets -- that a tabular file may be mapped against. A JSON description of these `CGA <http://gis.harvard.edu>`_-curated JoinTargets may be retrieved via API at ``http://worldmap.harvard.edu/datatables/api/jointargets/``. Please note: login is required. You may use any WorldMap account credentials via HTTP Basic Auth.
-
-Example of JoinTarget information returned via the API:
-
-.. code-block:: json
-
-    {
-	  "data":[
-	    {
-	      "layer":"geonode:census_tracts_2010_boston_6f6",
-	      "name":"Census Tracts, Boston (GEOID10: State+County+Tract)",
-	      "geocode_type_slug":"us-census-tract",
-	      "geocode_type":"US Census Tract",
-	      "attribute":{
-	        "attribute":"CT_ID_10",
-	        "type":"xsd:string"
-	      },
-	      "abstract":"As of the 2010 census, Boston, MA contains 7,288 city blocks [truncated for example]",
-	      "title":"Census Tracts 2010, Boston (BARI)",
-	      "expected_format":{
-	        "expected_zero_padded_length":-1,
-	        "is_zero_padded":false,
-	        "description":"Concatenation of state, county and tract for 2010 Census Tracts.  Reference: https://www.census.gov/geo/maps-data/data/tract_rel_layout.html\r\n\r\nNote:  Across the US, this can be a zero-padded \"string\" but the original Boston layer has this column as \"numeric\" ",
-	        "name":"2010 Census Boston GEOID10 (State+County+Tract)"
-	      },
-	      "year":2010,
-	      "id":28
-	    },
-	    {
-	      "layer":"geonode:addresses_2014_boston_1wr",
-	      "name":"Addresses, Boston",
-	      "geocode_type_slug":"boston-administrative-geography",
-	      "geocode_type":"Boston, Administrative Geography",
-	      "attribute":{
-	        "attribute":"LocationID",
-	        "type":"xsd:int"
-	      },
-	      "abstract":"Unique addresses present in the parcels data set, which itself is derived from [truncated for example]",
-	      "title":"Addresses 2015, Boston (BARI)",
-	      "expected_format":{
-	        "expected_zero_padded_length":-1,
-	        "is_zero_padded":false,
-	        "description":"Boston, Administrative Geography, Boston Address Location ID.  Example: 1, 2, 3...nearly 120000",
-	        "name":"Boston Address Location ID (integer)"
-	      },
-	      "year":2015,
-	      "id":18
-	    },
-	    {
-	      "layer":"geonode:bra_neighborhood_statistical_areas_2012__ug9",
-	      "name":"BRA Neighborhood Statistical Areas, Boston",
-	      "geocode_type_slug":"boston-administrative-geography",
-	      "geocode_type":"Boston, Administrative Geography",
-	      "attribute":{
-	        "attribute":"BOSNA_R_ID",
-	        "type":"xsd:double"
-	      },
-	      "abstract":"BRA Neighborhood Statistical Areas 2015, Boston. Provided by [truncated for example]",
-	      "title":"BRA Neighborhood Statistical Areas 2015, Boston (BARI)",
-	      "expected_format":{
-	        "expected_zero_padded_length":-1,
-	        "is_zero_padded":false,
-	        "description":"Boston, Administrative Geography, Boston BRA Neighborhood Statistical Area ID (integer).  Examples: 1, 2, 3, ... 68, 69",
-	        "name":"Boston BRA Neighborhood Statistical Area ID (integer)"
-	      },
-	      "year":2015,
-	      "id":17
-	    }
-	  ],
-	  "success":true
-    }
-
-How Geoconnect Uses Join Target Information
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-When a user attempts to map a tabular file, the application looks in the Geoconnect database for ``JoinTargetInformation``. If this information is more than 10 minutes* old, the application will retrieve fresh information and save it to the db.
-
-(* Change the timing via the Django settings variable ``JOIN_TARGET_UPDATE_TIME``.)
-
-This JoinTarget info is used to populate HTML forms used to match a tabular file column to a JoinTarget column. Once a JoinTarget is chosen, the JoinTarget ID is an essential piece of information used to make an API call to the WorldMap and attempt to map the file.
-
-Retrieving Join Target Information from WorldMap API
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The ``get_join_targets()`` function in ``dataverse_layer_services.py`` uses the WorldMap API, retrieves a list of available tabular file JointTargets. (See the `dataverse_layer_services code in GitHub <https://github.com/IQSS/geoconnect/blob/master/gc_apps/worldmap_connect/dataverse_layer_services.py#L275>`_.)
-
-Saving Join Target Information to Geoconnect Database
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The ``get_latest_jointarget_information()`` in ``utils.py`` retrieves recent JoinTarget Information from the database. (See the `utils code in GitHub <https://github.com/IQSS/geoconnect/blob/master/gc_apps/worldmap_connect/utils.py#L16>`_.)
-
-Setting Up WorldMap Sample Data
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-For the dataset page, this script gives a query to add test WorldMap map data. After the query is run, the explore in WorldMap option should display for tabular files or shapefiles. In the example SQL queries below, substitute ``$DATASET_ID`` and ``$DATAFILE_ID`` with the appropriate ID's.
-
-To add sample map data for a tabular file:
-
-.. code::
-
-    INSERT INTO maplayermetadata (id, isjoinlayer, joindescription, embedmaplink, layerlink, layername, mapimagelink, worldmapusername, dataset_id, datafile_id) 
-    VALUES (DEFAULT, true, 'This file was joined with WorldMap layer x, y, z',
-    'https://worldmap.harvard.edu/maps/embed/?layer=geonode:zip_codes_2015_zip_s9i','https://worldmap.harvard.edu/data/geonode:zip_codes_2015_zip_s9i',
-    'geonode:zip_codes_2015_zip_s9i',
-    'http://worldmap.harvard.edu/download/wms/27289/png?layers=geonode%3Azip_codes_2015_zip_s9i&#38;width=865&#38;bbox=-71.1911091251%2C42.2270382738%2C-70.9228275369%2C42.3976144794&#38;service=WMS&#38;format=image%2Fpng&#38;srs=EPSG%3A4326&#38;request=GetMap&#38;height=550',
-    'admin',$DATASET_ID,$DATAFILE_ID});
-
-To add sample map data for a tabular shapefile:
-
-.. code::
-
-    INSERT INTO maplayermetadata (id, isjoinlayer, embedmaplink, layerlink, layername, mapimagelink, worldmapusername, dataset_id, datafile_id) 
-    VALUES (DEFAULT, false,
-    'https://worldmap.harvard.edu/maps/embed/?layer=geonode:zip_codes_2015_zip_s9i','https://worldmap.harvard.edu/data/geonode:zip_codes_2015_zip_s9i',
-    'geonode:zip_codes_2015_zip_s9i',
-    'http://worldmap.harvard.edu/download/wms/27289/png?layers=geonode%3Azip_codes_2015_zip_s9i&#38;width=865&#38;bbox=-71.1911091251%2C42.2270382738%2C-70.9228275369%2C42.3976144794&#38;service=WMS&#38;format=image%2Fpng&#38;srs=EPSG%3A4326&#38;request=GetMap&#38;height=550',
-    'admin',$DATASET_ID,$DATAFILE_ID);
-
 ----
 
 Previous: :doc:`unf/index` | Next: :doc:`remote-users`

doc/sphinx-guides/source/developers/intro.rst

@@ -64,7 +64,6 @@ As a developer, you also may be interested in these projects related to Datavers
 - dataverse-metrics - aggregate and visualize metrics for installations of Dataverse around the world: https://github.com/IQSS/dataverse-metrics
 - Configuration management scripts - Ansible, Puppet, etc.: See :ref:`advanced` section in the Installation Guide.
 - :doc:`/developers/unf/index` (Java) -  a Universal Numerical Fingerprint: https://github.com/IQSS/UNF
-- GeoConnect (Python) - create a map by uploading files to Dataverse: https://github.com/IQSS/geoconnect
 - `DataTags <https://github.com/IQSS/DataTags>`_ (Java and Scala) - tag datasets with privacy levels: https://github.com/IQSS/DataTags
 - `TwoRavens <http://2ra.vn>`_ (Javascript) - a `d3.js <http://d3js.org>`_ interface for exploring data and running Zelig models: https://github.com/IQSS/TwoRavens
 - `Zelig <http://zeligproject.org>`_ (R) - run statistical models on files uploaded to Dataverse: https://github.com/IQSS/Zelig

doc/sphinx-guides/source/installation/config.rst

@@ -774,7 +774,7 @@ Once you have created the analytics file, run this curl command to add it to you
 Tracking Button Clicks
 ++++++++++++++++++++++
 
-The basic analytics configuration above tracks page navigation. However, it does not capture potentially interesting events, such as those from users clicking buttons on pages, that do not result in a new page opening. In Dataverse, these events include file downloads, requesting access to restricted data, exporting metadata, social media sharing, requesting citation text, launching external tools or WorldMap, contacting authors, and launching computations.
+The basic analytics configuration above tracks page navigation. However, it does not capture potentially interesting events, such as those from users clicking buttons on pages, that do not result in a new page opening. In Dataverse, these events include file downloads, requesting access to restricted data, exporting metadata, social media sharing, requesting citation text, launching external tools, contacting authors, and launching computations.
 
 Both Google and Matomo provide the optional capability to track such events and Dataverse has added CSS style classes (btn-compute, btn-contact, btn-download, btn-explore, btn-export, btn-preview, btn-request, btn-share, and downloadCitation) to it's HTML to facilitate it.
 
@@ -1758,20 +1758,6 @@ The ``:TwoRavensUrl`` option is no longer valid. See :doc:`r-rapache-tworavens`
 
 The ``:TwoRavensTabularView`` option is no longer valid. See :doc:`r-rapache-tworavens` and the :doc:`/admin/external-tools` section of the Admin Guide.
 
-:GeoconnectCreateEditMaps
-+++++++++++++++++++++++++
-
-Set ``GeoconnectCreateEditMaps`` to true to allow the user to create maps using Geoconnect. This boolean enables the map configure tool option for a data file and the ingest to create a shape file.
-
-``curl -X PUT -d true http://localhost:8080/api/admin/settings/:GeoconnectCreateEditMaps``
-
-:GeoconnectViewMaps
-+++++++++++++++++++
-
-Set ``GeoconnectViewMaps`` to true to allow a user to view existing maps. This boolean enables the map explore tool option for a data file.
-
-``curl -X PUT -d true http://localhost:8080/api/admin/settings/:GeoconnectViewMaps``
-
 .. _:DatasetPublishPopupCustomText:
 
 :DatasetPublishPopupCustomText

doc/sphinx-guides/source/installation/index.rst

@@ -17,7 +17,6 @@ Installation Guide
    config
    upgrading
    r-rapache-tworavens
-   geoconnect
    shibboleth
    oauth2
    oidc

doc/sphinx-guides/source/installation/installation-main.rst

@@ -90,7 +90,6 @@ For more information, please see https://docs.payara.fish/documentation/payara-s
 - exporting to Schema.org format (and showing JSON-LD in HTML's <meta/> tag)
 - exporting to DDI format
 - which Dataverse installation an "external tool" should return to
-- which Dataverse installation Geoconnect should return to
 - URLs embedded in SWORD API responses
 
 The supplied site URL will be saved under the JVM option :ref:`dataverse.siteUrl`.