Merge pull request #8391 from GlobalDataverseCommunityConsortium/IQSS/#…

…8380-counterprocesor_version_update

Iqss/#8380 counter-processor version update

doc/release-notes/8380-counter-processor-update.md

@@ -0,0 +1 @@
+This release includes support for counter-processor-0.1.04 for processing Make Data Count metrics. If you are running Make Data Counts support, you should reinstall/reconfigure counter-processor as described in the latest Guides. (For existing installations, note that counter-processor-0.1.04 requires a newer version of python so you will need to follow the full counter-processor install. Also note that if you configure the new version the same way, it will reprocess the days in the current month when it is first run. This is normal and will not affect the metrics in Dataverse.)

doc/sphinx-guides/source/_static/util/counter_daily.sh

@@ -1,6 +1,6 @@
 #! /bin/bash
 
-COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-0.0.1"
+COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-0.1.04"
 MDC_LOG_DIRECTORY="/usr/local/payara5/glassfish/domains/domain1/logs/mdc"
 
 # counter_daily.sh

doc/sphinx-guides/source/admin/make-data-count.rst

@@ -83,9 +83,9 @@ Configure Counter Processor
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-0.0.1``
+  * ``cd /usr/local/counter-processor-0.1.04``
 
-* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-0.0.1``.
+* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-0.1.04``.
 
 * Edit the config file and pay particular attention to the FIXME lines.
 
@@ -98,7 +98,7 @@ Soon we will be setting up a cron job to run nightly but we start with a single
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-0.0.1``
+  * ``cd /usr/local/counter-processor-0.1.04``
 
 * If you are running Counter Processor for the first time in the middle of a month, you will need create blank log files for the previous days. e.g.:
 
@@ -112,7 +112,7 @@ Soon we will be setting up a cron job to run nightly but we start with a single
 
 * Run Counter Processor.
 
-  * ``CONFIG_FILE=counter-processor-config.yaml python36 main.py``
+  * ``CONFIG_FILE=counter-processor-config.yaml python39 main.py``
 
   * A JSON file in SUSHI format will be created in the directory you specified under "output_file" in the config file.
 

doc/sphinx-guides/source/developers/make-data-count.rst

@@ -51,9 +51,9 @@ Once you are done with your configuration, you can run Counter Processor like th
 
 ``su - counter``
 
-``cd /usr/local/counter-processor-0.0.1``
+``cd /usr/local/counter-processor-0.1.04``
 
-``CONFIG_FILE=counter-processor-config.yaml python36 main.py``
+``CONFIG_FILE=counter-processor-config.yaml python39 main.py``
 
 (Please note that the Counter Processor README says you can also pass in values like ``START_DATE``, ``END_DATE`` etc. at the command line if you find this to be more convenient.)
 
@@ -84,7 +84,7 @@ Second, if you are also sending your SUSHI report to Make Data Count, you will n
 
 ``curl -H "Authorization: Bearer $JSON_WEB_TOKEN" -X DELETE https://$MDC_SERVER/reports/$REPORT_ID``
 
-To get the ``REPORT_ID``, look at the logs generated in ``/usr/local/counter-processor-0.0.1/tmp/datacite_response_body.txt``
+To get the ``REPORT_ID``, look at the logs generated in ``/usr/local/counter-processor-0.1.04/tmp/datacite_response_body.txt``
 
 To read more about the Make Data Count api, see https://github.com/datacite/sashimi
 

doc/sphinx-guides/source/installation/prerequisites.rst

@@ -96,7 +96,7 @@ PostgreSQL
 ----------
 
 Installing PostgreSQL
-=======================
+=====================
 
 The application has been tested with PostgreSQL versions up to 13 and version 10+ is required. We recommend installing the latest version that is available for your OS distribution. *For example*, to install PostgreSQL 13 under RHEL7/derivative::
 
@@ -416,43 +416,51 @@ Counter Processor is required to enable Make Data Count metrics in a Dataverse i
 Installing Counter Processor
 ============================
 
-Counter Processor has only been tested on el7 (see "Linux" above). Please note that a scripted installation using Ansible is mentioned in the :doc:`/developers/make-data-count` section of the Developer Guide.
+A scripted installation using Ansible is mentioned in the :doc:`/developers/make-data-count` section of the Developer Guide.
 
 As root, download and install Counter Processor::
 
         cd /usr/local
-        wget https://github.com/CDLUC3/counter-processor/archive/v0.0.1.tar.gz
-        tar xvfz v0.0.1.tar.gz
+        wget https://github.com/CDLUC3/counter-processor/archive/v0.1.04.tar.gz
+        tar xvfz v0.1.04.tar.gz
+        cd /usr/local/counter-processor-0.1.04
+
+Installing GeoLite Country Database
+===================================
+
+Counter Processor can report per country results if the optional GeoLite Country Database is installed. At present, this database is free but to use it one must signing an agreement (EULA) with MaxMind. 
+(The primary concern appears to be that individuals can opt-out of having their location tracked via IP address and, due to various privacy laws, MaxMind needs a way to comply with that for products it has "sold" (for no cost in this case). Their agreement requires you to either configure automatic updates to the GeoLite Country database or be responsible on your own for managing take down notices.)
+The process required to sign up, download the database, and to configure automated updating is described at https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/ and the links from that page.
 
-As root, change to the Counter Processor directory you just created, download the GeoLite2-Country tarball, untar it, and copy the geoip database into place::
+As root, change to the Counter Processor directory you just created, download the GeoLite2-Country tarball from MaxMind, untar it, and copy the geoip database into place::
 
-        cd /usr/local/counter-processor-0.0.1
-        wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz
+        <download or move the GeoLite2-Country.tar.gz to the /usr/local/counter-processor-0.1.04 directory>
         tar xvfz GeoLite2-Country.tar.gz
         cp GeoLite2-Country_*/GeoLite2-Country.mmdb maxmind_geoip
 
+Creating a counter User
+=======================
+
 As root, create a "counter" user and change ownership of Counter Processor directory to this new user::
 
         useradd counter
-        chown -R counter:counter /usr/local/counter-processor-0.0.1
+        chown -R counter:counter /usr/local/counter-processor-0.1.04
 
 Installing Counter Processor Python Requirements
 ================================================
 
-Counter Processor requires Python 3.6.4 or higher. The following commands are intended to be run as root but we are aware that Pythonistas might prefer fancy virtualenv or similar setups. Pull requests are welcome to improve these steps!
-
-Enable the EPEL repo if you haven't already::
+Counter Processor version 0.1.04 requires Python 3.7 or higher. This version of Python is available in many operating systems, and is purportedly available for RHEL7 or CentOS 7 via Red Hat Software Collections. Alternately, one may compile it from source.
 
-        yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
+The following commands are intended to be run as root but we are aware that Pythonistas might prefer fancy virtualenv or similar setups. Pull requests are welcome to improve these steps!
 
-Install Python 3.6::
+Install Python 3.9::
 
-        yum install python36
+        yum install python39
 
 Install Counter Processor Python requirements::
 
-        python3.6 -m ensurepip
-        cd /usr/local/counter-processor-0.0.1
+        python3.9 -m ensurepip
+        cd /usr/local/counter-processor-0.1.04
         pip3 install -r requirements.txt
 
 See the :doc:`/admin/make-data-count` section of the Admin Guide for how to configure and run Counter Processor.

scripts/vagrant/counter-processor-config.yaml

@@ -44,7 +44,7 @@ output_file: /dataverse/sushi_sample_logs
 output_format: json
 
 # the name of the platform that goes into your reports
-platform: Dash
+platform: Dataverse
 
 # Don't put your api token in here if you're going to commit it, but put in separate secrets.yaml in same
 # directory as the config or else set a environment variable when starting up in order to override the key.

scripts/vagrant/setup-counter-processor.sh

@@ -11,7 +11,7 @@ python3.6 -m ensurepip
 COUNTER_USER=counter
 echo "Ensuring Unix user '$COUNTER_USER' exists"
 useradd $COUNTER_USER || :
-COMMIT='a73dbced06f0ac2f0d85231e4d9dd4f21bee8487'
+COMMIT='7974dad259465ba196ef639f48dea007cae8f9ac'
 UNZIPPED_DIR="counter-processor-$COMMIT"
 if [ ! -e $UNZIPPED_DIR ]; then
   ZIP_FILE="${COMMIT}.zip"
@@ -20,19 +20,7 @@ if [ ! -e $UNZIPPED_DIR ]; then
   unzip $ZIP_FILE
 fi
 cd $UNZIPPED_DIR
-GEOIP_DIR='maxmind_geoip'
-GEOIP_FILE='GeoLite2-Country.mmdb'
-GEOIP_PATH_TO_FILE="$GEOIP_DIR/$GEOIP_FILE"
-if [ ! -e $GEOIP_PATH_TO_FILE ]; then
-  echo "let's do this thing"
-  TARBALL='GeoLite2-Country.tar.gz'
-  wget https://geolite.maxmind.com/download/geoip/database/$TARBALL
-  tar xfz GeoLite2-Country.tar.gz
-  # Glob (*) below because of directories like "GeoLite2-Country_20181204".
-  GEOIP_UNTARRED_DIR='GeoLite2-Country_*'
-  mv $GEOIP_UNTARRED_DIR/$GEOIP_FILE $GEOIP_PATH_TO_FILE
-  rm -rf $TARBALL $GEOIP_UNTARRED_DIR
-fi
+echo "Installation of the GeoLite2 country database for counter-processor can no longer be automated. See the Installation Guide for the manual installation process."
 pip3 install -r requirements.txt
 # For now, parsing sample_logs/counter_2018-05-08.log
 for i in `echo {00..31}`; do

src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetMetricsServiceBean.java

@@ -118,7 +118,12 @@ public List<DatasetMetrics> parseSushiReport(JsonObject report){
 
     public List<DatasetMetrics> parseSushiReport(JsonObject report, Dataset dataset) {
         List<DatasetMetrics> datasetMetricsAll = new ArrayList<>();
-        JsonArray reportDatasets = report.getJsonArray("report_datasets");
+        //Current counter-processor v 0.1.04+ format
+        JsonArray reportDatasets = report.getJsonArray("report-datasets");
+        if(reportDatasets==null) {
+            //Try counter-processor v 0.0.1 name
+            reportDatasets = report.getJsonArray("report_datasets");
+        }
         for (JsonValue reportDataset : reportDatasets) {
             List<DatasetMetrics> datasetMetricsDataset = new ArrayList<>();
             String globalId = null;