Merge pull request #11 from IQSS/master

2nd Beta Deployment, 5/20

.gitignore

@@ -12,3 +12,5 @@ michael-local
 ehthumbs.db
 Thumbs.db
 .vagrant
+*.pyc
+scripts/api/py_api_wrapper/demo-data/*

doc/Sphinx/source/API/data-deposit.rst

@@ -0,0 +1,8 @@
+Data Deposit API
+================
+
+Please note that while Dataverse Network (DVN) 3.6 and higher support a
+new Data Deposit API documented at
+http://thedata.harvard.edu/guides/dataverse-api-main.html#data-deposit-api
+the API has not yet been added Dataverse 4.0 Beta.  That work will be
+completed in https://redmine.hmdc.harvard.edu/issues/3385 

doc/Sphinx/source/API/dataverse-apis.rst → doc/Sphinx/source/API/data-sharing.rst

@@ -1,13 +1,5 @@
-Dataverse Network APIs
-++++++++++++++++++++++++
-
-We strongly encourage anyone interested in building tools to
-interoperate with the Dataverse Network to utilize our open source
-APIs. Please visit our `website <http://thedata.org/book/apps>`__  for
-examples of external apps that have been built to work with our APIs.
-
 Data Sharing API
-==================
+================
 
 As of version 3.0, a new API for programmatic access to the DVN data and
 metadata has been added. The API allows a remote, non-DVN
@@ -307,17 +299,3 @@ type, file name and such)
 | Authorization header supplied, but the authenticated user is not
 | authorized to directly access the object protected by Access
 | Permissions and/or Access Restrictions (“Terms of Use”).
-
-
-Data Deposit API
-===========
-
-As of version 3.6, a new API for programmatic deposit of data and metadata to the Dataverse Network will be added. The API will allow a remote, non-Dataverse Network archive/application to deposit files and metadata to a Dataverse Network installation.
-
-The latest information on this plugin is available here:
-`https://redmine.hmdc.harvard.edu/issues/3108 <https://redmine.hmdc.harvard.edu/issues/3108>`__
-
-
-
-
-

doc/Sphinx/source/API/index.rst

@@ -4,11 +4,17 @@
    contain the root `toctree` directive.
 
 API Guide
-=======================================================
+=========
+
+We strongly encourage anyone interested in building tools to
+interoperate with the Dataverse Network to utilize our open source
+APIs. Please visit http://datascience.iq.harvard.edu/collaborations for
+examples of external apps that have been built to work with our APIs.
 
 Contents:
 
 .. toctree::
    :maxdepth: 2
 
-  dataverse-apis
+  data-sharing
+  data-deposit

doc/Sphinx/source/Developers/index.rst

@@ -12,5 +12,6 @@ Contents:
    :maxdepth: 2
 
   dev-main
+  tools
 
 

doc/Sphinx/source/Developers/tools.rst

@@ -0,0 +1,20 @@
+=====
+Tools
+=====
+
+PageKite
+++++++++
+
+PageKite is a fantastic service that can be used to share your
+local development environment over the Internet on a public IP address.
+
+With PageKite running on your laptop, the world can access a URL such as
+http://pdurbin.pagekite.me to see what you see at http://localhost:8080
+
+Sign up at https://pagekite.net and follow the installation instructions or simply download https://pagekite.net/pk/pagekite.py
+
+The first time you run ``./pagekite.py`` a file at ``~/.pagekite.rc`` will be
+created. You can edit this file to configure PageKite to serve up port 8080
+(the default GlassFish HTTP port) or the port of your choosing.
+
+According to https://pagekite.net/support/free-for-foss/ PageKite (very generously!) offers free accounts to developers writing software the meets http://opensource.org/docs/definition.php such as Dataverse.

doc/Sphinx/source/User/appendix.rst

@@ -16,39 +16,12 @@ currently proposed metadata fields for 4.0 (per metadata block):
 - `Citation Metadata <https://docs.google.com/spreadsheet/ccc?key=0AjeLxEN77UZodDBaYTFPakhGaEpoa3hqZUJTOWZtclE&usp=sharing>`__ (compliant with `DDI 2.5 <http://www.ddialliance.org/>`__ and `DataCite 3.0 <http://schema.datacite.org/meta/kernel-3/index.html>`__) 
 - `Social Science & Humanities Metadata (DDI 2.5 compliant) <https://docs.google.com/spreadsheet/ccc?key=0AjeLxEN77UZodEppcTFHT1NnajNLV0tacE10NEdmUnc&usp=sharing>`__
 - `Astronomy and Astrophysics Metadata <https://docs.google.com/spreadsheet/ccc?key=0AjeLxEN77UZodEp4Qmp0QURkUWo1S0t4X3hia0FnZUE&usp=sharing>`__
-  : These metadata elements can be mapped/exported to the International Virtual Observatory Alliance’s (IVOA) Resource Metadata for the Virtual Observatory (VOResource Schema format) and based on `Virtual Observatory (VO) Discovery and Provenance Metadata <http://www.wf4ever-project.org/wiki/download/attachments/1179927/DPmetadata.pdf?version=1&modificationDate=1337186963000>`__) 
+  : These metadata elements can be mapped/exported to the International Virtual Observatory Alliance’s (IVOA) 
+  `VOResource Schema format <http://www.ivoa.net/documents/latest/RM.html>`__ and is based on 
+  `Virtual Observatory (VO) Discovery and Provenance Metadata <http://www.wf4ever-project.org/wiki/download/attachments/1179927/DPmetadata.pdf?version=1&modificationDate=1337186963000>`__
 - `Biomedical Metadata <https://docs.google.com/spreadsheet/ccc?key=0AjeLxEN77UZodExsRTB2SEpVWWd1Qmx6M09HSkExd3c&usp=sharing>`__ 
   (based on `ISA-Tab <http://isatab.sourceforge.net/format.html>`__ and `Stem Cell Commons <http://stemcellcommons.org/>`__)
 
-DDI 2.5
---------
 
-The Dataverse metadata is compliant with the `DDI schema
-version 2.5 <http://www.ddialliance.org/>`__. The Metadata fields in the Citation and Social Science blocks
-associated with each Dataset contain most of the fields
-in the study description section of the DDI. That way the Dataverse
-metadata can be mapped easily to a DDI, and be exported into XML
-format for preservation and interoperability.
-
-DataCite Schema 3.0
---------------------
-
-Dataverse metadata is compliant with the `DataCite Metadata Schema v.3.0 <http://schema.datacite.org/meta/kernel-3/index.html>`__ requirements. That way the Dataverse
-metadata can be mapped easily to a DataCite, and be exported into XML
-format for preservation and interoperability.
-
-DCMI Terms
------------
-
-Dataverse metadata is compliant with `Dublin Core Metadata Initiative Terms <http://dublincore.org/documents/dcmi-terms/>`__ (DCMI Terms) requirements. That way the Dataverse
-metadata can be mapped easily to DCMI Terms, and be exported into XML
-format for preservation and interoperability.
-
-
-FGDC/CSDGM (imports)
----------------------
-
-For imports only, Dataverse data is compliant with the `Content Standard
-for Digital Geospatial Metadata (CSDGM), Vers. 2 (FGDC-STD-001-1998) <http://www.fgdc.gov/metadata>`__ (FGDC).
 
 

pom.xml

@@ -61,7 +61,46 @@
             <version>3.0.1</version>
             <type>jar</type>
         </dependency>
-	<dependency>
+        <!-- BEGIN Data Deposit API v1 (SWORD v2) -->
+        <dependency>
+            <groupId>org.swordapp</groupId>
+            <artifactId>sword2-server</artifactId>
+            <version>1.0</version>
+            <type>jar</type>
+            <classifier>classes</classifier>
+        </dependency>
+        <dependency>
+            <groupId>org.swordapp</groupId>
+            <artifactId>sword2-server</artifactId>
+            <version>1.0</version>
+            <type>war</type>
+        </dependency>
+        <dependency>
+            <!-- required by org.swordapp.server.sword2-server -->
+            <groupId>org.apache.abdera</groupId>
+            <artifactId>abdera-core</artifactId>
+            <version>1.1.3</version>
+        </dependency>
+        <dependency>
+            <!-- required by org.swordapp.server.sword2-server -->
+            <groupId>org.apache.abdera</groupId>
+            <artifactId>abdera-parser</artifactId>
+            <version>1.1.3</version>
+        </dependency>
+        <dependency>
+            <!-- required by org.swordapp.server.sword2-server -->
+            <groupId>commons-fileupload</groupId>
+            <artifactId>commons-fileupload</artifactId>
+            <version>1.3.1</version>
+        </dependency>
+        <dependency>
+            <!-- required by org.swordapp.server.sword2-server -->
+            <groupId>xom</groupId>
+            <artifactId>xom</artifactId>
+            <version>1.1</version>
+        </dependency>
+        <!-- END Data Deposit API v1 (SWORD v2) -->
+        <dependency>
             <groupId>javax</groupId>
             <artifactId>javaee-api</artifactId>
             <version>7.0</version>

scripts/api/data-deposit/create-dataset

@@ -0,0 +1,7 @@
+#!/bin/bash
+USERNAME=pete
+PASSWORD=pete
+DVN_SERVER=localhost:8181
+DATAVERSE_ALIAS=peteTop
+curl --insecure --data-binary "@scripts/api/data-deposit/data/atom-entry-study.xml" -H "Content-Type: application/atom+xml" https://$USERNAME:$PASSWORD@$DVN_SERVER/dvn/api/data-deposit/v1/swordv2/collection/dataverse/$DATAVERSE_ALIAS \
+| xmllint -format -

scripts/api/data-deposit/data/atom-entry-study.xml

@@ -0,0 +1,41 @@
+<?xml version="1.0"?>
+<!--
+modified from http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html#protocoloperations_editingcontent_metadata
+
+comments at https://docs.google.com/document/d/1Rn70XItfA6_mJ4JA3I7m4em0spfdXIIHhwkrgCl40uE/edit?usp=sharing
+
+see also https://redmine.hmdc.harvard.edu/issues/3246
+-->
+<entry xmlns="http://www.w3.org/2005/Atom"
+       xmlns:dcterms="http://purl.org/dc/terms/">
+   <!-- some embedded metadata -->
+   <dcterms:title>Roasting at Home</dcterms:title>
+   <dcterms:creator>Peets, John</dcterms:creator>
+   <dcterms:creator>Stumptown, Jane</dcterms:creator>
+   <!-- Producer with financial or admin responsibility of the data -->
+   <dcterms:publisher>Coffee Bean State University</dcterms:publisher>
+   <!-- related publications -->
+   <dcterms:isReferencedBy holdingsURI="http://dx.doi.org/10.1038/dvn333" agency="DOI" IDNo="10.1038/dvn333">Peets, J., &amp; Stumptown, J. (2013). Roasting at Home. New England Journal of Coffee, 3(1), 22-34.</dcterms:isReferencedBy>
+   <!-- production date -->
+   <dcterms:date>2013-07-11</dcterms:date>
+   <!-- Other Identifier for the data in this study (or potentially global id if unused) -->
+   <!--
+   <dcterms:identifier>hdl:1XXZY.1/XYXZ</dcterms:identifier>
+   -->
+   <dcterms:description>Considerations before you start roasting your own coffee at home.</dcterms:description>
+   <!-- keywords -->
+   <dcterms:subject>coffee</dcterms:subject>
+   <dcterms:subject>beverage</dcterms:subject>
+   <dcterms:subject>caffeine</dcterms:subject>
+   <!-- geographic coverage -->
+   <dcterms:coverage>United States</dcterms:coverage>
+   <dcterms:coverage>Canada</dcterms:coverage>
+   <!-- kind of data -->
+   <dcterms:type>aggregate data</dcterms:type>
+   <!-- List of sources of the data collection-->
+   <dcterms:source>Stumptown, Jane. 2011. Home Roasting. Coffeemill Press.</dcterms:source>
+   <!-- restrictions -->
+   <dcterms:rights>Creative Commons CC-BY 3.0 (unported) http://creativecommons.org/licenses/by/3.0/</dcterms:rights>
+   <!-- related materials -->
+   <dcterms:relation>Peets, John. 2010. Roasting Coffee at the Coffee Shop. Coffeemill Press</dcterms:relation>
+</entry>

scripts/api/data-deposit/delete-dataset

@@ -0,0 +1,9 @@
+#!/bin/bash -x
+USERNAME=pete
+PASSWORD=pete
+DVN_SERVER=localhost:8181
+#GLOBAL_ID=hdl:TEST/12345
+GLOBAL_ID=`scripts/api/data-deposit/list-datasets | xpath '//id/text()' | cut -d'/' -f11,12`
+#curl --insecure -X DELETE https://$DVN_SERVER/api/datasets/$DATABASE_ID?key=$USERNAME
+curl --insecure -i -X DELETE https://$USERNAME:$PASSWORD@$DVN_SERVER/dvn/api/data-deposit/v1/swordv2/edit/study/$GLOBAL_ID
+#| xmllint -format -

scripts/api/data-deposit/list-datasets

@@ -0,0 +1,7 @@
+#!/bin/bash
+USERNAME=pete
+PASSWORD=pete
+DVN_SERVER=localhost:8181
+DATAVERSE_ALIAS=peteTop
+curl --insecure https://$USERNAME:$PASSWORD@$DVN_SERVER/dvn/api/data-deposit/v1/swordv2/collection/dataverse/$DATAVERSE_ALIAS \
+| xmllint -format -

scripts/api/data-deposit/service-document

@@ -0,0 +1,2 @@
+#!/bin/bash
+curl --insecure https://pete:pete@localhost:8181/dvn/api/data-deposit/v1/swordv2/service-document | xmllint -format -

scripts/api/data-deposit/show-atom-entry

@@ -0,0 +1,8 @@
+#!/bin/sh
+USERNAME=pete
+PASSWORD=pete
+DVN_SERVER=localhost:8181
+#GLOBAL_ID=hdl:TEST/12345
+GLOBAL_ID=`scripts/api/data-deposit/list-datasets | xpath '//id/text()' | cut -d'/' -f11,12`
+curl --insecure -s https://$USERNAME:$PASSWORD@$DVN_SERVER/dvn/api/data-deposit/v1/swordv2/edit/study/$GLOBAL_ID \
+| xmllint -format -

scripts/api/data-deposit/show-files

@@ -0,0 +1,3 @@
+#!/bin/sh
+#scripts/api/data-deposit/show-statement | xpath "//entry/content/@*[name()='type' or name()='src']"
+scripts/api/data-deposit/show-statement | xpath '//entry/id/text()' | cut -d'/' -f11,12

scripts/api/data-deposit/show-statement

@@ -0,0 +1,8 @@
+#!/bin/sh
+USERNAME=pete
+PASSWORD=pete
+DVN_SERVER=localhost:8181
+GLOBAL_ID=`scripts/api/data-deposit/list-datasets | xpath '//id/text()' | cut -d'/' -f11,12`
+curl --insecure -s https://$USERNAME:$PASSWORD@$DVN_SERVER/dvn/api/data-deposit/v1/swordv2/statement/study/$GLOBAL_ID \
+| xmllint -format - \
+#| xpath '//entry/title'

scripts/api/data-deposit/upload-file

@@ -0,0 +1,7 @@
+#!/bin/sh
+USERNAME=pete
+PASSWORD=pete
+DVN_SERVER=localhost:8181
+GLOBAL_ID=`scripts/api/data-deposit/list-datasets | xpath '//id/text()' | cut -d'/' -f11,12`
+curl -s --insecure --data-binary @scripts/api/data-deposit/data/example.zip -H "Content-Disposition: filename=example.zip" -H "Content-Type: application/zip" -H "Packaging: http://purl.org/net/sword/package/SimpleZip" https://$USERNAME:$PASSWORD@$DVN_SERVER/dvn/api/data-deposit/v1/swordv2/edit-media/study/$GLOBAL_ID \
+| xmllint -format -