Merge pull request #4426 from IQSS/3657-tworavens-as-modular-explore

3657 tworavens as modular explore

CONTRIBUTING.md

@@ -1,22 +1,19 @@
 # Contributing to Dataverse
 
-Thank you for your interest in contributing to Dataverse!  We are open to contributions from everyone. You don't need permission to participate, just jump in using the resources below.  If you have questions, reach out to us on the [#dataverse IRC channel][], and hang around a while, as it may take time for community members to de-idle.
+Thank you for your interest in contributing to Dataverse!  We are open to contributions from everyone. You don't need permission to participate. Just jump in. If you have questions, please reach out using one or more of the channels described below.
 
-We aren't just looking for developers, there are many ways to contribute to Dataverse.  We welcome contributions of ideas, bug reports, usability research/feedback, documentation, code, and more!
+We aren't just looking for developers. There are many ways to contribute to Dataverse.  We welcome contributions of ideas, bug reports, usability research/feedback, documentation, code, and more!
 
 ## Ideas/Feature Requests
 
-Your idea or feature request might already be captured in the Dataverse [issue tracker] on GitHub but if not, the best way to bring it to the community's attention is by posting on the [dataverse-community Google Group][]. You're also welcome make some noise in the [#dataverse IRC channel][] (which is [logged][]) or cram your idea into 140 characters and mention [@dataverseorg][] on Twitter. To discuss your idea privately, please email it to support@dataverse.org
+Your idea or feature request might already be captured in the Dataverse [issue tracker] on GitHub but if not, the best way to bring it to the community's attention is by posting on the [dataverse-community Google Group][] or bringing it up on a [Community Call][]. You're also welcome make some noise in the [#dataverse IRC channel][] (which is [logged][]) or cram your idea into 280 characters and mention [@dataverseorg][] on Twitter. To discuss your idea privately, please email it to support@dataverse.org
 
 There's a chance your idea is already on our roadmap, which is available at http://dataverse.org/goals-roadmap-and-releases
 
 [#dataverse IRC channel]: http://chat.dataverse.org
 [logged]: http://irclog.iq.harvard.edu/dataverse/today
 [issue tracker]: https://github.com/IQSS/dataverse/issues
 [@dataverseorg]: https://twitter.com/dataverseorg
-[Functional Requirements Document (FRD for short)]: https://docs.google.com/document/d/1PRyAlP6zlUlUuHfgyUezzuaVQ4JnapvgtGWo0o7tLEs/edit?usp=sharing
-[Balsamiq]: https://iqssharvard.mybalsamiq.com/projects
-[Functional Requirements Document folder on Google Drive]: https://drive.google.com/folderview?id=0B3_V6vFxEcx-fl92ek92OG1nTmhQenBRX1Z4OVJBLXpURmh2d2RyX1NZRUp6YktaYUU5YTA&usp=sharing
 
 ## Usability testing
 
@@ -26,15 +23,15 @@ Please email us at support@dataverse.org if you are interested in participating
 
 An issue is a bug (a feature is no longer behaving the way it should) or a feature (something new to Dataverse that helps users complete tasks). You can browse the Dataverse [issue tracker] on GitHub by open or closed issues or by milestones.
 
-Before submitting an issue, please search the existing issues by using the search bar at the top of the page. If there is an existing issue that matches the issue you want to report, please add a comment to it.
+Before submitting an issue, please search the existing issues by using the search bar at the top of the page. If there is an existing open issue that matches the issue you want to report, please add a comment to it.
 
-If there is no pre-existing issue, please click on the "New Issue" button, log in, and write in what the issue is (unless it is a security issue which should be reported privately to security@dataverse.org).
+If there is no pre-existing issue or it has been closed, please click on the "New Issue" button, log in, and write in what the issue is (unless it is a security issue which should be reported privately to security@dataverse.org).
 
 If you do not receive a reply to your new issue or comment in a timely manner, please email support@dataverse.org with a link to the issue.
 
 ### Writing an Issue
 
-For the subject of an issue, please start it by writing the feature or functionality it relates to, i.e. "Create Account:..." or "Dataset Page:...". In the body of the issue, please outline the issue you are reporting with as much detail as possible. In order for the Dataverse development team to best respond to the issue, we need as much information about the issue as you can provide. Include steps to reproduce bugs. Indicate which version you're using. We love screenshots!
+For the subject of an issue, please start it by writing the feature or functionality it relates to, i.e. "Create Account:..." or "Dataset Page:...". In the body of the issue, please outline the issue you are reporting with as much detail as possible. In order for the Dataverse development team to best respond to the issue, we need as much information about the issue as you can provide. Include steps to reproduce bugs. Indicate which version you're using, which is shown at the bottom of the page. We love screenshots!
 
 ### Issue Attachments
 
@@ -51,13 +48,20 @@ The source for the documentation at http://guides.dataverse.org/en/latest/ is in
 
 ## Code/Pull Requests
 
-Before you start coding, please reach out to us either on the [dataverse-community Google Group][], the [dataverse-dev Google Group][], [IRC][] (#dataverse on freenode), or via support@dataverse.org to make sure the effort is well coordinated and we avoid merge conflicts.
+We love code contributions. Developers are not limited to the main Dataverse code in this git repo. You can help with API client libraries in your favorite language that are mentioned in the [API Guide][] or create a new library. You can help work on configuration management code that's mentioned in the [Installation Guide][]. The Installation Guide also covers a new concept called "external tools" that allows developers to create their own tools that are available from within an installation of Dataverse.
 
-Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request.
+[API Guide]: http://guides.dataverse.org/en/latest/api
+[Installation Guide]: http://guides.dataverse.org/en/latest/installation
 
-After making your pull request, your goal should be to help it advance through our kanban board at https://waffle.io/IQSS/dataverse . If no one has moved your pull request to the code review column in a timely manner, please reach out. We maintain a list of [community contributors][] so please let us know if you'd like to be added or removed from the list. Thanks!
+If you are interested in working on the main Dataverse code, great! Before you start coding, please reach out to us either on the [dataverse-community Google Group][], the [dataverse-dev Google Group][], [IRC][] (#dataverse on freenode), or via support@dataverse.org to make sure the effort is well coordinated and we avoid merge conflicts. We maintain a list of [community contributors][] and [dev efforts][] the community is working on so please let us know if you'd like to be added or removed from either list.
+
+Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request. That page also includes tips on making a pull request.
+
+After making your pull request, your goal should be to help it advance through our kanban board at https://waffle.io/IQSS/dataverse . If no one has moved your pull request to the code review column in a timely manner, please reach out. Thanks!
 
 [dataverse-community Google Group]: https://groups.google.com/group/dataverse-community
+[Community Call]: https://dataverse.org/community-calls
 [dataverse-dev Google Group]: https://groups.google.com/group/dataverse-dev
 [IRC]: http://chat.dataverse.org
 [community contributors]: https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing
+[dev efforts]: https://groups.google.com/d/msg/dataverse-community/X2diSWYll0w/ikp1TGcfBgAJ

doc/sphinx-guides/source/_static/installation/files/root/external-tools/awesomeTool.json

@@ -1,6 +1,7 @@
 {
   "displayName": "Awesome Tool",
   "description": "The most awesome tool.",
+  "type": "explore",
   "toolUrl": "https://awesometool.com",
   "toolParameters": {
     "queryParameters": [

doc/sphinx-guides/source/_static/installation/files/root/external-tools/dataExplorer.json

@@ -0,0 +1,19 @@
+{
+  "displayName": "Data Explorer",
+  "description": "The Data Explorer provides a GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis.",
+  "type": "explore",
+  "toolUrl": "https://scholarsportal.github.io/Dataverse-Data-Explorer/",
+  "toolParameters": {
+    "queryParameters": [
+      {
+        "fileId": "{fileId}"
+      },
+      {
+        "siteUrl": "{siteUrl}"
+      },
+      {
+        "key": "{apiToken}"
+      }
+    ]
+  }
+}

doc/sphinx-guides/source/_static/installation/files/root/external-tools/twoRavens.json

@@ -0,0 +1,16 @@
+{
+  "displayName": "TwoRavens",
+  "description": "A system of interlocking statistical tools for data exploration, analysis, and meta-analysis.",
+  "type": "explore",
+  "toolUrl": "https://tworavens.dataverse.example.edu/dataexplore/gui.html",
+  "toolParameters": {
+    "queryParameters": [
+      {
+        "dfId": "{fileId}"
+      },
+      {
+        "key": "{apiToken}"
+      }
+    ]
+  }
+}

doc/sphinx-guides/source/developers/intro.rst

@@ -12,20 +12,21 @@ Intended Audience
 
 This guide is intended primarily for developers who want to work on the main Dataverse code base at https://github.com/IQSS/dataverse
 
-To get started, you'll want to set up your :doc:`dev-environment` and make sure you understand the branching strategy described in the :doc:`version-control` section. :doc:`testing` is expected. Opinions about :doc:`coding-style` are welcome!
+(See "Related Projects" below for other code you can work on!)
+
+To get started, you'll want to set up your :doc:`dev-environment` and make sure you understand the branching strategy described in the :doc:`version-control` section and how to make a pull request. :doc:`testing` is expected. Opinions about :doc:`coding-style` are welcome!
 
 If you have any questions at all, please reach out to other developers per https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md
 
 Roadmap
 -------
 
-For the Dataverse development roadmap, please see https://github.com/IQSS/dataverse/milestones
+For the Dataverse development roadmap, please see https://dataverse.org/goals-roadmap-and-releases
 
-The `Contributing to Dataverse <https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md>`_ document in the root of the source tree provides guidance on:
+Kanban Board
+------------
 
-- the use of `labels <https://github.com/IQSS/dataverse/labels>`_ to organize and prioritize `issues <https://github.com/IQSS/dataverse/issues>`_ 
-- making pull requests
-- how to contact the development team
+You can get a sense of what's currently in flight (in dev, in QA, etc.) by looking at https://waffle.io/IQSS/dataverse
 
 Related Guides
 --------------
@@ -39,13 +40,15 @@ Related Projects
 
 As a developer, you also may be interested in these projects related to Dataverse:
 
+- External Tools - add additional features to Dataverse: See the :doc:`/installation/external-tools` section of the Installation Guide.
+- Dataverse API client libraries - use Dataverse APIs from various languages: :doc:`/api/client-libraries`
 - Miniverse - expose metrics from a Dataverse database: https://github.com/IQSS/miniverse
-- `Zelig <http://zeligproject.org>`_ (R) - run statistical models on files uploaded to Dataverse: https://github.com/IQSS/Zelig
-- `TwoRavens <http://2ra.vn>`_ (Javascript) - a `d3.js <http://d3js.org>`_ interface for exploring data and running Zelig models: https://github.com/IQSS/TwoRavens
+- Configuration management scripts - Ansible, Puppet, etc.: See "Advanced Installation" in the :doc:`/installation/prep` section of the Installation Guide.
 - :doc:`/developers/unf/index` (Java) -  a Universal Numerical Fingerprint: https://github.com/IQSS/UNF
-- `DataTags <https://github.com/IQSS/DataTags>`_ (Java and Scala) - tag datasets with privacy levels: https://github.com/IQSS/DataTags
 - GeoConnect (Python) - create a map by uploading files to Dataverse: https://github.com/IQSS/geoconnect
-- Dataverse API client libraries - use Dataverse APIs from various languages: :doc:`/api/client-libraries`
+- `DataTags <https://github.com/IQSS/DataTags>`_ (Java and Scala) - tag datasets with privacy levels: https://github.com/IQSS/DataTags
+- `TwoRavens <http://2ra.vn>`_ (Javascript) - a `d3.js <http://d3js.org>`_ interface for exploring data and running Zelig models: https://github.com/IQSS/TwoRavens
+- `Zelig <http://zeligproject.org>`_ (R) - run statistical models on files uploaded to Dataverse: https://github.com/IQSS/Zelig
 - Third party apps - make use of Dataverse APIs: :doc:`/api/apps`
 - chat.dataverse.org - chat interface for Dataverse users and developers: https://github.com/IQSS/chat.dataverse.org
 - [Your project here] :)

doc/sphinx-guides/source/installation/config.rst

@@ -934,14 +934,12 @@ The relative path URL to which users will be sent after signup. The default sett
 :TwoRavensUrl
 +++++++++++++
 
-The location of your TwoRavens installation.  Activation of TwoRavens also requires the setting below, ``TwoRavensTabularView``
+The ``:TwoRavensUrl`` option is no longer valid. See :doc:`r-rapache-tworavens` and :doc:`external-tools`.
 
 :TwoRavensTabularView
 +++++++++++++++++++++
 
-Set ``TwoRavensTabularView`` to true to allow a user to view tabular files via the TwoRavens application. This boolean affects whether a user will see the "Explore" button.
-
-``curl -X PUT -d true http://localhost:8080/api/admin/settings/:TwoRavensTabularView``
+The ``:TwoRavensTabularView`` option is no longer valid. See :doc:`r-rapache-tworavens` and :doc:`external-tools`.
 
 :GeoconnectCreateEditMaps
 +++++++++++++++++++++++++

doc/sphinx-guides/source/installation/external-tools.rst

@@ -1,22 +1,36 @@
 External Tools
 ==============
 
-External tools can provide additional features that are not part of Dataverse itself. In order to make external tools available within Dataverse, you need to configure Dataverse to be aware of them. At this point, our support for external tools is in an experimental stage, and is still in the process of being finalized.
+External tools can provide additional features that are not part of Dataverse itself, such as data exploration. See the "Writing Your Own External Tool" section below for more information on developing your own tool for Dataverse.
 
 .. contents:: |toctitle|
   :local:
 
+Inventory of External Tools
+---------------------------
+
+Support for external tools is just getting off the ground but TwoRavens has been converted into an external tool. See the :doc:`/user/data-exploration/tworavens` section of the User Guide for more information on TwoRavens from the user perspective and :doc:`r-rapache-tworavens` for more information on installing TwoRavens.
+
+- TwoRavens: a system of interlocking statistical tools for data exploration, analysis, and meta-analysis: http://2ra.vn
+- Data Explorer (planned): a GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. Keep an eye on https://github.com/IQSS/dataverse/issues/4249 to know when Data Explorer can be used as an external tool.
+- [Your tool here! Please get in touch! :) ]
+
 Downloading and Adjusting an External Tool Manifest File
 --------------------------------------------------------
 
+In order to make external tools available within Dataverse, you need to configure Dataverse to be aware of them.
+
 External tools must be expressed in an external tool manifest file, a specific JSON format Dataverse requires. The author of the external tool may be able to provide you with a JSON file and installation instructions.  The JSON file might look like this:
 
 .. literalinclude:: ../_static/installation/files/root/external-tools/awesomeTool.json
 
+``type`` is required and must be ``explore`` or ``configure`` to make the tool appear under a button called "Explore" or "Configure", respectively. Currently external tools only operate on tabular files that have been successfully ingested. (For more on ingest, see the :doc:`/user/tabulardataingest/ingestprocess` of the User Guide.)
+
 In the example above, a mix of required and optional reserved words appear that can be used to insert dynamic values into tools. The supported values are:
 
 - ``{fileId}`` (required) - The Dataverse database ID of a file the external tool has been launched on.
-- ``{apiToken}`` (optional) - The Dataverse API token of the user launching the external tool.
+- ``{siteUrl}`` (optional) - The URL of the Dataverse installation that hosts the file with the fileId above.
+- ``{apiToken}`` (optional) - The Dataverse API token of the user launching the external tool, if available.
 
 Making an External Tool Available in Dataverse
 ----------------------------------------------
@@ -45,3 +59,5 @@ Writing Your Own External Tool
 If you have an idea for an external tool, please let the Dataverse community know by posting about it on the dataverse-community mailing list: https://groups.google.com/forum/#!forum/dataverse-community
 
 If you need help with your tool, please feel free to post on the dataverse-dev mailing list: https://groups.google.com/forum/#!forum/dataverse-dev
+
+Once you've gotten your tool working, please make a pull request to update the list of tools above.

doc/sphinx-guides/source/installation/installation-main.rst

@@ -52,15 +52,13 @@ The script will prompt you for some configuration values. If this is a test/eval
 - Name of the Postgres User: dvnapp
 - Postgres user password: secret
 - Remote Solr indexing service: LOCAL
-- Will this Dataverse be using TwoRavens application: NOT INSTALLED
 - Rserve Server: localhost
 - Rserve Server Port: 6311
 - Rserve User Name: rserve
 - Rserve User Password: rserve
 - Administration Email address for the installation;
 - Postgres admin password - We'll need it in order to create the database and user for the Dataverse to use, without having to run the installer as root. If you don't know your Postgres admin password, you may simply set the authorization level for localhost to "trust" in the PostgreSQL ``pg_hba.conf`` file (See the PostgreSQL section in the Prerequisites). If this is a production evnironment, you may want to change it back to something more secure, such as "password" or "md5", after the installation is complete.
 - Network address of a remote Solr search engine service (if needed) - In most cases, you will be running your Solr server on the same host as the Dataverse application (then you will want to leave this set to the default value of ``LOCAL``). But in a serious production environment you may set it up on a dedicated separate server.
-- The URL of the TwoRavens application GUI, if this Dataverse node will be using a companion TwoRavens installation. Otherwise, leave it set to ``NOT INSTALLED``. 
 
 If desired, these default values can be configured by creating a ``default.config`` (example :download:`here <../_static/util/default.config>`) file in the installer's working directory with new values (if this file isn't present, the above defaults will be used).