diff --git a/docs/source/arch/data_catalog.rst b/docs/source/arch/data_catalog.rst deleted file mode 100644 index 34a95e6f..00000000 --- a/docs/source/arch/data_catalog.rst +++ /dev/null @@ -1,24 +0,0 @@ -Pavics-DataCatalog -================== - -PAVICS-SDI bundles a number of cataloguing services that allow users to search for and add data to its backend catalog service. - -The main cataloguing component within the PAVICS-SDI architecture is `PAVICS-DataCatalog `_, a service that identifies and enables querying of `CF-compliant `_ climate data organized within THREDDS Data Servers. PAVICS-DataCatalog is built with the `Apache Solr `_ search platform. - -PAVICS Catalogue Search ------------------------ -The ``pavicssearch`` service closely mimics the API for the Earth System Grid Federation (ESGF) search. The search fields (or *facets*) include author, category, cf_standard_name, experiment, frequency, institute, model, project, source, subject, title, units, variable and variable_long_name. The search results are returned in a ``json`` file and include the URL to download the file or access it through DAP. - -Examples --------- -The following request will launch the crawler on the server filesystem to catalog available data:: - - http://localhost:8009/pywps?service=WPS&request=execute&version=1.0.0&identifier=pavicrawler&storeExecuteResponse=true&status=true&DataInputs= - -The following request will search for all files that are part of the RCP8.5 experiment and based on the CRCM4 model:: - - http://localhost:8009/pywps?service=WPS&request=execute&version=1.0.0&identifier=pavicsearch&DataInputs=constraints=model:CRCM4,experiment:rcp85 - -PAVICS Catalogue Credits ------------------------- -PAVICS-DataCatalog is developed by researchers at CRIM and Ouranos. diff --git a/docs/source/arch/frontend.rst b/docs/source/arch/frontend.rst deleted file mode 100644 index aef2bbf9..00000000 --- a/docs/source/arch/frontend.rst +++ /dev/null @@ -1,30 +0,0 @@ -======== -Frontend -======== - -.. image:: images/PAVICS_data_overlay.png - :alt: An example of the PAVICS-frontend in action - -The PAVICS-frontend (`GitHub Repository `_) is the custom graphical interface for users to leverage the services offered by the platform. It provides a map interface to display both gridded climate data and geospatial layers, launch services and workflows and monitor the progress and status of user's jobs. It also provides limited capacity to dynamically interact with datasets. Users can create projects that store datasets, workflows and outputs for future use. - -The frontend is written with the React.js library and interacts with the backend through API calls to REST and WPS services. - -Gridded data rendering ----------------------- - -A core component of the frontend is the gridded data renderer. Gridded netCDF datasets selected for visualization are displayed on the base map using `ncWMS`_, a server capable of converting multidimensional netCDF data into images. The frontend sends ``GetMap`` requests to the ncWMS server according to the display's bounding box and zoom level. These requests are handled by OpenLayer and its plugins. - -The conversion from the raw netCDF data to an image requires a mapping between data values and a colorbar. The color scale can be selected from a menu and its min and max values modified by the user. The color scale information is sent to ncWMS to render the data into an image. - -The colorbar palette is displayed as an image, provided as the response to a ``GetLegendGraphic`` request to ncWMS. - -For high resolution data, rendering suffers from lag that can make browsing netCDF files a frustrating experience. We are investigating options to reduce delivery latency. - -Take a look at the frontend tutorial for information on the frontend's usage! - -.. todo:: - - Describe the relationship between the frontend and Phoenix. - - -.. _ncWMS: https://reading-escience-centre.github.io/ncwms/ diff --git a/docs/source/arch/index.rst b/docs/source/arch/index.rst index e6e531e8..1901b4e4 100644 --- a/docs/source/arch/index.rst +++ b/docs/source/arch/index.rst @@ -9,5 +9,3 @@ System Architecture overview backend jupyter - frontend - data_catalog diff --git a/docs/source/arch/overview.rst b/docs/source/arch/overview.rst index 3fc93fa2..50ad1366 100644 --- a/docs/source/arch/overview.rst +++ b/docs/source/arch/overview.rst @@ -28,15 +28,12 @@ Raven JupyterLab A notebook interface to demonstrate how WPS services can be used from a programming environment. -PAVICS-frontend - The user interface (UI) handling user accounts, workspace, workflows and data visualization. Development of the UI has paused as it consumed a lot of resources, consider it as a prototype. - -PAVICS-DataCatalog - Storing and serving information about available climate data. - Magpie Authentication and authorization services. +Weaver + Workflow Execution Management Service (EMS) and Application, Deployment and Execution Service (ADES) supporting legacy WPS services as well as OGC API - Processes REST bindings. + THREDDS netCDF data server. @@ -44,9 +41,10 @@ GeoServer Geospatial data server. + These components work together to offer users a seamless access to data and a suite of services that can convert raw climate data into useful information, graphics and tables. Credits ------- -PAVICS is led by `Ouranos `_, a regional climatology research consortium, and `CRIM `_, an informatics and software research institute, (both located in Montreal, Quebec, Canada) to provide climate scientists with a set of tools to acquire and analyze climate data. The project was funded by the CANARIE research software program. +PAVICS is led by `Ouranos `_, a regional climatology research consortium, and `CRIM `_, an informatics and software research institute, (both located in Montreal, Quebec, Canada) to provide climate scientists with a set of tools to acquire and analyze climate data. The project was initially funded by the CANARIE research software program, and has since benefited from contributions from the Open Geospatial Consortium, the Québec Ministry of Environment and Fight Against Climate Change, Environment and Climate Change Canada and the Canadian Foundation for Innovation. diff --git a/docs/source/conf.py b/docs/source/conf.py index 9ce8ef24..7c426e42 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -43,7 +43,6 @@ 'sphinx.ext.viewcode', 'sphinx.ext.intersphinx', 'sphinx.ext.autosectionlabel', - 'sphinx-jsonschema', 'nbsphinx', ] diff --git a/docs/source/gui/frontend.rst b/docs/source/gui/frontend.rst deleted file mode 100644 index 25235cec..00000000 --- a/docs/source/gui/frontend.rst +++ /dev/null @@ -1,59 +0,0 @@ -Using the graphical interface -============================= - -The frontend is composed of a base map, a tool menu and a dashboard. - - -The basemap works like similar online maps, with the mouse wheel allowing zoom control, while click and drag lets you move laterally on the map. At the moment, only the standard rectangular projection is supported. There are three different options for the basemap: a satellite view, and an administrative view with and without labels. Let us know which other projection and basemap would be useful for your needs. - -At the bottom of the screen is a colorbar with min and max values. Use these to set the limits on the colorbar for displayed netCDF datasets. - -Tool menu -~~~~~~~~~ -The tool menu is located at the bottom left corner and looks like a pie. It opens five panels that provide methods for interacting with the map and overlayed datsets available within the PAVICS-SDI: - -Clicked Point Information - When a netCDF dataset is displayed on the map, clicking on a grid cell will display the information stored at that point. - -A Time Series Chart - When a netCDF dataset is displayed on the map, clicking on a grid cell will load the time series at that point and display it on a graphic. - -A Data Layer Visibility Switcher - This panel lets user select which basemap, netCDF dataset and geospatial layer is displayed on the map. - -A Temporal Slider - When a netCDF dataset is displayed on the map, the time slider controls which time slice is displayed. You may pick a date then go forward or backward in time by specific increments. - -Other Map Controls - TODO - - -Each panel element can be used to view/inspect different types and display additional information of the active data. - - -.. - Example - ~~~~~~~ - - For an example of a climate analysis process using the PAVICS-frontend, see this short :download:`hands-on video `. - - -Dashboard ---------- -The right hand side dashboard contains four different sections: dataset search, project workspace, process and workflow launcher and process monitoring. - -Dataset Search - This interface is used to search for netCDF files on the PAVICS-SDI platform. It initially displays typical search categories that can be refined by loading additional search facets. Click on values for the different categories to restrict the search. Search results appear in the bottom section of the dashboard. Select the datasets of interest and add them to your workspace to visualize them or feed them into an analytical process. - - Search queries or search results can be saved for later use. TODO. - -Project Workspace - The workspace area lets you create projects in which an ensemble of files, search queries, workflows and process outputs can be stored. At the moment it is not possible for users to upload files or geospatial layers to the workspace. Let us know if this is a feature you'd like to have. - -Process and Workflows - This is the interface where computations are launched. You may launch a workflow (see :ref:`sec-workflows`) or a process (see :ref:`sec-processes`). The workflow dashboard let's you select from existing workflows that have been defined within the project or edit a new workflow from a template. Saving the workflow will trigger a validator that will warn you of syntax errors. Once the workflow has been validated, you may launch it already if there are no user defined inputs to be specified. Otherwise a form will appear to let you enter input values before launching the workflow. A notification will let you know if the workflow launched sucessfully or not. - - The process interface first asks you to identify the process provider. We realize that you probably have no idea which services are offered by which provider, and for now, we suggest you search for relevant processes in this documentation, note the package they are coming from and use this as the provider. We'll eventually *flatten* the process list and allow you to search from the list of processes. - - - diff --git a/docs/source/gui/index.rst b/docs/source/gui/index.rst deleted file mode 100644 index bf83b840..00000000 --- a/docs/source/gui/index.rst +++ /dev/null @@ -1,7 +0,0 @@ -======================== -Graphical User Interface -======================== - - - .. toctree:: - frontend \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 5fc9abbb..62c58b42 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -21,8 +21,6 @@ We plan to build extensive documentation for PAVICS. We're just getting started, * :doc:`Processes ` documents all available processes on the PAVICS platform. -* :doc:`Graphical User Interface ` walks you through the components of the web browser frontend. - .. _issue tracker: https://github.com/Ouranosinc/pavics-sdi/issues .. _birdhouse: http://bird-house.github.io/ @@ -38,9 +36,7 @@ Contents tutorials/index notebooks/index processes/index - workflows/index projects/index - gui/index dev/index arch/index provenance/index diff --git a/docs/source/provenance/index.rst b/docs/source/provenance/index.rst index 9b846e51..5831d51f 100644 --- a/docs/source/provenance/index.rst +++ b/docs/source/provenance/index.rst @@ -2,19 +2,11 @@ Provenance ========== -The code base is divided into multiple components each having its own release schedule. Each component is stored on a -individual github repository, where every change of the master branch triggers a test suite run and a build of the -corresponding docker image (available on `DockerHub`_). Development occurs in code branches, and modifications are only -merged after code reviews have been completed and all tests passed. Official releases are tagged after significant code -changes and are authorized by ?. The project documentation is entirely contained within the various sections of this -document. - -pavics-sdi is relying on a number of different packages developed by other teams. Minor pavics releases will be created following major releases of these critical third party packages if they do not coincide with a major internal release. Package whose upgrade whose trigger a minor pavics release include ocgis, flyingpigeon and malleefowl. - -Ouranos deployment ------------------- -A pre-release version is deployed on an experimental server and integration testing is performed to make sure the platform is in working order. If everything is in order, the pre-release version becomes the release version and is deployed on the pavics server. +Software +-------- +The PAVICS platform relies on a number of independent components, each with its own release schedule. Each component that Ouranos and CRIM maintain is stored in an individual github repository, where every change of the master branch triggers a test suite run. Development occurs in code branches, and modifications are only merged after code reviews have been completed and all tests passed. Official releases are tagged after significant code changes and are authorized by each library's maintainer. +These releases are then deployed by updating the version of the docker image loaded by docker-compose in `birdhouse-deploy`_. Relevant changes are described in file :file:`CHANGES.md`. Pull requests trigger a test suite, where documentation and tutorial notebooks are executed against the platform and compare results with expected outputs. .. todo:: @@ -22,5 +14,5 @@ A pre-release version is deployed on an experimental server and integration test Review by CRIM. +_`birdhouse-deploy`: https://github.com/bird-house/birdhouse-deploy -.. _DockerHub: https://hub.docker.com/ \ No newline at end of file diff --git a/docs/source/workflows/examples.rst b/docs/source/workflows/examples.rst deleted file mode 100644 index 5d15d431..00000000 --- a/docs/source/workflows/examples.rst +++ /dev/null @@ -1,41 +0,0 @@ -======== -Examples -======== - -Running sequential tasks -======================== - -This is the simplest workflow one can think of, it simply consists in running process A and using its output to feed process B. To make the examples more concrete, let's imagine a ``random_word`` process that takes as input ``n`` the number of random words to return, and returns ``text``, a string joining all these words separated by a space. Our second process ``count_characters`` will return ``n`` the number of a given character (a, b, c) in input ``text``. A workflow to generate 10 random words and count the number of *e* would then look like this: - -.. code-block:: json - - { - "name" : "count_e", - "tasks": [ - { - "name": "create_sentence", - "url": "http://myserver.org:8090/wps", - "identifier": "random_word", - "inputs": { - "n": 10, - } - - }, - { - "name": "count_letter_e", - "url": "http://myserver.org:8090/wps", - "identifier": "count_characters" - "linked_inputs": { - "text": { - "task": "create_sentence", - "output": "text", - } - }, - "inputs": { - "char": "e" - } - }, - ] - } - -When the ``count_e`` workflow is launched, the first task is executed using ``n=10``. Then the second task is executed using ``char=e``, and the ``text`` value taken from the ``text`` output of the ``create_sentence`` task, defined in ``linked_inputs`` by an :json:object:`Input_description` object. \ No newline at end of file diff --git a/docs/source/workflows/index.rst b/docs/source/workflows/index.rst deleted file mode 100644 index 94cc9676..00000000 --- a/docs/source/workflows/index.rst +++ /dev/null @@ -1,23 +0,0 @@ -.. _sec-workflows: - -=============== -Using Workflows -=============== - - -.. toctree:: - - vocabulary - examples - schema - - - - - - - - - - - diff --git a/docs/source/workflows/schema.rst b/docs/source/workflows/schema.rst deleted file mode 100644 index 42f9bd57..00000000 --- a/docs/source/workflows/schema.rst +++ /dev/null @@ -1,9 +0,0 @@ -.. _schema: - -=============== -Workflow schema -=============== - -This is the json schema describing workflows. It is defined in :file:`malleefowl/custom_workflow.py` - -.. jsonschema:: workflow_schema.json \ No newline at end of file diff --git a/docs/source/workflows/vocabulary.rst b/docs/source/workflows/vocabulary.rst deleted file mode 100644 index 887fc5e6..00000000 --- a/docs/source/workflows/vocabulary.rst +++ /dev/null @@ -1,69 +0,0 @@ -=================== -Workflow vocabulary -=================== - -Creating a climate product usually involves multiple steps: - -#. Selecting datasets -#. Subsetting the data to a specific region and period -#. Either regridding the multiple datasets to a common grid or computing spatial averages -#. Computing climate indices -#. Create graphs or tables from the results - -Typically each step would involve calling one or many individual processes, and it's convenient to combine these steps into a *workflow*. Here we use *workflow* to mean a formal description of the logical organization and ordering of individual processes. The workflow logic is encapsulated in a ``json`` file using a vocabulary (called a `schema `_) that we describe below. - -Workflows are built by combining ``Workflow_task`` into ``Group_of_task``. These groups are then executed sequentially or in parallel, as indicated in the ``Workflow`` field (see the :ref:`schema`). - - -.. json:object:: Workflow - - High-level object describing a workflow. - - :property str name: Workflow name - :property list tasks: Array of :json:object:`Workflow_task`. [optional*] - :property list parallel_groups: Array of :json:object:`Group_of_tasks` being executed on multiple processes. [optional*] - -.. note:: - - Either ``tasks`` or ``parallel_groups`` must be specified. - - -.. json:object:: Workflow_task - - Describe an individual task. - - :property str name: Unique name given to each workflow task - :property str url: Url of the WPS provider - :property str identifier: Identifier of a WPS process - :property dict inputs: Dictionary of inputs that must be fed to the WPS process. The key is the input name and the value is either the input data or an array of data if multiple values are allowed for this input. [optional] - :property dict linked_inputs: Dictionary of dynamic inputs that must be fed to the WPS process and obtained by the output of other tasks. The key is the input name (or None) and the value is an :json:object:`Input_description` dictionary or an array of them if multiple values are allowed for this input. [optional] - :property list progress_range: [optiona] Array [start, end] defining the overall progress range of this task. Default value is [0, 100]. - -.. note:: - - * Allow to plan the execution of a task after another one without feeding any output of the previous one to an input. - - -.. json:object:: Group_of_task - - :property str name: Group of task name. - :property int max_processes: Number of processes to run concurrently to process the data. - :property map: Object describing what has to be mapped inside the group or an array of data that has to be mapped directly. - :proptype map: :json:object:`Input_description` - :property reduce: Object describing what has to be reduced before leaving the group. - :proptype reduce: :json:object:`Input_description` - :property list tasks: Array of :json:object:`Workflow_task` to run concurrently inside the group. - - -.. json:object:: Input_description - - Identifies the output of a process included in the workflow that serves as an input in a downstream process. - - :property str task: Name of the task from which the input should be taken (if from a :json:object:`Group_of_task`, this is from its map element). - :property str output: Output name in the task provided above. Not required if the task has only one output or when referring to the map/reduce tasks of a group. [optional] - :property boolean as_reference: Specify if the task output should be obtained as a reference (URL) or not (data directly). False is the default value if ommited. [optional] - -.. note:: - - The workflow executor is able obviously to assign a reference output to an expected reference input and a data output to an expected data input but will also be able to read the value of a reference output to send the expected data input. However, a data output linked to an expected reference input will yield an exception. - diff --git a/docs/source/workflows/workflow_schema.json b/docs/source/workflows/workflow_schema.json deleted file mode 100644 index 00902552..00000000 --- a/docs/source/workflows/workflow_schema.json +++ /dev/null @@ -1 +0,0 @@ -{"description": "Advanced workflow schema", "minProperties": 2, "title": "Workflow", "additionalProperties": false, "definitions": {"workflow_task_schema": {"additionalProperties": false, "required": ["name", "url", "identifier"], "type": "object", "description": "Describe a WPS process task", "properties": {"inputs": {"minItems": 1, "patternProperties": {".*": {"oneOf": [{"type": "string", "description": "Data that must be fed to this input"}, {"minItems": 1, "items": {"type": "string"}, "type": "array", "description": "Array of data that must be fed to this input"}]}}, "type": "object", "description": "Dictionary of inputs that must be fed to the WPS process"}, "name": {"type": "string", "description": "Unique name given to each workflow task"}, "url": {"type": "string", "description": "Url of the WPS provider"}, "linked_inputs": {"minItems": 1, "patternProperties": {".*": {"oneOf": [{"$ref": "#/definitions/input_description_schema"}, {"minItems": 1, "items": {"$ref": "#/definitions/input_description_schema"}, "type": "array", "description": "Array of input description that must be fed to this input"}]}}, "type": "object", "description": "Dictionary of dynamic inputs that must be fed to the WPS process and obtained by the output of other tasks"}, "identifier": {"type": "string", "description": "Identifier of a WPS process"}, "progress_range": {"minItems": 2, "items": {"minimum": 0, "type": "number", "maximum": 100}, "type": "array", "description": "Progress range to map the whole progress of this task", "maxItems": 2}}}, "input_description_schema": {"additionalProperties": false, "required": ["task"], "type": "object", "description": "Description of an input source", "properties": {"output": {"type": "string", "description": "Task output name"}, "as_reference": {"type": "boolean", "description": "Specify if the task output should be obtained as a reference or not"}, "task": {"type": "string", "description": "Task name"}}}, "group_of_task_schema": {"additionalProperties": false, "required": ["name", "max_processes", "map", "reduce", "tasks"], "type": "object", "description": "Describe a group of tasks to be run concurrently", "properties": {"max_processes": {"minimum": 1, "type": "number", "description": "Number of processes to run concurrently to process the data"}, "tasks": {"minItems": 1, "items": {"$ref": "#/definitions/workflow_task_schema"}, "type": "array", "description": "Array of workflow task to run concurrently inside the group"}, "reduce": {"$ref": "#/definitions/input_description_schema"}, "name": {"type": "string", "description": "Group of task name"}, "map": {"oneOf": [{"$ref": "#/definitions/input_description_schema"}, {"minItems": 1, "items": {"type": "string"}, "type": "array", "description": "Array of data that has to be mapped directly"}]}}}}, "$schema": "http://json-schema.org/draft-04/schema#", "required": ["name"], "type": "object", "properties": {"tasks": {"minItems": 1, "items": {"$ref": "#/definitions/workflow_task_schema"}, "type": "array", "description": "Array of workflow task"}, "name": {"type": "string", "description": "Workflow name"}, "parallel_groups": {"minItems": 1, "items": {"$ref": "#/definitions/group_of_task_schema"}, "type": "array", "description": "Array of group of tasks being executed on multiple processes"}}} \ No newline at end of file diff --git a/environment-dev.yml b/environment-dev.yml index 1a660a18..7e6ad0ae 100644 --- a/environment-dev.yml +++ b/environment-dev.yml @@ -12,5 +12,4 @@ dependencies: - pip - pip: - sphinx-intl - - sphinx-jsondomain # unmaintained and has sphinx <2.0 hardcoded in dependencies - - sphinx-jsonschema + - Jinja2<=3.0.3 diff --git a/setup.py b/setup.py index 9a62dc9c..75c979d7 100644 --- a/setup.py +++ b/setup.py @@ -18,8 +18,6 @@ "sphinx>=1.4", "sphinx-intl", "sphinx_rtd_theme", - "sphinx-jsondomain", # unmaintained and has sphinx <2.0 hardcoded in dependencies - "sphinx-jsonschema", "nbsphinx", "jinja2", "jupyter",