You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Evaluating a new open-source, standards-based framework for web portal development in the geosciences
Principal Investigator
• Signell, Richard P. (USGS, Coastal and Marine Program, Woods Hole, MA USA)
Collaborators
• Ring, Kevin (CSIRO, Data61, Australia)
• Walker, Jordan I. (USGS, Office of Water Information, Middleton, WI)
• Dalyander, Soupy (USGS, Coastal and Marine Program, St. Petersburg, FL, USA)
Purpose and Objectives
Web portals are one of the principle ways geospatial information is communicated to the public. A few prominent USGS examples are the Geo Data Portal, EarthExplorer, the Derived Downscaled Climate Projection Portal, the Alaska Portal Map, the Coastal Hazards Change Portal, The National Map. Currently web portals are developed at relatively high effort and cost, with web developers working with highly skilled data specialists on custom solutions that meet user needs. To address this issue, the Australian National Government funded the development of an opensource framework for building web portals called TerriaJS, which began in early 2015 (see presentation to the USGS CDI Tech Stack Working Group ). TerriaJS takes advantages of capabilities in modern browsers to deliver a browser-only solution that consumes web map services from both ESRI and OGC, the most commonly-used web map service standards employed at the USGS and throughout the geoscience community. Because TerriaJS runs completely within the web browser, it is possible to generate custom portals using simple configuration files that can be located anywhere on the internet. This means that basic portals based on webmap services can be created by non-javascript developers such as scientists, environmental managers and emergency response support personnel. It also means they can be constructed rapidly, in hours or days instead of weeks or months. Finally, TerriaJS should also reduce the development cost of more sophisticated portals by providing a broad framework that covers a wide range of common portal mapping needs.
While TerriaJS appeared promising, to understand its value we needed to examine more closely the current capabilities, potential and risks:
What are the generic portal needs that can be addressed by the existing framework? Does the framework architecture allow expansion to address more sophisticated needs beyond basic web service mapping?
Does the framework have sufficient documentation and interaction/encouragement from the lead developers to actually function as a community-driven open source project, or are enhancements only reasonably created by core developers?
We addressed these questions by taking a deep dive into TerriaJS by adding specific enhancements needed to support access to meteorological, oceanographic and hydrologic model data, and using the framework to create several web portals using a combination of developer resources from the USGS Office of Water Information and the Australian CSIRO/Data61 team.
Accomplishments
• We achieved the main objective of assessing the role of TerriaJS in the USGS landscape of available tools for creating web services. Using a combination of USGS and Data61 developer resources, we made critical enhancements for dealing with met, ocean and hydrology model data, tested deployment in the USGS computational environment, and developed several demos using TerriaJS that allowed us to assess performance and ease of installation and use.
• The code enhancements enabled by the CDI project were been merged back into the master branch on GitHub [1]. These enhancements were controls that allow the user to select layers, change color ranges and select styles from ncWMS and ncWMS2 services integrated into THREDDS Data Servers. This allows effective exploration of model data via WMS in TerriaJS, and thanks to this project, this capability is now available not just to USGS, but to everyone.
• A configuration was set up to explore multiple time dependent datasets from geospatial standards, purported strengths of TerriaJS. We created a bird migration example, using CZML (Cesium Markup Language, JSON that allows representation of geometric information in space and time) to display the bird migration data, and using WMS from a NOAA THREDDS server to display underlying monthly temperature data [2]. This example serves to show the power and flexibility of TerriaJS in handling time dependent data, as well as ease of configuration by end users.
• The USGS system for automatic deployment of services controlled by continuous integration on a code repository was tested. The system performed as advertised, with code deployed on a “staging” branch on bitbucket [3], automatically deploying an instance running on USGS infrastructure [4].
Lessons Learned
• Developing enhancements for TerriaJS was indeed challenging. USGS developer Jordan Walker found the documentation scarce, the code complex, and even building the software tricky due to numerous changing dependencies. Part of this was due to unfortunate timing, as TerriaJS switched to a new user interface just after our CDI project was initiated. Most enhancement therefore ended up being developed by Kevin Ring, the lead developer for TerriaJS. Hopefully this situation will improve as documentation and experience with the new UI grows. The good news is that the new UI plays nicely on mobile, essential for uptake by the USGS community.
• When enhancements were made to support the ncWMS and ncWMS2 services, we discovered that displaying information from unstructured grid (e.g. triangular grid) models in TerriaJS was extremely slow, and significantly slower than the Godiva3 ncWMS2 client. We discovered this is due to a combination of factors: (1) slow delivery in general by the Java-based ncWMS2 service, (2) requesting tiles in projected coordinates rather than the native geographic coordinates, and (3) requesting many more tiles than Godiva3 for the same spatial extent. This has triggered new, ongoing work to understand how to make delivery of information from unstructured grid models more efficient.
• Although automatic deployment to the USGS infrastructure was successful, the TerriaJS endpoint was crippled because of a USGS requirement that only data services accessible via HTTPS could be displayed. Our use cases, designed to show integration of data from heterogenous data services from the broad geospatial community, therefore did not work, as few external data providers supported HTTPS. An additional weakness of the USGS TerriaJS endpoint is that it is only accessible via the internal USGS network. To overcome these shortcomings, we set up a publicly-accessible TerriaJS endpoint [5] on a USGS/WHOI cooperative computer that doesn’t have the HTTPS restrictions.
• We had hoped to bring in new USGS data from Soupy Dalyander to demonstrate integration of in the TerriaJS platform, but security concerns with Docker prevented deployment of the THREDDS Data Server Docker container. We hope these are addressed by USGS in the future, as Docker makes deploying and maintaining services much easier, isolates server processes from each other, making services on specific machines more robust. We hope to test deployment of THREDDS Docker container on the USGS Cloud Hosting Solution in the near future.
Figure 1. Snapshot of surface currents from the IOOS forecast model for New England, which uses a native triangular grid, but is regridded at varying resolution by ncWMS2. Shown also are the user controls for selecting the layer, the elevation and the color range. The ability to access ncWMS2 endpoints, as well as control the color range, elevation and style were all CDI Enhancements to TerriaJS. These enhancements have been merged into the master branch of TerriaJS [1] so the entire community can benefit.
Figure 2. Snapshot of bird migration from CZML data source superimposed on monthly temperature data from WMS (via a NOAA THREDDS server). Shown also are the user controls added in this project for selecting the style and color range. Note that both the CZML file and the JSON configuration for this demo are on GitHub [6], and referenced directly in the URL. This means that users can create custom portals on their own, without any interaction from the provider of the TerriaJS endpoint, which just serves to deliver the TerriaJS code to the browser. TerriaJS runs completely within the browser.
Evaluating a new open-source, standards-based framework for web portal development in the geosciences
Principal Investigator
• Signell, Richard P. (USGS, Coastal and Marine Program, Woods Hole, MA USA)
Collaborators
• Ring, Kevin (CSIRO, Data61, Australia)
• Walker, Jordan I. (USGS, Office of Water Information, Middleton, WI)
• Dalyander, Soupy (USGS, Coastal and Marine Program, St. Petersburg, FL, USA)
Purpose and Objectives
Web portals are one of the principle ways geospatial information is communicated to the public. A few prominent USGS examples are the Geo Data Portal, EarthExplorer, the Derived Downscaled Climate Projection Portal, the Alaska Portal Map, the Coastal Hazards Change Portal, The National Map. Currently web portals are developed at relatively high effort and cost, with web developers working with highly skilled data specialists on custom solutions that meet user needs. To address this issue, the Australian National Government funded the development of an opensource framework for building web portals called TerriaJS, which began in early 2015 (see presentation to the USGS CDI Tech Stack Working Group ). TerriaJS takes advantages of capabilities in modern browsers to deliver a browser-only solution that consumes web map services from both ESRI and OGC, the most commonly-used web map service standards employed at the USGS and throughout the geoscience community. Because TerriaJS runs completely within the web browser, it is possible to generate custom portals using simple configuration files that can be located anywhere on the internet. This means that basic portals based on webmap services can be created by non-javascript developers such as scientists, environmental managers and emergency response support personnel. It also means they can be constructed rapidly, in hours or days instead of weeks or months. Finally, TerriaJS should also reduce the development cost of more sophisticated portals by providing a broad framework that covers a wide range of common portal mapping needs.
While TerriaJS appeared promising, to understand its value we needed to examine more closely the current capabilities, potential and risks:
What are the generic portal needs that can be addressed by the existing framework? Does the framework architecture allow expansion to address more sophisticated needs beyond basic web service mapping?
Does the framework have sufficient documentation and interaction/encouragement from the lead developers to actually function as a community-driven open source project, or are enhancements only reasonably created by core developers?
We addressed these questions by taking a deep dive into TerriaJS by adding specific enhancements needed to support access to meteorological, oceanographic and hydrologic model data, and using the framework to create several web portals using a combination of developer resources from the USGS Office of Water Information and the Australian CSIRO/Data61 team.
Accomplishments
• We achieved the main objective of assessing the role of TerriaJS in the USGS landscape of available tools for creating web services. Using a combination of USGS and Data61 developer resources, we made critical enhancements for dealing with met, ocean and hydrology model data, tested deployment in the USGS computational environment, and developed several demos using TerriaJS that allowed us to assess performance and ease of installation and use.
• The code enhancements enabled by the CDI project were been merged back into the master branch on GitHub [1]. These enhancements were controls that allow the user to select layers, change color ranges and select styles from ncWMS and ncWMS2 services integrated into THREDDS Data Servers. This allows effective exploration of model data via WMS in TerriaJS, and thanks to this project, this capability is now available not just to USGS, but to everyone.
• A configuration was set up to explore multiple time dependent datasets from geospatial standards, purported strengths of TerriaJS. We created a bird migration example, using CZML (Cesium Markup Language, JSON that allows representation of geometric information in space and time) to display the bird migration data, and using WMS from a NOAA THREDDS server to display underlying monthly temperature data [2]. This example serves to show the power and flexibility of TerriaJS in handling time dependent data, as well as ease of configuration by end users.
• The USGS system for automatic deployment of services controlled by continuous integration on a code repository was tested. The system performed as advertised, with code deployed on a “staging” branch on bitbucket [3], automatically deploying an instance running on USGS infrastructure [4].
Lessons Learned
• Developing enhancements for TerriaJS was indeed challenging. USGS developer Jordan Walker found the documentation scarce, the code complex, and even building the software tricky due to numerous changing dependencies. Part of this was due to unfortunate timing, as TerriaJS switched to a new user interface just after our CDI project was initiated. Most enhancement therefore ended up being developed by Kevin Ring, the lead developer for TerriaJS. Hopefully this situation will improve as documentation and experience with the new UI grows. The good news is that the new UI plays nicely on mobile, essential for uptake by the USGS community.
• When enhancements were made to support the ncWMS and ncWMS2 services, we discovered that displaying information from unstructured grid (e.g. triangular grid) models in TerriaJS was extremely slow, and significantly slower than the Godiva3 ncWMS2 client. We discovered this is due to a combination of factors: (1) slow delivery in general by the Java-based ncWMS2 service, (2) requesting tiles in projected coordinates rather than the native geographic coordinates, and (3) requesting many more tiles than Godiva3 for the same spatial extent. This has triggered new, ongoing work to understand how to make delivery of information from unstructured grid models more efficient.
• Although automatic deployment to the USGS infrastructure was successful, the TerriaJS endpoint was crippled because of a USGS requirement that only data services accessible via HTTPS could be displayed. Our use cases, designed to show integration of data from heterogenous data services from the broad geospatial community, therefore did not work, as few external data providers supported HTTPS. An additional weakness of the USGS TerriaJS endpoint is that it is only accessible via the internal USGS network. To overcome these shortcomings, we set up a publicly-accessible TerriaJS endpoint [5] on a USGS/WHOI cooperative computer that doesn’t have the HTTPS restrictions.
• We had hoped to bring in new USGS data from Soupy Dalyander to demonstrate integration of in the TerriaJS platform, but security concerns with Docker prevented deployment of the THREDDS Data Server Docker container. We hope these are addressed by USGS in the future, as Docker makes deploying and maintaining services much easier, isolates server processes from each other, making services on specific machines more robust. We hope to test deployment of THREDDS Docker container on the USGS Cloud Hosting Solution in the near future.
Figure 1. Snapshot of surface currents from the IOOS forecast model for New England, which uses a native triangular grid, but is regridded at varying resolution by ncWMS2. Shown also are the user controls for selecting the layer, the elevation and the color range. The ability to access ncWMS2 endpoints, as well as control the color range, elevation and style were all CDI Enhancements to TerriaJS. These enhancements have been merged into the master branch of TerriaJS [1] so the entire community can benefit.
Figure 2. Snapshot of bird migration from CZML data source superimposed on monthly temperature data from WMS (via a NOAA THREDDS server). Shown also are the user controls added in this project for selecting the style and color range. Note that both the CZML file and the JSON configuration for this demo are on GitHub [6], and referenced directly in the URL. This means that users can create custom portals on their own, without any interaction from the provider of the TerriaJS endpoint, which just serves to deliver the TerriaJS code to the browser. TerriaJS runs completely within the browser.
References
[1] https://github.com/TerriaJS/terriajs
[2] http://gamone.whoi.edu/terriajs/#https://cdn.rawgit.com/USGS-CMG/terriajs-dive/master/examples/bird_migration.json (http://tinyurl.com/terriajs-birds)
[3] https://my.usgs.gov/bitbucket/projects/CDI/repos/terriamap/browse?at=refs/heads/staging
[4] https://maps-staging.snafu.cr.usgs.gov/TerriaMap/CDI-Dive/
[5] http://gamone.whoi.edu/terriajs
[6] https://github.com/USGS-CMG/terriajs-dive
The text was updated successfully, but these errors were encountered: