Identify synergies with birdhouse project

[huard]

Hi all,
I'm involved in the birdhouse project whose objective is to offer climate services through OGC's Web Processing Services (WPS) standard. The philosophy, as with Pangeo, is to process data remotely rather than having to download it. One major difference is that we're not trying to provide a full programming environment, but only a library of key analytical processes that are often used. Scientists would run these processes remotely, get the output then use whatever environment they're familiar with to complete the analysis. We're slowly building an infrastructure where these processes can be chained in workflows.

I think it would make sense for us to use the tools developed in Pangeo in our computational backend. We're using OCGIS, which as I understand could eventually support xarray.

So basically I just wanted to say hi and let you know of our interest for your work.

[tomLandry]

Hello everyone. I work at Computer research institute of Montreal (CRIM) and I'm a close collaborator to @huard at Ouranos. My institution isn't hosting any climate scientist, but we do have in house several data scientists, remote sensing specialists, web developers, Big Data specialist, engineers, researchers and so on. We are also active in the birdhouse project, trying to expand its Earth observations capabilities, amongst other things.

I therefore reiterate David's notice of interest. Oh... and hi!

[jacobtomlinson]

Hi both of you! Welcome aboard!

I would be keen to know more. You mention that your project is only a library but then you mention you are building an infrastructure. Where does the library fit into the infrastructure?

[tomLandry]

That is an excellent question @jacobtomlinson . There's a long answer that would not fit very well here. I'll follow up on this. A short answer will be published and presented in FOSS4G Tanzania 2018.

[rabernat]

Welcome! This sounds like a great area for collaboration. So far we have mostly been focused on the interactive jupyter experience, but the same basic infrastructure should be able to form the basis of scalable services and APIs such as the one you described. WPS targets the GIS community, and we are still working on trying to define the relationship of Pangeo to the GIS world (see e.g. #216, #66) Most of us come from a different background and are not super familiar with GIS technologies.

[huard]

@jacobtomlinson I'll still try to answer it ; )
What I mean is that users only see the library (not in the .lib sense, but the collection sense) of processes that we provide. On the backend however, there is an architecture built to serve these processes, load balance them, deal with access permissions to datasets and services, etc.

[jacobtomlinson]

Ah ok, so it's more like an API which provides an OGC WPS service. And you have a library which allows you to use the service in your code?

[huard]

There is no other API than WPS at the moment, but you may call WPS processes programatically using owslib (if that's your question). One idea floating around is to write a client generator, that would parse the WPS server getCapabilities and describeProcess output to dynamically generate a python client, so you could call remote services from the console without knowing anything about WPS.

But all this is just on the user side, on the climate computation backend, we're mostly using ocgis, and our DKRZ colleagues have wrapped some CDO functionalities into WPS processes. What we'd like to try is to write cloud-optimized processes using Pangeo.

[jacobtomlinson]

Yes that is what I meant. That could be a good idea, similar to the way swagger works.

My concern with WPS systems is that they are great if you know what processes what you are going to do with it, so it makes a lot of sense in an operational environment when you are producing routine data products. But if you want to do something new it needs to be implemented in the WPS before it can do it, which makes it tricky to use in a research environment.

But maybe Pangeo could help make these things more flexible!

[tomLandry]

For a more thorough view on Birdhouse, I post here a poster made by Carsten and al. from DKRZ. So this is part of the response you where looking for wrt librairies vs infrastructure, at the EU level. We also have a plan solidly in place for Canada infrastructure, but not public - yet.
https://drive.google.com/file/d/1hCYd6TIad_AtigemqGlumilkcn1Zo9vH/view?usp=sharing

[huard]

Exactly, I've heard someone is working on WPS support for swagger, don't know the schedule for this though.

Totally agree. That's why the client makes sense. Provides you with backed-in functionality in an environment that let's you experiment.

Once things cool down on our end, we'll give Pangeo a spin and come back with more concrete ideas for collaboration.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tomLandry]

I say we keep the issue open. We'll coordinate (remote) participation for the Pangeo developer meeting.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.