Test OWSLib PR allowing POST for WFS getfeature requests
#297
Comments
|
If I recall correctly, I had established that there was a problem via a very large request run in a Jupyter notebook. If I can find the issue where I raised this and the notebook/environment where I established the problem existed, it'll make things much easier to formally establish that the issue is fixed. I'll have some time available to put some work into this as we near the Symposium. |
|
Thanks for your work on getting POST support into OWSlib. I'm pinging you on this thread as this problem is RAVEN-centric. I seem to not be grasping the differences between GET and POST request formatting. Here's a snippet from a request formatting function in a PR I'm currently working towards: Does the `WebFeatureService().getfeature() method work for POST queries? I've combed your PR and haven't been able to find anything suggesting that it wouldn't work, but I seem to only be receiving this cryptic error when the method is POST: Any advice? |
|
Yes, I did the implementation for Post in 1.1.0 and 2.0.0. I tested with a few public WFS services, with both versions, but I did have this kind of error when testing on Boreas (https://boreas.ouranos.ca/geoserver/wfs) An ordinary Post request to that address with an xml filter worked well, but through OWSLib, that error always popped up with Post. It has to do with this part : https://github.com/f-PLT/OWSLib/blob/wfs-getfeature-post/owslib/feature/__init__.py#L314 Basically, I took this part from the If I got rid of the url finding cited above and went with All this is from memory, I'll take an other look at it on Monday to confirm things, and I could also give you the python script I used while implementing this. Assuming it's the same cause, of course. |
Fails with |
|
@f-PLT Making this test function pass is our use case. |
|
I see this header is already set in the code... stumped. |
|
Got this stack-trace server-side from |
|
I understand that OWSlib should be working with any instance of GeoServer that uses the proper standards, but I just want to raise a point here that our current instance is quite old (http://geoserver.org/release/2.9.3/). Do we know with certainty that this isn't a bug with POST in the build? I'll try doing a quick look over the advancements in the past few years to see if this is covered anywhere. Otherwise, perhaps the three of us can chat sometime? |
|
When I implemented the Post method, I ran accros this problem, but only while testing with Boreas (the only Ouranos Geoserver I tried). I had no problems with other Geoservers, either external or locally deployed. The only way I got it to work was to sacrifice this bit of code : https://github.com/f-PLT/OWSLib/blob/wfs-getfeature-post/owslib/feature/__init__.py#L312-L320 If I made it use the "outside" address The url given to OWSLib is The same code is present for the Get method, but it does not interfere with the request. I'm not familiar enough with how requests work and the finer points between Get vs Post requests, but my guess is it could be that the Post request does not handle redirection (or if it's possible to redirect it in the context of how OWSLib handles requests, which is outside of what I implemented). It would also be consistent with this: https://stackoverflow.com/questions/31620987/redirecting-http-post-requests-to-https-post-requests#:~:text=You%20cannot%20redirect.,3. and https://softwareengineering.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect From the above, to redirect a Post request seems like something we should avoid, so the ideal solution would be for the Geoserver's GetCapabilities to list the secured address. However, I'm pretty sure we rely on Geoserver being unsecured locally so it can interact with the other services. Maybe new flag in getFeature() to force the use of the provided url, instead of searching for the provided one in the GetCapabilities? Not sure how good a design that would be though. |
|
I can check into it a bit more with David B. this week, see if he has a better idea. |
Maybe we can take this opportunity to address Ouranosinc/pavics-sdi#183? Since right now we are unable to test Geoserver on other PAVICS instance than Boreas meaning if we perform this upgrade, where are we going to test the upgrade? |
Looks like bird-house/birdhouse-deploy#14 a Twitcher/Magpie bug or new feature needed. |
I've read those 2 links and agree that redirecting
I assume other services can be reconfigured to use the secure
If the flag is optional, then why not? You are not introducing any breaking changes and not forcing it on users either. |
I agree 100%
Well, short term, yes, I agree. It's also a really quick fix. Just to be sure we're talking about the same thing, the flag would need to be in OWSLib's From a design perspective, I'm wary of simply adding a flag to solve this problem. It's not a problem internal to OWSLib. The Post request, the HTTP protocol and getCapabilities (albeit with an url that is being redirected to httpS) are doing what they're suppose to do. The problem is (most probably) on our end with the way we've configured Geoserver. Introducing a flag argument in a public module for a very specific use case like this seems, well, out of place? I wouldn't like to see, down the line, 10 flags or more for different things unrelated to the core of the WFS standard. Just to be clear, I'm simply voicing my concerns here. At the end of the day, we do have time constraints and I'll implement what I'm told to implement. |
|
I'll also be double-checking again tomorrow to make sure a standard post request works with the Resquests library, and also without the url fetching done in OWSLib. |
|
I agree with @f-PLT on not using owslib to fix issues with our own infrastructure. |
|
@huard and I had checked over your examples the other week, and everything seemed fine when using the Requests library. For the purposes of Raven, I could potentially patch in a workaround based on your code if the priority is getting Raven up and running sooner.
I can throw some time into that issue this week if it's in the interest of the greater project development. For the HTTPS configurations, not quite my tempo. I know that if we can modernize the version, the data management side of the GeoServer would be much simpler for additional instances. This is a bit out of scope in this thread but let me know if there are other aspects of GeoServer that could use a helping hand. |
|
There is no need for a quick fix on this.
|
Good to know. If there's something else that is within Raven that needs help with, ping away. Shall I focus on Ouranosinc/pavics-sdi#183 then ? |
|
Hitting directly geoserver http://pavics.ouranos.ca:8087/geoserver/wfs?request=GetCapabilities&version=1.1.0 (only works inside Ouranos firewall), I am getting proper Not sure why behind Nginx and Twitcher/Magpie https://pavics.ouranos.ca/geoserver/wfs?request=GetCapabilities&version=1.1.0, we end up with the wrong http protocol and the wrong port Will attempt to put Geoserver directly behind Nginx see if the error persist. If yes then it's Twitcher/Magpie and most likely relate to bird-house/birdhouse-deploy#14 |
|
Digging further, geoserver is actually not behind Twitcher/Magpie, only behind Nginx. It's the same Nginx for everything else and the usual headers for performing front proxy is there. Will dig into why geoserver is miss configured. As for why geoserver data is not protected behind Twitcher/Magpie, it is a little weird since Magpie already have the config to protect it https://github.com/bird-house/birdhouse-deploy/blob/cc4ca20665e5eeb3d3ef51e1588521f4b42bef4d/birdhouse/config/magpie/providers.cfg.template#L74-L88 This predates me starting at Ouranos so should we enable this protection or it's not protected by Magpie for a particular reason? Someone remember? |
|
Does placing it behind Magpie complicate testing for CI ensembles/notebooks? Will users/services need authentication in order to perform WMS/WFS requests if GeoServer is behind it? I can't recall if I was ever privy to this discussion (it sounds like something my predecessor would have covered). |
For this issue, I don't think updating geoserver will do much. However, I did try Kartoza's latest geoserver image and the setup/configuration process has been streamlined. (env file and no need for JAVA_OPTS in the docker-compose) If we need to add the plugin for NetCDF to Geoserver, it could be a good opportunity to update at the same time. It's not a priority right now though. As for the configuration, I found this, which could be helpful : https://docs.geoserver.org/2.9.3/user/configuration/globalsettings.html#proxy-base-url |
It's also before I started working for CRIM, I'll bring it up at our next standup. |
|
Update on this.
This works but it's a manual UI change. I'd rather have everything automated. This would be the fall back if nothing else work.
This is not working. Not sure if I implemented it wrong or our geoserver do not support it (we have a custom docker build "inspired" from kartoza build). My attempt: bird-house/birdhouse-deploy@fddbd1e (not found too many documentation about configuring tomcat properties, I might get this wrong). Our custom geoserver docker build: https://github.com/bird-house/birdhouse-deploy/blob/cc4ca20665e5eeb3d3ef51e1588521f4b42bef4d/birdhouse/docker/geoserver/Dockerfile I'll spend the rest of the day finding alternative. Else are we okay with the manual change? I can do the manual change right now on Boreas for you guys to continue testing. |
Should not, Thredds is being Magpie and we have e2e notebook test using data from Thredds. Only 1 notebook test Thredds with authentication all the rest are accessing Thredds publicly.
I guess yes, if some path is protected. Else, like the current Thredds, everything is currently public so no auth required. In the current state (not behind Magpie) everything is public as well. |
I just did the manual change. https://pavics.ouranos.ca/geoserver/wfs?request=GetCapabilities&version=1.1.0 returns This is what we want? In the mean time, I'll keep looking for an automated way to set it. |
|
That looks right. I can try my hand at testing the POST capabilities using @f-PLT's scripts to get confirmation! |
|
This just pass for me https://github.com/bird-house/flyingpigeon/blob/7b7a5ce1dda2cde93db34667368ac224a75d58cc/tests/test_subset.py#L1-L10 (branch test_post) |
|
Seems to now be working perfectly for me as well: Gives me: Edit: Interesting that I don't receive the geometry though... Might be more to this... |
|
Excellent ! Trevor, we'll likely need this feature for the raven distributed model configuration we spoke about today, so this is great. |
By adding propertyname=None in a wfs 1.1.0 request, you'll get it to work. This is an unintended behavior, caused by a preset here : https://github.com/geopython/OWSLib/blob/master/owslib/feature/wfs110.py#L223 It looks voluntary, and works for a Get request, but for 2.0.0, it's defaulted to None : https://github.com/geopython/OWSLib/blob/master/owslib/feature/wfs200.py#L227 I'll see if I can fix it without breaking something else! |
|
I'll fix my part of the code so the default |
|
I got the automated way working but GetCapabilities returns https://pavics.ouranos.ca:443/geoserver/wfs (there's 443 in there). Pretty sure it's still working, my previous https://github.com/bird-house/flyingpigeon/blob/7b7a5ce1dda2cde93db34667368ac224a75d58cc/tests/test_subset.py#L1-L10 still pass. So just for your info if you notice something weird. It's live on Boreas right now for testing ... sorry I do not have another server with proper test data. |
…properly-behind-proxy Fix geoserver not configured properly behind proxy. Hitting https://pavics.ouranos.ca/geoserver/wfs?request=GetCapabilities&version=1.1.0 Before fix (wrong scheme and wrong port): ``` <ows:Operation name="GetCapabilities"> <ows:DCP> <ows:HTTP> <ows:Get xlink:href="http://pavics.ouranos.ca:80/geoserver/wfs"/> <ows:Post xlink:href="http://pavics.ouranos.ca:80/geoserver/wfs"/> </ows:HTTP> </ows:DCP> ``` After fix: ``` <ows:Operation name="GetCapabilities"> <ows:DCP> <ows:HTTP> <ows:Get xlink:href="https://pavics.ouranos.ca:443/geoserver/wfs"/> <ows:Post xlink:href="https://pavics.ouranos.ca:443/geoserver/wfs"/> </ows:HTTP> </ows:DCP> ``` This config automate manual step to set proxy base url in Geoserver UI https://docs.geoserver.org/2.9.3/user/configuration/globalsettings.html#proxy-base-url I had to override the docker image entrypoint to edit the `server.xml` on the fly before starting Geoserver (Tomcat) since setting Java proxy config did not seem to work (see first commit). Related to Ouranosinc/raven#297.
Description
We need to make WFS requests for info on many subregions. For a large number of regions, the GET requests exceeds the maximum size. See geopython/OWSLib#439
Francis made a PR geopython/OWSLib#706 to solve this, and we'd need a review of the PR, as well as confirmation that it solves our issue, illustrated in https://pavics-sdi.readthedocs.io/projects/raven/en/latest/notebooks/Subset_climate_data_over_watershed.html#Subsetting-a-gridded-climate-dataset
https://github.com/Ouranosinc/raven/blob/master/docs/source/notebooks/gridded_data_subset.ipynb
The notebook sets the number of features to 2, but it should work with 100s of features.
TODO:
method="POST"toflyingpigeon.subset_base.get_featureThe text was updated successfully, but these errors were encountered: