fix for embedded json in wps request

[cehbrecht]

Overview

This PR is a quick fix to parse an embedded json (complex type) from the wps execute request (xml).

Probably there is more work to do for embedding json in OWSLib and PyWPS. Using CDATA or always use base64 encoding?

Related Issue / Discussion

Additional Information

Contribution Agreement

(as per https://github.com/geopython/pywps/blob/master/CONTRIBUTING.rst#contributions-and-licensing)

• I'd like to contribute [feature X|bugfix Y|docs|something else] to PyWPS. I confirm that my contributions to PyWPS will be compatible with the PyWPS license guidelines at the time of contribution.
• [x ] I have already previously agreed to the PyWPS Contributions and Licensing Guidelines

[cehbrecht]

A wps execute request with embedded json looks like this:

<wps100:Execute
  xmlns:wps100="http://www.opengis.net/wps/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  service="WPS"
  version="1.0.0"
  xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsExecute_request.xsd">
  <ows110:Identifier xmlns:ows110="http://www.opengis.net/ows/1.1">orchestrate</ows110:Identifier>
  <wps100:DataInputs>
    <wps100:Input>
      <ows110:Identifier xmlns:ows110="http://www.opengis.net/ows/1.1">workflow</ows110:Identifier>
      <wps100:Data>
        <wps100:ComplexData encoding="utf-8" mimeType="application/json">{"inputs": {"tas": ["CMIP6.ScenarioMIP.MIROC.MIROC6.ssp119.r1i1p1f1.day.tas.gn.v20191016"]}, "steps": {"subset_tas_1": {"run": "subset", "in": {"collection": "inputs/tas", "time": "2016-01-01/2016-03-31"}}}, "outputs": {"output": "subset_tas_1/output"}, "doc": "workflow"}</wps100:ComplexData>
      </wps100:Data>
    </wps100:Input>
  </wps100:DataInputs>
  <wps100:ResponseForm>
    <wps100:ResponseDocument storeExecuteResponse="true" status="true" lineage="false">
      <wps100:Output asReference="true">
        <ows110:Identifier xmlns:ows110="http://www.opengis.net/ows/1.1">output</ows110:Identifier>
      </wps100:Output>
      <wps100:Output asReference="true">
        <ows110:Identifier xmlns:ows110="http://www.opengis.net/ows/1.1">prov</ows110:Identifier>
      </wps100:Output><wps100:Output asReference="true">
        <ows110:Identifier xmlns:ows110="http://www.opengis.net/ows/1.1">prov_plot</ows110:Identifier>
      </wps100:Output>
    </wps100:ResponseDocument>
  </wps100:ResponseForm>
</wps100:Execute>

[huard]

What happens if the json content includes xml characters (<>) ?

[tomkralidis]

Good point @huard ; can/should we CDATA the content?

[huard]

Wait, I got confused, this is already the inside of the CDATA content, correct ?

Is there really something to do in OWSLib? I'm using JSON ComplexInputs in production for a while and everything works fine (using the birdy client).

[coveralls]

Coverage increased (+0.01%) to 75.498% when pulling ed7be29 on cehbrecht:fix-json-input into 81bb1e4 on geopython:pywps-4.2.

[tomkralidis]

Not sure, I don't see CDATA in the example in #560 (comment)

[cehbrecht]

Not sure, I don't see CDATA in the example in #560 (comment)

Yes, I think there was no CDATA so far. We were just lucky :)

[cehbrecht]

I would need a quick fix for now ... and make it better later.

[cehbrecht]

The code is very tricky here:

pywps/pywps/app/WPSRequest.py

Lines 576 to 586 in 20e1e25

	def _get_rawvalue_value(data, encoding=None):
	"""Return real value of CDATA section"""
	try:
	if encoding is None or encoding == "":
	return data
	elif encoding == 'base64':
	return base64.b64decode(data)
	return base64.b64decode(data)
	except Exception:
	return data

When the base64 decoding fails then it would return the original data. That is why sometimes our json input parameter worked and sometimes not ... depending on the characters.

ping @agstephens

[cehbrecht]

@tomkralidis @huard I'm merging this PR as a quick-fix. Needs to be done right later (#562)

[gschwind]

Hello,

As I said in my comment here, in my opinion the proper fix is to not decode string as base64 when the attribute encoding is not base64.

The json embedded data must be properly escaped, using CDATA or XML entities when needed. The etree.parser will remove them properly. Moreover the encoding of such embedded json has to be the same as the XML as far I understand XML.

As side note, json standard does not specify any encoding, it is defined as Unicode code point, thus any encoding of Unicode is a valid choice. In that regards the other option to send json is by specifying charset in the mime type as follow: mimetype=application/json; charset=utf-8 and using encoding=base64 . But usually the paramter charset of mimetype is used for text mime type not application one.

[gschwind]

After a double check even more in WPS 1.0.0, it seems that there is not base64 references, only the WPS 2.0.0 has base64 reference.