Points returned from ES have incorrect coordinates in Geoserver

[timcroydon]

Geoserver version: 2.12.1 (clean, out-the-box install)
Elasticgeo version: 2.12.1-RC1
ES version: 6.1.1 (clean install)
OS: OSX High Sierra

I have created an index in ES containing a few simple points. When I set up the store/layer in Geoserver everything connects and the point field (location) is detected correctly, with the SRS set to EPSG:4326.

Points are returned correctly from ES as stored (e.g. around [51,0]) - this output is from Wireshark:

{"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":4,"max_score":1.0,"hits":[{"_index":"geotest","_type":"points","_id":"EEbU5WABrjASy1turEup","_score":1.0,"_source":{"name":"point 2","location":"51, 0"}},{"_index":"geotest","_type":"points","_id":"D0bU5WABrjASy1tujkuW","_score":1.0,"_source":{"name":"point 1","location":"51, -1"}},{"_index":"geotest","_type":"points","_id":"EUbU5WABrjASy1tuyktz","_score":1.0,"_source":{"name":"point 3","location":"50, 0"}},{"_index":"geotest","_type":"points","_id":"EkbU5WABrjASy1tu4UtE","_score":1.0,"_source":{"name":"point 4","location":"50, -1"}}]}}


However, when previewing the data, the points are shifted to a completely different area (e.g. [51,-1] -> [-33.78845215, -78.78570557]. Example GML output shown here, but values are same for all output formats:

<gml:featureMember>
  <estest:points fid="points.D0bU5WABrjASy1tujkuW">
    <estest:_id>D0bU5WABrjASy1tujkuW</estest:_id>
    <estest:_index>geotest</estest:_index>
    <estest:_type>points</estest:_type>
    <estest:_score>1.0</estest:_score>
    <estest:_relative_score>1.0</estest:_relative_score>
    <estest:location>
      <gml:Point srsName="http://www.opengis.net/gml/srs/epsg.xml#4326">
        <gml:coordinates xmlns:gml="http://www.opengis.net/gml" decimal="." cs="," ts=" ">-33.78845215,-78.78570557</gml:coordinates>
      </gml:Point>
    </estest:location>
    <estest:name>point 1</estest:name>
    <estest:name.fields.keyword>point 1</estest:name.fields.keyword>
  </estest:points>
</gml:featureMember>

I'm not sure if this is an Elasticgeo issue or a Geoserver config issue - the logs don't reveal much, even at TRACE level and I'm not sure how to resolve. As far as I can tell, everything is set to EPSG:4326 so I don't see why any conversion should take place.

Thanks in advance for any pointers!

[sjudeng]

The plugin supports geo-points with formats described in https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html. The string format currently strictly expects a format of "lat,lon" (no space) as indicated in the above doc. When a string coordinate doesn't match this format it's treated as a Geohash, which is why the coordinates end up that way (e.g. the Geohash of 51, -1 corresponds to a coordinate of [-33.78845215, -78.78570557]).

I'll keep this issue open for updating to support spaces in geo-point strings. In the meantime you can workaround the issue by storing locations without spaces (e.g. {"location": "50,-1"}) or using one of the other supported formats (array, object or Geohash).

[timcroydon]

Thank you.

You are correct - removing the space does fix the issue.

I did have a go with the other formats though and, while object and geohash worked fine, arrays didn't. e.g. in ES, I have an object like this:

{
  "_index": "geotest",
  "_type": "points",
  "_id": "FkaF_2ABrjASy1tuREtV",
  "_score": 1,
    "_source": {
    "name": "point 1",
    "location": [
      -1,
      50
    ]
  }
}

which comes out as this on GML, i.e. missing the estest:location property:

<gml:featureMember>
<estest:points fid="points.FkaF_2ABrjASy1tuREtV">
<estest:_id>FkaF_2ABrjASy1tuREtV</estest:_id>
<estest:_index>geotest</estest:_index>
<estest:_type>points</estest:_type>
<estest:_score>1.0</estest:_score>
<estest:_relative_score>1.0</estest:_relative_score>
<estest:name>point 1</estest:name>
<estest:name.fields.keyword>point 1</estest:name.fields.keyword>
</estest:points>
</gml:featureMember>

[sjudeng]

With arrays does it work if you store floats instead of integers (e.g. {"location": [-1.0, 50.0]})?

[timcroydon]

No, it doesn't, although the data is returned correctly by ES, e.g.:

{"took":1,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":4,"max_score":1.0,"hits":
[{"_index":"geotest","_type":"points","_id":"F0aF_2ABrjASy1tuUkti","_score":1.0,"_source":{"name":"point 1","location":"50,0"}},
{"_index":"geotest","_type":"points","_id":"FEaF_2ABrjASy1tuEktV","_score":1.0,"_source":{"name":"point 1","location":"51,0"}},
{"_index":"geotest","_type":"points","_id":"FUaF_2ABrjASy1tuNUu0","_score":1.0,"_source":{"name":"point 1","location":"51,-1"}},
{"_index":"geotest","_type":"points","_id":"FkaF_2ABrjASy1tuREtV","_score":1.0,"_source":{"name":"point 1","location":[10.123,60.456]}}]}}

GML shows correct points for the first 3 points, but no location property for the last array one. (Mixing formats doesn't seem to matter).

[sjudeng]

For the plugin the geo-point format does need to be consistent for all documents. Did you check the result if all documents contain a float array location?

[timcroydon]

Yes. Exactly the same thing. I tried recreating the ES index, and then restarting Geoserver and recreating the workspace/store/layer and same thing. Interestingly, doing 'compute bounds from data' it produces incorrect values too (ranges should have been [1.1, 1.1] to [4.4, 4.4])

Incidentally, if you create a point in ES of the form [0.0, 50.0], ES returns [0, 50] in query results anyway. I tried with, e.g. 1.1, 2.2, etc. to prevent truncation of zeros though.

Corresponding ES data:

{"took":0,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":4,"max_score":1.0,"hits":[{"_index":"geotest","_type":"points","_id":"I0asAGEBrjASy1tupUuU","_score":1.0,"_source":{
  "name": "point 3",
  "location": [3.3, 3.3] 
}
},{"_index":"geotest","_type":"points","_id":"IUasAGEBrjASy1tubUth","_score":1.0,"_source":{
  "name": "point 1",
  "location": [1.1, 1.1] 
}
},{"_index":"geotest","_type":"points","_id":"IkasAGEBrjASy1tuiUvO","_score":1.0,"_source":{
  "name": "point 2",
  "location": [2.2, 2.2] 
}
},{"_index":"geotest","_type":"points","_id":"JEasAGEBrjASy1tuxUsX","_score":1.0,"_source":{
  "name": "point 4",
  "location": [4.4, 4.4] 
}
}]}}

(GML as before)

[sjudeng]

In Elasticsearch can you confirm that the type for the location field is still geo_point in the new index? Also in GeoServer check the feature type details for the layer and confirm that the type for the location property is Point.

[timcroydon]

Elasticsearch index mapping:

{
  "geotest": {
    "mappings": {
      "points": {
        "properties": {
          "location": {
            "type": "geo_point"
          },
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
}

And in Geoserver, it correctly detects and identifies the type (not sure what the asterisk indicates - presumably that it's a geo type?):

[sjudeng]

Okay thanks for debugging. I'll see if I can reproduce this and report back here with any findings. In the meantime you can use object, geohash or string (without spaces) as a workaround.

[timcroydon]

Thank you for looking at this. The plugin looks extremely useful to our project.

Unfortunately I'm not in a position to dig into the code myself just now, but happy to help where I can - let me know if you need more info.

[sjudeng]

The updates to hopefully resolve this have been merged. I'll create new release binaries once GeoServer 2.12.2 is released. In the meantime you can build and test the updates off master.

[sjudeng]

The 2.12.2-RC1 release candidate that includes this update is now published.

[timcroydon]

Brilliant, that works fine now (2.12.2-RC1).

I tested with arrays, strings (with and without space), lat/lon as properties and geohash and all display correctly in Geoserver layer preview.

Thanks for your help and fast turnaround 🙂