Skip to content
This repository has been archived by the owner on Mar 6, 2019. It is now read-only.

Interpolation and Polylines issues #25

Closed
fpurcell opened this issue Aug 28, 2017 · 8 comments
Closed

Interpolation and Polylines issues #25

fpurcell opened this issue Aug 28, 2017 · 8 comments
Assignees

Comments

@fpurcell
Copy link

Hi Diana,

I just tried rebuilding the example pelias.json in the dockerfiles directory. I'm not seeing polylines get processed, and as a result interpolation doesn't build. Below is a listing of the data directory (note that the interpolation folder is empty, and that there isn't a polylines directory):

[pelias_docker@cs-dv-mapgeo01 dockerfiles]$ ls ~/data/*
/srv/pelias_docker/data/elasticsearch:
elasticsearch

/srv/pelias_docker/data/interpolation:

/srv/pelias_docker/data/openaddresses:
README.txt us

/srv/pelias_docker/data/openstreetmap:
portland_oregon.osm.pbf

/srv/pelias_docker/data/placeholder:
graph.json store.sqlite3 wof.extract

/srv/pelias_docker/data/tiger:
downloads shapefiles

/srv/pelias_docker/data/whosonfirst:
data meta

Here's the failure from the log file:

pelias-interpolation@0.0.0-semantic-release build /code/pelias/interpolation
./script/build.sh

  • importing polylines
    poyline line not found

npm ERR! Linux 3.10.0-514.26.2.el7.x86_64
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "build"
npm ERR! node v4.8.4
npm ERR! npm v2.15.11
npm ERR! code ELIFECYCLE
npm ERR! pelias-interpolation@0.0.0-semantic-release build: ./script/build.sh
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the pelias-interpolation@0.0.0-semantic-release build script './script/build.sh'.
npm ERR! This is most likely a problem with the pelias-interpolation package,

@fpurcell
Copy link
Author

Recent commits (e5fdc67) have Pelias building interpolation and polylines via Docker. The problem I'm now seeing with interpolation is that I keep getting 404 responses, ala:

info: [api] [controller:search] [queryType:fallback] [es_result_count:7]
error: [interpolation] http://interpolation:4300/search/geojson?number=838&street=Southeast%20Lambert%20Street&lat=45.467254&lon=-122.604483 returned status 404: {}
error: [api] [middleware:interpolation] http://interpolation:4300/search/geojson?number=838&street=Southeast%20Lambert%20Street&lat=45.467254&lon=-122.604483 returned status 404: {}

@fpurcell
Copy link
Author

fpurcell commented Aug 30, 2017

A bit more on interpolation ... this request:
http://cs-dv-mapgeo01:4000/v1/search?text=888%20SE%20Lambert%20St

Returns this logging:

info: [api] [lang] 'en' via 'header'
info: [api] [req] endpoint=/v1/search text=888 SE Lambert St, size=10, private=false, name=English, iso6391=en, iso6393=eng, defaulted=false, querySize=20, number=888, street=se lambert st
info: [api] [query:address_search_using_ids] [parser:libpostal] [param:querySize]
info: [api] [controller:search] [queryType:fallback] [es_result_count:7]
error: [interpolation] http://interpolation:4300/search/geojson?number=888&street=Southeast%20Lambert%20Street&lat=45.467254&lon=-122.604483 returned status 404: {}
error: [api] [middleware:interpolation] http://interpolation:4300/search/geojson?number=888&street=Southeast%20Lambert%20Street&lat=45.467254&lon=-122.604483 returned status 404: {}
info: [api] [dupe][skipping] query=888 SE Lambert St, previous=openstreetmap, hit=Southeast Lambert Street openstreetmap:polyline:18867
info: [api] [dupe][skipping] query=888 SE Lambert St, previous=openstreetmap, hit=Southeast Lambert Street openstreetmap:polyline:18869
info: [api] [dupe][skipping] query=888 SE Lambert St, previous=openstreetmap, hit=Southeast Lambert Street openstreetmap:polyline:18868
info: [api] [dupe][skipping] query=888 SE Lambert St, previous=openstreetmap, hit=Southeast Lambert Street openstreetmap:polyline:18870
error: [interpolation] http://interpolation:4300/search/geojson?number=888&street=Southeast%20Lambert%20Street&lat=45.467079&lon=-122.626516 returned status 404: {}
error: [interpolation] http://interpolation:4300/search/geojson?number=888&street=Southeast%20Lambert%20Street&lat=45.467381&lon=-122.596327 returned status 404: {}
error: [interpolation] http://interpolation:4300/search/geojson?number=888&street=Southeast%20Lambert%20Street&lat=45.466836&lon=-122.589073 returned status 404: {}
error: [interpolation] http://interpolation:4300/search/geojson?number=888&street=Southeast%20Lambert%20Street&lat=45.46688&lon=-122.580125 returned status 404: {}
info: [api] ::ffff:172.25.32.37 - - [30/Aug/2017:19:53:58 +0000] "GET /v1/search?text=888%20SE%20Lambert%20St HTTP/1.1" 200 3106

And this response:

{
type: "Feature",
geometry: {
type: "Point",
coordinates: [
-122.604483,
45.467254
]
},
properties: {
id: "polyline:18864",
gid: "openstreetmap:street:polyline:18864",
layer: "street",
source: "openstreetmap",
source_id: "polyline:18864",
name: "Southeast Lambert Street",
street: "Southeast Lambert Street",
confidence: 0.8,
match_type: "fallback",
accuracy: "centroid",
country: "United States",
country_gid: "whosonfirst:country:85633793",
country_a: "USA",
region: "Oregon",
region_gid: "whosonfirst:region:85688513",
region_a: "OR",
county: "Multnomah County",
county_gid: "whosonfirst:county:102081631",
locality: "Portland",
locality_gid: "whosonfirst:locality:101715829",
neighbourhood: "Brentwood-Darlington",
neighbourhood_gid: "whosonfirst:neighbourhood:420781227",
label: "Southeast Lambert Street, Portland, OR, USA"
},
bbox: [
-122.609368,
45.467248,
-122.599598,
45.467255
]
},

@fpurcell
Copy link
Author

fpurcell commented Sep 7, 2017

NOTE: I have tried to both 'build' and 'pull' the Docker components (e.g., I changed line #9 to "docker-compose pull;"), and I get the same result. In both cases, before running ./build.sh, I had a fresh Docker system, with no existing containers, images nor volumes on the system (I also rebooted the machine, just for good measure).

@fpurcell
Copy link
Author

fpurcell commented Sep 7, 2017

BTW, all this is with minimal change to what's in dockerfiles:

[pelias_docker@cs-dv-mapgeo01 dockerfiles]$ git diff
diff --git a/build.sh b/build.sh
index 39a3aa8..88568e6 100755
--- a/build.sh
+++ b/build.sh
@@ -6,7 +6,7 @@
 docker-compose down || true;
 
 # rebuild the images
-docker-compose build;
+docker-compose pull;
 
 time sh ./prep_data.sh;
 
diff --git a/pelias.json b/pelias.json
index 333362b..990ab66 100644
--- a/pelias.json
+++ b/pelias.json
@@ -64,7 +64,7 @@
       "importVenues": false,
       "importPostalcodes": true,
       "importPlace": "101715829",
-      "api_key": "your-api-key"
+      "api_key": "mapzen-my-key"
     },
     "interpolation": {
       "download": {

@dianashk dianashk self-assigned this Sep 20, 2017
@dianashk
Copy link
Contributor

@fpurcell, we've just pushed some updates to the dockerfile setup. please test them out when you get a chance and let us know if there are any outstanding problems.

@fpurcell
Copy link
Author

Hi Diana,

I'm still seeing issues with interpolation, and similar (e.g., "fallback") results with this query: http://cs-dv-mapgeo01:4000/v1/search?text=888%20SE%20Lambert%20St

Starting from a clean system (all Docker containers and volumes removed, all downloaded data removed, server restarted, etc...), running dockerfiles/build.sh produces the following errors when it comes to polylines. (BTW, the only git diff from commit 6a96744 is the mapzen api_key in pelias.json):

Starting pelias_baseimage
Pulling polylines (pelias/polylines:latest)...
latest: Pulling from pelias/polylines
Digest: sha256:8fc31962b73a4c8602c38787d491a79436695dc2ba835a4c5664e36427f52fff
Status: Downloaded newer image for pelias/polylines:latest
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 9, in
load_entry_point('docker-compose==1.9.0', 'console_scripts', 'docker-compose')()
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 65, in main
command()
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 117, in perform_command
handler(command, command_options)
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 712, in run
run_one_off_container(container_options, self.project, service, options)
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 998, in run_one_off_container
pty.start(sockets)
File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 334, in start
self._hijack_tty(pumps)
File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 373, in _hijack_tty
pump.flush()
File "/usr/lib/python2.7/site-packages/dockerpty/io.py", line 378, in flush
raise e
OSError: [Errno 9] Bad file descriptor
Starting pelias_baseimage
Pulling placeholder (pelias/placeholder:latest)...
latest: Pulling from pelias/placeholder
Digest: sha256:ed1a5b86a797ff95f8cff060ac792934512cd603649d8c8e838ddc2fa174a6a6
Status: Downloaded newer image for pelias/placeholder:latest
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 9, in
load_entry_point('docker-compose==1.9.0', 'console_scripts', 'docker-compose')()
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 65, in main
command()
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 117, in perform_command
handler(command, command_options)
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 712, in run
run_one_off_container(container_options, self.project, service, options)
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 998, in run_one_off_container
pty.start(sockets)
File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 334, in start
self._hijack_tty(pumps)
File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 373, in _hijack_tty
pump.flush()
File "/usr/lib/python2.7/site-packages/dockerpty/io.py", line 378, in flush
raise e
OSError: [Errno 9] Bad file descriptor
Starting pelias_baseimage
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 9, in
load_entry_point('docker-compose==1.9.0', 'console_scripts', 'docker-compose')()
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 65, in main
command()
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 117, in perform_command
handler(command, command_options)
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 712, in run
run_one_off_container(container_options, self.project, service, options)
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 998, in run_one_off_container
pty.start(sockets)
File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 334, in start
self._hijack_tty(pumps)
File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 373, in _hijack_tty
pump.flush()
File "/usr/lib/python2.7/site-packages/dockerpty/io.py", line 378, in flush
raise e
OSError: [Errno 9] Bad file descriptor
Starting pelias_baseimage

Attached is the entire log file: nohup.txt

Thanks,
Frank

@fpurcell
Copy link
Author

fpurcell commented Sep 29, 2017

I re-ran build.sh with 'docker-compose pull;', and see the same errors as above.

Starting pelias_baseimage
Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 9, in <module>
    load_entry_point('docker-compose==1.9.0', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 65, in main
    command()
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 117, in perform_command
    handler(command, command_options)
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 712, in run
    run_one_off_container(container_options, self.project, service, options)
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 998, in run_one_off_container
    pty.start(sockets)
  File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 334, in start
    self._hijack_tty(pumps)
  File "/usr/lib/python2.7/site-packages/dockerpty/pty.py", line 373, in _hijack_tty
    pump.flush()
  File "/usr/lib/python2.7/site-packages/dockerpty/io.py", line 378, in flush
    raise e
OSError: [Errno 9] Bad file descriptor
Starting pelias_baseimage
...

nohup-pull.txt

@fpurcell
Copy link
Author

fpurcell commented Oct 5, 2017

5 days later, and some very good news. 'docker-compose pull;', is now seeing a Pelias instance build where interpolation runs successfully. (Tempering that success, 'docker-compose build;' has issues, ala #31).

http://cs-dv-mapgeo01:4000/v1/search?text=888%20SE%20Lambert%20St

{
"geocoding": {
"version": "0.2",
"attribution": "http://pelias.mapzen.com/v1/attribution",
"query": {
"text": "888 SE Lambert St",
"size": 10,
"private": false,
"lang": {
"name": "English",
"iso6391": "en",
"iso6393": "eng",
"defaulted": false
},
"querySize": 20,
"parsed_text": {
"number": "888",
"street": "se lambert st"
}
},
"engine": {
"name": "Pelias",
"author": "Mapzen",
"version": "1.0"
},
"timestamp": 1507175080699
},
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
-122.656975,
45.468603
]
},
"properties": {
"id": "polyline:20949",
"gid": "mixed:address:polyline:20949",
"layer": "address",
"source": "mixed",
"name": "888 Southeast Lambert Street",
"housenumber": "888",
"street": "Southeast Lambert Street",
"confidence": 0.8,
"match_type": "interpolated",
"accuracy": "point",
"country": "United States",
"country_gid": "whosonfirst:country:85633793",
"country_a": "USA",
"region": "Oregon",
"region_gid": "whosonfirst:region:85688513",
"region_a": "OR",
"county": "Multnomah County",
"county_gid": "whosonfirst:county:102081631",
"locality": "Portland",
"locality_gid": "whosonfirst:locality:101715829",
"neighbourhood": "Sellwood-Moreland",
"neighbourhood_gid": "whosonfirst:neighbourhood:85867151",
"label": "888 Southeast Lambert Street, Portland, OR, USA"
}
},
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
-122.604483,
45.467254
]
},
"properties": {
"id": "polyline:20948",
"gid": "openstreetmap:street:polyline:20948",
"layer": "street",
"source": "openstreetmap",
"source_id": "polyline:20948",
"name": "Southeast Lambert Street",
"street": "Southeast Lambert Street",
"confidence": 0.8,
"match_type": "fallback",
"accuracy": "centroid",
"country": "United States",
"country_gid": "whosonfirst:country:85633793",
"country_a": "USA",
"region": "Oregon",
"region_gid": "whosonfirst:region:85688513",
"region_a": "OR",
"county": "Multnomah County",
"county_gid": "whosonfirst:county:102081631",
"locality": "Portland",
"locality_gid": "whosonfirst:locality:101715829",
"neighbourhood": "Brentwood-Darlington",
"neighbourhood_gid": "whosonfirst:neighbourhood:420781227",
"label": "Southeast Lambert Street, Portland, OR, USA"
},
"bbox": [
-122.609368,
45.467248,
-122.599598,
45.467255
]
},
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
-122.626516,
45.467079
]
},
"properties": {
"id": "polyline:20950",
"gid": "openstreetmap:street:polyline:20950",
"layer": "street",
"source": "openstreetmap",
"source_id": "polyline:20950",
"name": "Southeast Lambert Street",
"street": "Southeast Lambert Street",
"confidence": 0.8,
"match_type": "fallback",
"accuracy": "centroid",
"country": "United States",
"country_gid": "whosonfirst:country:85633793",
"country_a": "USA",
"region": "Oregon",
"region_gid": "whosonfirst:region:85688513",
"region_a": "OR",
"county": "Multnomah County",
"county_gid": "whosonfirst:county:102081631",
"locality": "Portland",
"locality_gid": "whosonfirst:locality:101715829",
"neighbourhood": "Eastmoreland",
"neighbourhood_gid": "whosonfirst:neighbourhood:85817069",
"label": "Southeast Lambert Street, Portland, OR, USA"
},
"bbox": [
-122.635486,
45.467069,
-122.617028,
45.468617
]
}
],
"bbox": [
-122.656975,
45.467069,
-122.599598,
45.468617
]
}

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants