Initial libpostal integration

[trescube]

This is the big PR to incorporate libpostal functionality. It incorporates several large changes of functionality:

• query/search.js - disables all scoring temporarily, switches to FallbackQuery/GeodisambiguationQuery
• middleware/trimByGranularity.js - choose most granular results when FallbackQuery was used

Because addressIt is still needed for the /autocomplete endpoint, a separate query/texetparser.js was added just for it. For a discussion on what FallbackQuery and Geodisambiguation, see:

• https://github.com/pelias/query/blob/master/layout/FilteredBooleanQuery.js
• https://github.com/pelias/query/blob/master/layout/GeodisambiguationQuery.js

[missinglink]

this should be true?

[trescube]

Ideally yes but waiting on #601

[orangejulius]

Looks like the final commit, a merge of a branch on github that you had probably rebased locally (2744244), both duplicated commit history and added a few unwanted changes (like adding stats-lite as a dependency twice). We can fix it whenever you have a few minutes

[vesameskanen]

I did a quick test with this branch, and found out that house number analysis is now working quite well, and we no longer get streets from wrong cities. However, I am quite concerned about the character conversion which libpostal does: ä -> ae etc. This breaks big part of searches. Any ideas how to fix this?

[albarrentine]

@vesameskanen the libpostal models need to do those normalizations at some level so länderstraße, laenderstrasse, and landerstrasse can share statistical parameters.

I'm not sure if Pelias is handling umlaut transliteration now (looks like it mostly does accent stripping, "Muenchen" for instance does not return Munich but "Munchen" does), but it should be possible to use libpostal without breaking any such searches by either indexing the transliterated forms from libpostal, or using the compound analyzer approach detailed here or here. From what I understand, Dutch and the Nordic languages usually drop the umlaut when writing on an ASCII keyboard, etc. with two exceptions in Swedish : ü => y and å => aa, but these are a little less frequently used.

There has been some discussion in the libpostal repo about returning only segments of the original input without any lowercasing or transliteration (or only performing those normalizations on some internal representation) for people looking to display the parser results in some way. It's a little complicated to do that in the current implementation of the transliterator, and not a huge priority for search, but it's on the radar.

[vesameskanen]

Hi @thatdatabaseguy, many thanks for your reply and also for the great libpostal tool, which seems to parse also Finnish addresses amazingly well.
I was already suspecting that the consequences of normalization need to be addressed on Pelias side. @trescube probably already has a plan how to handle this :) We are running a Pelias instance with Finnish data here, and try to keep our code base as well synced with upstream as possible, because Pelias is developing fast. Poking the actual ES queries Pelias creates is a bit challenging at this point.

[vesameskanen]

OK, found out the following related pull requests:

#pelias/schema#146
#pelias/schema#149

I guess a solution is under way :)