This is an implementation of a plugin for Elasticsearch using the implementation of Nakatani Shuyo’s language detector.
It uses 3-gram character and a Bayesian filter with various normalizations and feature sampling. The precision is over 99% for 53 languages.
The plugin offers a mapping type to specify fields where you want to enable language detection. Detected languages are indexed into a subfield of the field named 'lang', as you can see in the example. The field can be queried for language codes.
You can use the multi_field mapping type to combine this plugin with the attachment mapper plugin, to enable language detection in base64-encoded binary data. Currently, UTF-8 texts are supported only.
The plugin offers also a REST endpoint, where a short text can be posted to in UTF-8, and the plugin responds with a list of recognized languages.
Here is a list of languages code recognized:
Code |
Description |
af |
Afrikaans |
ar |
Arabic |
bg |
Bulgarian |
bn |
Bengali |
cs |
Czech |
da |
Danish |
de |
German |
el |
Greek |
en |
English |
es |
Spanish |
et |
Estonian |
fa |
Farsi |
fi |
Finnish |
fr |
French |
gu |
Gujarati |
he |
Hebrew |
hi |
Hindi |
hr |
Croatian |
hu |
Hungarian |
id |
Indonesian |
it |
Italian |
ja |
Japanese |
kn |
Kannada |
ko |
Korean |
lt |
Lithuanian |
lv |
Latvian |
mk |
Macedonian |
ml |
Malayalam |
mr |
Marathi |
ne |
Nepali |
nl |
Dutch |
no |
Norwegian |
pa |
Eastern Punjabi |
pl |
Polish |
pt |
Portuguese |
ro |
Romanian |
ru |
Russian |
sk |
Slovak |
sl |
Slovene |
so |
Somali |
sq |
Albanian |
sv |
Swedish |
sw |
Swahili |
ta |
Tamil |
te |
Telugu |
th |
Thai |
tl |
Tagalog |
tr |
Turkish |
uk |
Ukrainian |
ur |
Urdu |
vi |
Vietnamese |
zh-cn |
Chinese |
zh-tw |
Traditional Chinese characters (Taiwan, Hongkong, Macau) |
|
Note
|
The examples are written for Elasticsearch 5.x and need to be adapted to earlier versions of Elastiscearch. |
In this example, we create a simple detector field, and write text to it for detection.
DELETE /test
PUT /test
{
"mappings": {
"docs": {
"properties": {
"text": {
"type": "langdetect",
"languages" : [ "en", "de", "fr" ]
}
}
}
}
}
PUT /test/docs/1
{
"text" : "Oh, say can you see by the dawn`s early light, What so proudly we hailed at the twilight`s last gleaming?"
}
PUT /test/docs/2
{
"text" : "Einigkeit und Recht und Freiheit für das deutsche Vaterland!"
}
PUT /test/docs/3
{
"text" : "Allons enfants de la Patrie, Le jour de gloire est arrivé!"
}
POST /test/_search
{
"query" : {
"term" : {
"text" : "en"
}
}
}
POST /test/_search
{
"query" : {
"term" : {
"text" : "de"
}
}
}
POST /test/_search
{
"query" : {
"term" : {
"text" : "fr"
}
}
}Just indexing the language code is not enough in most cases. The language-detected text
should be passed to a specific analyzer to apply language-specific analysis. This plugin
allows that by the language_to parameter.
DELETE /test
PUT /test
{
"mappings": {
"docs": {
"properties": {
"text": {
"type": "langdetect",
"languages": [
"de",
"en",
"fr",
"nl",
"it"
],
"language_to": {
"de": "german_field",
"en": "english_field"
}
},
"german_field": {
"analyzer": "german",
"type": "string"
},
"english_field": {
"analyzer": "english",
"type": "string"
}
}
}
}
}
PUT /test/docs/1
{
"text" : "Oh, say can you see by the dawn`s early light, What so proudly we hailed at the twilight`s last gleaming?"
}
POST /test/_search
{
"query" : {
"match" : {
"english_field" : "light"
}
}
}Using multifields, it is possible to store the text alongside with the detected language(s). Here, we use another (short nonsense) example text for demonstration, which has more than one detected language code.
DELETE /test
PUT /test
{
"mappings": {
"docs": {
"properties": {
"text": {
"type": "text",
"fields": {
"language": {
"type": "langdetect",
"languages": [
"de",
"en",
"fr",
"nl",
"it"
],
"store": true
}
}
}
}
}
}
}
PUT /test/docs/1
{
"text" : "Oh, say can you see by the dawn`s early light, What so proudly we hailed at the twilight`s last gleaming?"
}
POST /test/_search
{
"query" : {
"match" : {
"text" : "light"
}
}
}
POST /test/_search
{
"query" : {
"match" : {
"text.language" : "en"
}
}
}DELETE /test
PUT /test
{
"mappings": {
"docs": {
"properties": {
"text": {
"type" : "attachment",
"fields" : {
"content" : {
"type" : "text",
"fields" : {
"language" : {
"type" : "langdetect",
"binary" : true
}
}
}
}
}
}
}
}
}On a shell, enter commands
rm index.tmp
echo -n '{"content":"' >> index.tmp
echo "This is a very simple text in plain english" | base64 >> index.tmp
echo -n '"}' >> index.tmp
curl -XPOST --data-binary "@index.tmp" 'localhost:9200/test/docs/1'
rm index.tmpPOST /test/_refresh
POST /test/_search
{
"query" : {
"match" : {
"content" : "very simple"
}
}
}
POST /test/_search
{
"query" : {
"match" : {
"content.language" : "en"
}
}
}curl -XPOST 'localhost:9200/_langdetect?pretty' -d 'This is a test'
{
"languages" : [
{
"language" : "en",
"probability" : 0.9999972283490304
}
]
}curl -XPOST 'localhost:9200/_langdetect?pretty' -d 'Das ist ein Test'
{
"languages" : [
{
"language" : "de",
"probability" : 0.9999985460514316
}
]
}curl -XPOST 'localhost:9200/_langdetect?pretty' -d 'Datt isse ne test'
{
"languages" : [
{
"language" : "no",
"probability" : 0.5714275763833249
},
{
"language" : "nl",
"probability" : 0.28571402563882925
},
{
"language" : "de",
"probability" : 0.14285660343967294
}
]
}There is a "short text" profile which is better to detect languages in a few words.
curl -XPOST 'localhost:9200/_langdetect?pretty&profile=short-text' -d 'Das ist ein Test'
{
"profile" : "/langdetect/short-text/",
"languages" : [ {
"language" : "de",
"probability" : 0.9999993070517024
} ]
}These settings can be used in elasticsearch.yml to modify language detection.
Use with caution. You don’t need to modify settings. This list is just for the sake of completeness. For successful modification of the model parameters, you should study the source code and be familiar with probabilistic matching using naive bayes with character n-gram. See also Ted Dunning, Statistical Identification of Language, 1994.
Name |
Description |
|
a comma-separated list of language codes such as (de,en,fr…) used to restrict (and speed up) the detection process |
|
a substitution code for a language code |
|
number of trials, affects CPU usage (default: 7) |
|
additional smoothing parameter, default: 0.5 |
|
the width of smoothing, default: 0.05 |
|
safeguard to break loop, default: 10000 |
|
default: 0.1 |
|
detection is terminated when normalized probability exceeds this threshold, default: 0.99999 |
|
default 10000 |