Allow specific characters in token_chars of edge ngram tokenizer in addition to classes

[edudar]

I bumped into this while implementing autocomplete/typeahead functionality with highlighting.

My index settings are:

  analysis:
    tokenizer:
      autocomplete_highlight:
        type: edgeNGram
        min_gram: 1
        max_gram: 15
        token_chars: ["letter", "digit"]
    filter:
      autocomplete_ngram:
        type: edgeNGram
        min_gram: 1
        max_gram: 15
    analyzer:
      autocomplete_index:
        type: custom
        tokenizer: icu_tokenizer
        filter: [standard, icu_normalizer, icu_folding, autocomplete_ngram]
      autocomplete_search:
        type: custom
        tokenizer: icu_tokenizer
        filter: [standard, icu_normalizer, icu_folding, stop]
      autocomplete_highlight:
        type: custom
        tokenizer: autocomplete_highlight
        filter: [standard, icu_normalizer, icu_folding]

I do the search by autocomplete field and highlight on autocomplete_highlight. Everything works fine until I meet _ in a search query. icu_tokenizer keeps it while autocomplete_highlight tokenizer removes as it keeps letters and digits only. Here I can't keep _ only but full punctuation class instead that comes with a whole load of additional symbols that I don't need and they have to go.

I would be helpful to be able to specify exact characters to keep like _.

At the moment I've implemented char_filter that replaces _ with - but that's suboptimal as _ is considered a part of words (same as in icu_tokenizer) and is expected to match rather than being ignored.

[andrewthad]

I would like this as well, except that I'm need it for the ngram tokenizer, not the edge ngram tokenizer. Basically, I have a bunch of logs that end up in elasticsearch, and the only character need to be sure will break up tokens is a comma. Anything else is fair game for inclusion.

[DanielBaird]

I also need to specify an individual character in my edgeNGram filter's token_chars. In my situation, I need a hyphen - to be counted as an in-token character, without including brackets and whatnot.

[romseygeek]

cc @elastic/es-search-aggs

[jcox]

Not being able to specify an individual character like '_' in the edge_ngram tokenizer's token_chars list is particularly painful because the 'standard' ES tokenizer uses the UNIX Text Segmentation Algorithm, which does include UNDERSCORE.

It would be nice to be able to index with edge_ngram & search with the standard tokenizer. Currently, you can't if you have strings like "foo_bar".

[borstelmahlsdorf]

I need also a way to specify a single char as token_chars, in my case a "-"

[Piersen]

In my case, being able to specify token chars as any character except for specified delimiters would be useful

[poef124]

I wanted to know if there is a workaround to achieve this? I want to include certain punctuations like . : but not all the others

[kumropotash1]

Plus one from me.
I want a hyphen - to be considered as a char.

[seal256]

Need this feature

[zion3mx]

i need this feature with letter and digit.

[binkymilk]

I need this feature too for the ngram tokenizer! For both _ and -.