"Fielddata is disabled on text fields by default." Admin Feature

[bcumber]

I get the following exception from Elasticpress when I load the "Pages" in the admin screen:

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [post_title.sortable] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "devboxbennet-1",
                "node": "VbXshdquSwqnaeSD02K7vQ",
                "reason": {
                    "type": "illegal_argument_exception",
                    "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [post_title.sortable] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
                }
            }
        ],
        "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [post_title.sortable] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
        }
    },
    "status": 400
}

If I understand this error correctly (and I'm new to Elasticsearch, Wordpress, and ElasticPress so I could very well be misunderstanding) version 5.x of Elasticsearch does not index text fields for sorting by default. (This does seem to differ from previous versions 5.x vs 2.4)

The 5.x documentation provides a way to enable fielddata for text fields. I tried to do this:

PUT devboxbennet-1/_mapping/post
{
  "properties": {
    "post_title": { 
      "type":     "text",
      "fields": {
        "post_title": {
          "type": "text",
          "analyzer": "standard"
        },
        "raw": {
          "type": "keyword"
        },
        "sortable": {
          "type": "text",
          "analyzer": "ewp_lowercase",
          "fielddata": true
        }
      }
    }
  }
}

With this setting active, I don't see any errors on the elasticsearch side of things, but Wordpress errors out with a PHP Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/wp-includes/kses.php on line 1430. I would assume this is a separate error, on my side of things, but its worth mentioning. Either way, I can't seem to get the admin feature to work.

So my first question is: how can I get the admin feature to work by changing elasticsearch/wordpress settings? And my second question is: Is this how ElasticPress should be doing things?

[Ritesh-patel]

Hi @bcumber

Thanks for bringing this to our notice. Which version of ElasticPress you are using? We have pushed an update(v2.1.2) 2 weeks before for Elasticsearch 5.x related issues.

Regards,
Ritesh

[bcumber]

Hi @Ritesh-patel

I'm using ElasticPress v2.1.2, Wordpress v4.6.1, and Elasticsearch v5.0.1.

[Ritesh-patel]

Hi @bcumber

We are looking into this and will let you know once we have fix for it.

Regards,
Ritesh

[Ritesh-patel]

@tlovett1 I am able reproduce this one locally and I think we need to update mapping. What do you suggests?

Regards,
Ritesh

[bcumber]

@Ritesh-patel Out of curiosity, are you able to reproduce both the first Fielddata is disabled... error and the second Max execution time error? Or are you only seeing the first one?

[Ritesh-patel]

@bcumber I can only see the first one. If I set "fielddata": true and update mappings, I am not able reproduce the second one.

[bcumber]

@Ritesh-patel I'm not surprised the error is only on my end; my WordPress installation is very large. However, elasticsearch does warn that updating fielddata to be true for those text fields increases memory usage dramatically. Would it be possible to implement their preferred approach instead? https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#before-enabling-fielddata

[bcumber]

I tested changing the mapping of the sortable field to be keyword instead of text. It seems to work for me. However, that also required me to remove the analyzer property that was set. I can submit a PR if you'd like, but I'm honestly not sure what ramifications removing the analyzer property may have.

Also, the mapping needs to be changed for each instance of the sortable field as described in https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#field-conflicts

[Ritesh-patel]

Hi @bcumber

Yes, submitting a PR will be great, we can discuss further in PR.

[tlovett1]

Okay, we thought a lot about this one. Elasticsearch gives us absolutely no work around the current 5.1.x version of Elasticsearch. We could create entirely new sortable fields for a number of different post fields but that would be tedious and require a lot of architectural changes.

There's good news! Analyzers are supported by the keyword field in master now. We will wait for this to be released as a major ES version then support that in a mapping.

For now everyone has two options:

1. Stick with ES 2.4.x until 5.2.
2. Don't use field sorting.

[allan23]

@tlovett1 We've run into this issue with a client running ES 5.2.x. It may be worth revisiting as more and more users are upgrading from 2.4.

[tlovett1]

@allan23 can you do some research on a solution?

[jbejar]

I noticed that in the Elasticsearch Python DSL you can sort Text Fields, but just have to append ".keyword" to the name of the field. https://www.elastic.co/guide/en/elasticsearch/guide/current/multi-fields.html was helpful

[allan23]

@tlovett1 after reviewing the issue that the client experienced, I found that it was due to custom user indexing and failure to update their mapping upon ES upgrade. Closing this out.