[APM] index mapping update not working properly

[philippkahr]

Kibana version:
7.3.0 & 7.4.0
Elasticsearch version:
7.3.0 & 7.4.0
Server OS version:
Centos 7
Browser version:
Chrome Dev 79.0.3941.4
Browser OS version:
MacOS
Original install method (e.g. download page, yum, from source, etc.):
yum repo
Describe the bug:
I am using the apm-server 7.3.0 and 7.4.0 with the java apm-agent. There is this issue that the index mapping does not get updated when hitting the refresh index pattern for apm-*.
Steps to reproduce:

1. apm-server setup
2. Get some data from apm-server (maybe something like a website that adds some cookie values)
3. See that warning bang in the Kibana Discovery tab
4. Hit the index refresh button in the Kibana index pattern
5. nothing happens

Expected behavior:
Just refresh the fields and make them aggregate and searchable.

Any additional context:
This originated at my question here, which is way more detailed: https://discuss.elastic.co/t/apm-index-not-refreshing/195884

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

@Bargs shouldn't this be labeled with Team:KibanaApp? Apart from being an APM user, this looks like an issue with index patterns or the Index Management app . I'm not sure how the APM app can fix this?

[sorenlouv]

@philippkahr I've reproduced your issue, and now understand the confusion.

Afaict you are inserting fields like http.request.cookies.LASTUSERNAMEINPUT and are expecting it to become searchable after refreshing the index patterns.
However, since http.request is mapped as dynamic: false descendant fields will not get indexed in elasticsearch. Refreshing the index patterns won't change this.

The UI in Discover can be confusing since a warning give the user the impression that clicking the refresh button will solve the problem. It won't.

Solution A
Index your additional data as labels. These are dynamic so everything will become indexed and be searchable and aggregatable.

Solution B
Update the mapping manually. I see you already suggested that here. I'm curious why that didn't work?

PUT apm*/_mapping/
{
  "properties": {
      "http.request.cookies.LASTSKIN": {
        "type": "keyword",
        "index": true
      },
      "http.request.cookies.LASTUSERNAME": {
        "type": "keyword",
        "index": true
      }
    }
}

[philippkahr]

Hi @sqren

thanks for looking into my matter! Regarding the PUT apm*... that worked without any issue, so there is no trouble there. I just opened up this issue, cause I wasn't quite sure if this is intended behavior. Maybe the intended behavior is that the Kibana warning bang messages states field is dynamically mapped, change it to true in the fields.yml. However, I do not like to touch things like a fields.yml since I would then have to update it everytime I install an update. Sure I can template that with ansible, but I would have to ensure that my template fits the version.

The idea with labels is nice if you have access to the source code and can modify it properly as I saw that you need to add it as a dependency to your application and call the labels API https://www.elastic.co/guide/en/apm/agent/java/1.x/public-api.html#api-transaction-add-tag. However, I am more a DevOps / SysOps for bought products, like Jira, Bitbucket, Confluence. We are using APM basically to ensure that a user's claim about something being slow is valid or not.

I am not sure if we should close this issue and generate a new issue with a title like better index mapping warning bang or keep this one and hook up the right team to it :).

[sorenlouv]

The idea with labels is nice if you have access to the source code and can modify it properly as I saw that you need to add it as a dependency to your application and call the labels API

How did you add http.request.cookies to the ingested docs? Can't you do the same but add it as labels.cookies?

I am not sure if we should close this issue and generate a new issue with a title like better index mapping warning bang or keep this one and hook up the right team to it :).

It sounds like there are two different problems:

1. Discover shows a misleading warning. That deserves a separate issue
2. It is difficult to search for custom fields in APM docs

[philippkahr]

How did you add http.request.cookies to the ingested docs? Can't you do the same but add it as labels.cookies?

to be honest, I have no idea where those fields originate from. The java application is running with a tomcat and I have added the apm jar as an -javaagent to the CATALINA_OPTS, as shown here https://www.elastic.co/guide/en/apm/agent/java/master/setup-javaagent.html#setup-tomcat-unix , so I guess it must be coming from there? Maybe @felixbarny could explain this. I have apm-server and the java agent up to date. I am using a basic apm-server config, just like it is out of the box.

1. Discover shows a misleading warning. That deserves a separate issue

I will open one if that is ok with you?

2. It is difficult to search for custom fields in APM docs

That seems to be outta my league. I would gladly help, but I do not know how.

[sorenlouv]

I will open one if that is ok with you?

Please do! :)

That seems to be outta my league. I would gladly help, but I do not know how.

Totally fair. Didn't mean to imply you should do it. For now I'll wait for @felixbarny to chime in on this.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[Bargs]

@sqren my bad, I saw APM in the title and pulled the trigger without thinking 😬

[philippkahr]

@felixbarny here you go for a bit more information.

APM Server config

apm-server:
  host: "0.0.0.0:8200"
  rum:
    enabled: true
output.elasticsearch:
  hosts: ["elastic1:9200"]
  protocol: "https"
  username: "fancyuser"
  password: "fancypassword"
  ssl.certificate_authorities: ["/etc/ssl/certs/cachain.pem"]
  pipelines:
  - pipeline: "apm_match_user_id"
monitoring:
  enabled: true
  elasticsearch:
    username: fancyuser
    password: fancypassword
tags: ["itsm-test"]

Ingest pipeline apm_match_user_id

PUT _ingest/pipeline/apm_match_user_id
{
  "description" : "map weird user.id to user.id",
  "processors" : [
    {
      "set": {
      "field": "user.id",
      "value": "{{http.request.headers.Vm_user.0}}",
      "ignore_failure": true
      }
    }
  ]
}

Screenshots

I had to blackbox some stuff, but I guess it is still better than nothing.

[felixbarny]

Not sure what the actual question is. I assume it's this?

How did you add http.request.cookies to the ingested docs?

Our agents add them automatically by default (controlled by the capture_headers config option).

[philippkahr]

Hi @felixbarny,

yes, that was the primary question. Maybe we should change http.request from dynamic: false to true if capture_headers are enabled? I am not quite sure, how this should play out in the end. What is your opinion on that matter?

[sorenlouv]

@philippkahr I've been made aware of an experimental feature append_fields that might solve your problem.

Adding the following to apm-server.yml will enable dynamic indexing for http.request.cookies and it should work across updates:

setup.template.enabled: true
setup.template.overwrite: true
setup.template.append_fields:
  - name: http.request.cookies
    type: object
    dynamic: true

[philippkahr]

@sqren I deleted my index mapping and appended that to my 7.3.0 and 7.4.0 config and it worked! Thanks for the help. Maybe that should be pointed out in the documentation somewhere?

Should we close this issue now?

[sorenlouv]

I deleted my index mapping and appended that to my 7.3.0 and 7.4.0 config and it worked!

Great to hear!

Maybe that should be pointed out in the documentation somewhere?

I agree. @bmorelli25 can you figure out if this makes sense to document somewhere?
tldr: The Java agent (and possibly other agents) ingests documents with cookie values located in http.request.cookies. Since http.request has disabled dynamic indexing, and http.request.cookies is not declared in the mappings the values in http.request.cookies are not indexed and thus not searchable. One solution is outlined in #48674 (comment)

[bmorelli25]

Interesting. I could probably add something to our troubleshooting doc with a quick walkthrough. Opened a new issue to track on the docs board: #49070