The date_index_name processor calculates wrong week if index_name_format is 'yyyy-w'

[yoogie]

Elasticsearch version (bin/elasticsearch --version):
Version: 7.0.0, Build: default/zip/b7e28a7/2019-04-05T22:55:32.697037Z, JVM: 1.8.0_121

Plugins installed: []

JVM version (java -version):
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

OS version (uname -a if on a Unix-like system):
win7

Description of the problem including expected versus actual behavior:
Indexing a document using the date_index_name processor with "date_rounding" : "d" and "index_name_format": "yyyy-w" indexes the time 2017-12-04T17:00:58.580 in a index named x-2017-1 while expected index name would be x-2017-49

Steps to reproduce:

PUT _ingest/pipeline/testingestpipeline
{
  "description": "Pipeline for routing data to specific index by createtime",
  "processors": [
    {
      "date": {
        "field": "createdTime",
        "formats": [
          "date_time","date_time_no_millis", "date_hour_minute_second_millis", "date_hour_minute_second_fraction"
        ]
      },
      "date_index_name": {
        "field": "@timestamp",
        "date_rounding": "d",
        "index_name_prefix": "x-",
        "index_name_format": "yyyy-w"
      }
    }
  ]
}

DELETE x-*

PUT x-test/_doc/1?pipeline=testingestpipeline
{
  "createdTime": "2017-12-04T17:00:58.580"
}

Index name -> x-2017-1

If using index_name_format including month and day, the week gets correct

PUT _ingest/pipeline/workingingestpipeline
{
  "description": "Pipeline for routing data to specific index by createtime",
  "processors": [
    {
      "date": {
        "field": "createdTime",
        "formats": [
          "date_time","date_time_no_millis", "date_hour_minute_second_millis", "date_hour_minute_second_fraction"
        ]
      },
      "date_index_name": {
        "field": "@timestamp",
        "date_rounding": "d",
        "index_name_prefix": "x-",
        "index_name_format": "yyyy-MM-dd-w"
      }
    }
  ]
}

DELETE x-*

PUT x-test/_doc/1?pipeline=workingingestpipeline
{
  "createdTime": "2017-12-04T17:00:58.580"
}

Index name -> x-2017-12-04-49

[elasticmachine]

Pinging @elastic/es-core-features

[jakelandis]

@yoogie - thanks for reporting this.

This appears to be an issue with the date math index name resolution not the processor, and does appear to be a regression in 7.0.0. This is likely related to an internal change to start using Java (8) Time instead of Joda Time for internal time resolution.

I can reproduce this without the ingest node processor with the following date math index name:

<{2019-18||/d{yyyy-w|UTC}}>

Creating an index using this expression (using the encoded form):

PUT /%3C%7B2019-18%7C%7C%2Fd%7Byyyy-w%7CUTC%7D%7D%3E

on 6.5 (correct):

"index" : "2019-18"

on 7.0 (incorrect):

"index" : "2019-1"

cc @rjernst

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

I think the week portion of the format needs to be ww to get 2 digits. So not a bug, but something that we missed in the upgrade warnings to java 8 time.

[yoogie]

I don't see that changing to 'ww' would help. It does change the number of digits, but it still outputs 01 instead of 1, but expected would is still 49 for the example I posted at the beginning.

[pgomulka]

I think this is a bug, but it also was existing in 6.7 but not always visible.

For both 6.7 and 7.0 yyyy-w returns wrong results. For instance:
<{2017-48||/d{yyyy-w|UTC}}>
http://localhost:9200/%3C%7B2017-48%7C%7C%2Fd%7Byyyy-w%7CUTC%7D%7D%3E`

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "2016-48"
}

in the example run against 7.0 yyyy-w means we are mixing w week of week based year with y year of era. I don't think this is ok. It should be YYYY-w where Y is week based year. It does not work though. It throws an exception for me:

java.lang.IllegalArgumentException: temporal accessor [{WeekOfWeekBasedYear[WeekFields[SUNDAY,1]]=49, WeekBasedYear[WeekFields[SUNDAY,1]]=2017},ISO,Z] cannot be converted to zoned date time

If run against 6.7 yyyy-w also not works as y means year (can be negative for years before beginning of era) and w means week of weekyear.
So @yoogie example would probably also not work in 6.7.
It works for xxxx-w ... (x means weekyear in 6.7)

[pgomulka]

I think this might have been introduced by https://github.com/elastic/elasticsearch/pull/39568/files#diff-37eb9be6db62fad60f8c3d35b77e91d0R153