SmokeTestMultiNodeClientYamlTestSuiteIT/indices.stats/20_translog failed retaining too much translog

[DaveCTurner]

On repeated runs of org.elasticsearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT I hit the following failure:

org.elasticsearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT > test {yaml=indices.stats/20_translog/Translog retention without soft_deletes} FAILED
    java.lang.AssertionError: Failure at [indices.stats/20_translog:62]: field [indices.test.primaries.translog.size_in_bytes] is not less than or equal to [$creation_size]
    Expected: a value less than or equal to <110>
         but: <285> was greater than <110>

        Caused by:
        java.lang.AssertionError: field [indices.test.primaries.translog.size_in_bytes] is not less than or equal to [$creation_size]
        Expected: a value less than or equal to <110>
             but: <285> was greater than <110>

The REPRODUCE WITH line said:

REPRODUCE WITH: ./gradlew ':qa:smoke-test-multinode:integTestRunner' --tests "org.elasticsearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT" -Dtests.method="test {yaml=indices.stats/20_translog/Translog retention without soft_deletes}" -Dtests.seed=D3278D281C0378A6 -Dtests.security.manager=true -Dtests.jvms=4 -Dtests.locale=es-IC -Dtests.timezone=Kwajalein -Dcompiler.java=12 -Druntime.java=12

However this did not reproduce for me in ~30 retries.

I see a small number of similar failures from CI too, for instance https://gradle-enterprise.elastic.co/s/wi2oqndt4254w/console-log?task=:qa:smoke-test-multinode:integTestRunner.

There's not much information in the logs either:

  1> [2019-09-06T16:30:09,601][INFO ][o.e.s.SmokeTestMultiNodeClientYamlTestSuiteIT] [test] [yaml=indices.stats/20_translog/Translog retention without soft_deletes] before test
  1> [2019-09-06T16:30:10,653][INFO ][o.e.s.SmokeTestMultiNodeClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "_shards" : {
  1>         "total" : 2,
  1>         "successful" : 2,
  1>         "failed" : 0
  1>       },
  1>       "_all" : {
  1>         "primaries" : {
  1>           "translog" : {
  1>             "operations" : 1,
  1>             "size_in_bytes" : 285,
  1>             "uncommitted_operations" : 0,
  1>             "uncommitted_size_in_bytes" : 55,
  1>             "earliest_last_modified_age" : 0
  1>           }
  1>         },
  1>         "total" : {
  1>           "translog" : {
  1>             "operations" : 1,
  1>             "size_in_bytes" : 285,
  1>             "uncommitted_operations" : 0,
  1>             "uncommitted_size_in_bytes" : 55,
  1>             "earliest_last_modified_age" : 0
  1>           }
  1>         }
  1>       },
  1>       "indices" : {
  1>         "test" : {
  1>           "uuid" : "ZymXAFZgRFaFdk3a1DetOg",
  1>           "primaries" : {
  1>             "translog" : {
  1>               "operations" : 1,
  1>               "size_in_bytes" : 285,
  1>               "uncommitted_operations" : 0,
  1>               "uncommitted_size_in_bytes" : 55,
  1>               "earliest_last_modified_age" : 0
  1>             }
  1>           },
  1>           "total" : {
  1>             "translog" : {
  1>               "operations" : 1,
  1>               "size_in_bytes" : 285,
  1>               "uncommitted_operations" : 0,
  1>               "uncommitted_size_in_bytes" : 55,
  1>               "earliest_last_modified_age" : 0
  1>             }
  1>           }
  1>         }
  1>       }
  1>     },
  1>     "creation_size" : 110
  1>   }
  1> }]
  1> [2019-09-06T16:30:11,095][INFO ][o.e.s.SmokeTestMultiNodeClientYamlTestSuiteIT] [test] [yaml=indices.stats/20_translog/Translog retention without soft_deletes] after test

More logs attached here: failure-1567755613.tar.gz

[elasticmachine]

Pinging @elastic/es-distributed