[TEST] TokenBackwardsCompatibilityIT.testMixedCluster failing #31195

dliappis · 2018-06-08T11:05:41Z

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/1846/console

Reproduction steps:

./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v5.6.10-SNAPSHOT#twoThirdsUpgradedTestRunner -Dtests.seed=7C807A34A424F2C8 -Dtests.class=org.elasticsearch.upgrades.TokenBackwardsCompatibilityIT -Dtests.method="testMixedCluster" -Dtests.security.manager=true -Dtests.locale=pt-BR -Dtests.timezone=Africa/Bissau

Relevant output:

 > Throwable #1: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:26779], URI [_xpack/security/_authenticate], status line [HTTP/1.1 401 Unauthorized]
   > {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [_xpack/security/_authenticate]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [_xpack/security/_authenticate]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
   > 	at __randomizedtesting.SeedInfo.seed([7C807A34A424F2C8:71E12A5C778C8339]:0)
   > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:821)
   > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:182)
   > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:249)
   > 	at org.elasticsearch.upgrades.TokenBackwardsCompatibilityIT.assertTokenWorks(TokenBackwardsCompatibilityIT.java:175)
   > 	at org.elasticsearch.upgrades.TokenBackwardsCompatibilityIT.testMixedCluster(TokenBackwardsCompatibilityIT.java:103)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:26779], URI [_xpack/security/_authenticate], status line [HTTP/1.1 401 Unauthorized]
   > {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [_xpack/security/_authenticate]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [_xpack/security/_authenticate]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
   > 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:495)
   > 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:484)
   > 	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
   > 	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
   > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
   > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
   > 	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
   > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
   > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
   > 	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
   > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   > 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   > 	... 1 more
  1> [2018-06-08T01:16:41,322][INFO ][o.e.u.TokenBackwardsCompatibilityIT] [testGeneratingTokenInOldCluster]: before test
  1> [2018-06-08T01:16:41,349][INFO ][o.e.u.TokenBackwardsCompatibilityIT] [testGeneratingTokenInOldCluster]: after test
IGNOR/A 0.04s | TokenBackwardsCompatibilityIT.testGeneratingTokenInOldCluster
   > Assumption #1: this test should only run against the old cluster
  1> [2018-06-08T01:16:41,358][INFO ][o.e.u.TokenBackwardsCompatibilityIT] [testTokenWorksInMixedOrUpgradedCluster]: before test
  1> [2018-06-08T01:16:41,395][INFO ][o.e.u.TokenBackwardsCompatibilityIT] [testTokenWorksInMixedOrUpgradedCluster]: after test
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+6.x+intake/x-pack/qa/rolling-upgrade/with-system-key/build/testrun/v5.6.10-SNAPSHOT#twoThirdsUpgradedTestRunner/J0/temp/org.elasticsearch.upgrades.TokenBackwardsCompatibilityIT_7C807A34A424F2C8-001
  2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1605, maxMBSortInHeap=6.210594783023145, sim=RandomSimilarity(queryNorm=true): {}, locale=pt-BR, timezone=Africa/Bissau
  2> NOTE: Linux 3.16.0-4-amd64 amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=16,threads=1,free=350787576,total=514850816
  2> NOTE: All tests run in this JVM: [IndexAuditUpgradeIT, WatchBackwardsCompatibilityIT, TokenBackwardsCompatibilityIT]
Completed [3/5] in 2.23s, 4 tests, 1 error, 2 skipped <<< FAILURES!

There are no files in the temporary directory:

jenkins@worker-02a46c5b7f9fd3b1f:~/workspace/elastic+elasticsearch+6.x+intake/x-pack/qa/rolling-upgrade/with-system-key/build/testrun/v5.6.10-SNAPSHOT#twoThirdsUpgradedTestRunner/J0/temp/org.elasticsearch.upgrades.TokenBackwardsCompatibilityIT_7C807A34A424F2C8-001$ find .
.
./tempDir-001

The text was updated successfully, but these errors were encountered:

elasticmachine · 2018-06-08T11:09:14Z

Pinging @elastic/es-core-infra

dliappis · 2018-06-08T11:10:15Z

It appears that 7be13e5 didn't fix those.

dliappis · 2018-06-08T11:56:55Z

Interestingly enough a later job succeeded, however, the commit 5e1cef9 used in this failed job: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/1846/console already included the fix 7be13e5.

Relates #31195

nik9000 · 2018-06-08T15:04:54Z

These tests are tricky. You can run them a couple dozen times locally and they fail on Jenkins because it'll run them a hundred times. Or maybe I'm just doing something wrong locally. I was able to reproduce this failure and have a theory.

nik9000 · 2018-06-08T17:44:54Z

I pushed c5d38ad to silence this in the precise scenario that we've seen it fail. I have a fix that actually makes the test run and pass which I'll push soon.

nik9000 · 2018-06-08T18:03:11Z

I have a fix that actually makes the test run and pass which I'll push soon.

Not quite. I'm working with Jay to make something nicer.

nik9000 · 2018-06-11T15:10:40Z

Switching assignee to @jaymode because he has taken it over.

This commit fixes a backwards compatibility bug in the token service that causes token decoding to fail when there is a pre 6.0.0-beta2 node in the cluster. The token encoding is actually the culprit as a version check is missing around the serialization of the key hash bytes. This value was added in 6.0.0-beta2 and cannot be sent to nodes that do not know about this value. The version check has been added and the token service unit tests have been enhanced to randomly run with some 5.6.x nodes in the cluster service. Additionally, a small change was made to the way we check to see if the token metadata needs to be installed. Previously we would pass the metadata to the install method and check that the token metadata is null. This null check is now done prior to checking if the metadata can be installed. Relates elastic#30743 Closes elastic#31195

This commit fixes a backwards compatibility bug in the token service that causes token decoding to fail when there is a pre 6.0.0-beta2 node in the cluster. The token encoding is actually the culprit as a version check is missing around the serialization of the key hash bytes. This value was added in 6.0.0-beta2 and cannot be sent to nodes that do not know about this value. The version check has been added and the token service unit tests have been enhanced to randomly run with some 5.6.x nodes in the cluster service. Additionally, a small change was made to the way we check to see if the token metadata needs to be installed. Previously we would pass the metadata to the install method and check that the token metadata is null. This null check is now done prior to checking if the metadata can be installed. Relates #30743 Closes #31195

jaymode · 2018-06-13T18:40:50Z

Closed by #31254

dliappis added >test-failure Triaged test failures from CI >upgrade :Delivery/Build Build or test infrastructure labels Jun 8, 2018

dliappis assigned nik9000 Jun 8, 2018

dliappis added a commit that referenced this issue Jun 8, 2018

Mute TokenBackwardsCompatibilityIT.testMixedCluster

f4ea2e9

Relates #31195

nik9000 mentioned this issue Jun 8, 2018

Enable rolling upgrades from default distribution prior to 6.3.0 to default distribution post 6.3.0 #30731

Closed

9 tasks

nik9000 assigned jaymode and unassigned nik9000 Jun 11, 2018

jaymode mentioned this issue Jun 11, 2018

Security: fix token bwc with pre 6.0.0-beta2 #31254

Merged

jaymode closed this as completed Jun 13, 2018

gwbrown mentioned this issue Jan 11, 2019

[CI] TokenBackwardsCompatibilityIT failure in 6.x #37379

Closed

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TEST] TokenBackwardsCompatibilityIT.testMixedCluster failing #31195

[TEST] TokenBackwardsCompatibilityIT.testMixedCluster failing #31195

dliappis commented Jun 8, 2018

elasticmachine commented Jun 8, 2018

dliappis commented Jun 8, 2018

dliappis commented Jun 8, 2018 •

edited

Loading

nik9000 commented Jun 8, 2018

nik9000 commented Jun 8, 2018

nik9000 commented Jun 8, 2018

nik9000 commented Jun 11, 2018

jaymode commented Jun 13, 2018

[TEST] TokenBackwardsCompatibilityIT.testMixedCluster failing #31195

[TEST] TokenBackwardsCompatibilityIT.testMixedCluster failing #31195

Comments

dliappis commented Jun 8, 2018

elasticmachine commented Jun 8, 2018

dliappis commented Jun 8, 2018

dliappis commented Jun 8, 2018 • edited Loading

nik9000 commented Jun 8, 2018

nik9000 commented Jun 8, 2018

nik9000 commented Jun 8, 2018

nik9000 commented Jun 11, 2018

jaymode commented Jun 13, 2018

dliappis commented Jun 8, 2018 •

edited

Loading