Encode API key as base64 in common code #18945

ycombinator · 2020-06-03T16:17:40Z

What does this PR do?

Encodes the API key supplied to the Elasticsearch client with base64 encoding right before setting it in the Authorization request header.

Why is it important?

The Elasticsearch client is shared by the Elasticsearch output as well as the Elasticsearch monitoring reporter. Rather than base64-encoding the API key in each of these places, it is better to base64-encode it within the client code itself.

In fact, prior to this PR, we were base64-encoding the key in the Elasticsearch output but not in the Elasticsearch monitoring reporter.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Fixes monitoring.elasticsearch.api_key needs to be base64 encoded #18939

elasticmachine · 2020-06-03T16:21:30Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [ycombinator commented: jenkins, run tests please]
Start Time: 2020-06-08T16:53:48.520+0000
Duration: 84 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9317
Skipped	1574
Total	10891

Steps errors

Expand to view the steps failures

Name: Make -C generator/_templates/beat test
- Description: make -C generator/_templates/beat test
- Duration: 7 min 34 sec
- Start Time: 2020-06-08T18:08:01.825+0000
- log

ycombinator · 2020-06-03T16:55:00Z

From my checking, the APM Server code is not directly calling eslegclient.NewConnection anywhere so they should be good with this change. Nevertheless, it would be good for someone from @elastic/apm-server to confirm. Thank you!

elasticmachine · 2020-06-03T16:55:28Z

Pinging @elastic/integrations-services (Team:Services)

adriansr

thanks for the quick fix!

simitt · 2020-06-03T17:43:12Z

Thanks for the ping, the APM Server will not be affected by this change.

urso · 2020-06-04T10:01:56Z

libbeat/esleg/eslegclient/connection.go

@@ -436,7 +437,8 @@ func (conn *Connection) execHTTPRequest(req *http.Request) (int, []byte, error)
 	}

 	if conn.APIKey != "" {
-		req.Header.Add("Authorization", "ApiKey "+conn.APIKey)
+		encoded := base64.StdEncoding.EncodeToString([]byte(conn.APIKey))
+		req.Header.Add("Authorization", "ApiKey "+encoded)


would it make sense to cache the result in a private variable like conn.encodedAPIKey ?

Yes, it would. No need to encode it each time we make a request to ES. Thanks for the suggestion, will implement!

Done in c3e5b43.

urso · 2020-06-04T13:01:35Z

libbeat/esleg/eslegclient/connection.go

@@ -75,6 +76,8 @@ type ConnectionSettings struct {

 	Timeout         time.Duration
 	IdleConnTimeout time.Duration
+
+	encodedAPIKey string // Base64-encoded API key


Maybe we want to move the field to Connection ?

Sure, I guess because you want to keep ConnectionSettings just as the public settings API?

Done in dc4e921.

Sure, I guess because you want to keep ConnectionSettings just as the public settings API?

yep.

andrewkroh · 2020-06-04T22:03:27Z

On a related not I think our Beat docs are not clear on the expected format for the api_key value. I think the api_key reference docs should mention that the value is the "id and api_key joined by a colon". And this example is not correct since the value should be of the format '<id>:<api_key>'. Or am I misunderstanding how these are used?

From elasticsearch output:

ycombinator · 2020-06-04T22:05:22Z

Thanks for pointing that out, @andrewkroh. I agree that the example value in the doc is incorrect (it looks already base64-encoded). I will add the doc update to this PR.

ycombinator · 2020-06-04T23:46:08Z

Documentation for api_key setting updated in 22a25cc6be1c8f0c955f7a61a5c3f3503138628a. I deliberately used the same values and similar language from https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html. @andrewkroh @dedemorton would you mind taking a look? Thanks!

andrewkroh · 2020-06-04T23:47:53Z

libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc

@@ -37,13 +37,14 @@ output.elasticsearch:
  password: "{pwd}"
 ------------------------------------------------------------------------------

-To use an API key to connect to {es}, use `api_key`.
+To use an API key to connect to {es}, use `api_key`. The value must be the ID of
+the API key and the API key joined by a colon.


This info would be good down on line 137 with the api_key.

Done in 42d288616.

urso · 2020-06-05T10:56:33Z

libbeat/esleg/eslegclient/connection.go

-	if conn.APIKey != "" {
-		req.Header.Add("Authorization", "ApiKey "+conn.APIKey)
+	if conn.encodedAPIKey != "" {
+		req.Header.Add("Authorization", "ApiKey "+conn.encodedAPIKey)


nit: We still have an unnecessary alloc + copy here. If we initialize encodedAPIKey = "ApiKey " + ..., then we can just pass it to Header as is.

Changed in 47c9aacaaa04e9cc1e304d517144386a77483e02.

* Encode API key as base64 in common code * Adding comment on API key field * Adding CHANGELOG entries * Adding test * Base64-encode API key in constructor * Move encodedAPIKey field to Connection * Update doc on `api_key` setting value * Adding API key format to setting section * Compute entire API key auth header value up front

ycombinator marked this pull request as ready for review June 3, 2020 16:53

ycombinator requested review from adriansr and urso June 3, 2020 16:54

ycombinator added bug Team:Services (Deprecated) Label for the former Integrations-Services team v7.8.1 v7.9.0 v8.0.0 labels Jun 3, 2020

ycombinator added [zube]: In Review needs_backport PR is waiting to be backported to other branches. labels Jun 3, 2020

adriansr approved these changes Jun 3, 2020

View reviewed changes

urso reviewed Jun 4, 2020

View reviewed changes

urso approved these changes Jun 4, 2020

View reviewed changes

urso reviewed Jun 4, 2020

View reviewed changes

andrewkroh reviewed Jun 4, 2020

View reviewed changes

urso reviewed Jun 5, 2020

View reviewed changes

ycombinator added 7 commits June 5, 2020 10:52

Encode API key as base64 in common code

559eb3e

Adding comment on API key field

afa3e91

Adding CHANGELOG entries

218145f

Adding test

4afdf11

Base64-encode API key in constructor

36a5b81

Move encodedAPIKey field to Connection

97806e0

Update doc on api_key setting value

dce0e2a

ycombinator added 2 commits June 5, 2020 10:52

Adding API key format to setting section

2eb7a5f

Compute entire API key auth header value up front

9982b8d

ycombinator force-pushed the lb-mon-es-b64 branch from 47c9aac to 9982b8d Compare June 5, 2020 17:53

urso approved these changes Jun 8, 2020

View reviewed changes

ycombinator merged commit 3135985 into elastic:master Jun 9, 2020

ycombinator deleted the lb-mon-es-b64 branch June 9, 2020 08:42

zube bot added [zube]: Done and removed [zube]: In Review labels Jun 9, 2020

ycombinator mentioned this pull request Jun 9, 2020

[7.x] Encode API key as base64 in common code (#18945) #19064

Merged

ycombinator mentioned this pull request Jun 9, 2020

[7.8] Encode API key as base64 in common code (#18945) #19065

Merged

andresrc removed the [zube]: Done label Jun 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encode API key as base64 in common code #18945

Encode API key as base64 in common code #18945

ycombinator commented Jun 3, 2020 •

edited

Loading

elasticmachine commented Jun 3, 2020 •

edited

Loading

Build stats

Test stats 🧪

ycombinator commented Jun 3, 2020

elasticmachine commented Jun 3, 2020

adriansr left a comment

simitt commented Jun 3, 2020

urso Jun 4, 2020

ycombinator Jun 4, 2020

ycombinator Jun 4, 2020

urso Jun 4, 2020

ycombinator Jun 4, 2020

ycombinator Jun 4, 2020

urso Jun 5, 2020

andrewkroh commented Jun 4, 2020 •

edited

Loading

ycombinator commented Jun 4, 2020

ycombinator commented Jun 4, 2020 •

edited

Loading

andrewkroh Jun 4, 2020

ycombinator Jun 5, 2020

urso Jun 5, 2020

ycombinator Jun 5, 2020

Encode API key as base64 in common code #18945

Encode API key as base64 in common code #18945

Conversation

ycombinator commented Jun 3, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Jun 3, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

ycombinator commented Jun 3, 2020

elasticmachine commented Jun 3, 2020

adriansr left a comment

Choose a reason for hiding this comment

simitt commented Jun 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Jun 4, 2020 • edited Loading

ycombinator commented Jun 4, 2020

ycombinator commented Jun 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator commented Jun 3, 2020 •

edited

Loading

elasticmachine commented Jun 3, 2020 •

edited

Loading

andrewkroh commented Jun 4, 2020 •

edited

Loading

ycombinator commented Jun 4, 2020 •

edited

Loading