x-pack/filebeat/input/httpjson: add hash template function #31630

efd6 · 2022-05-16T04:32:45Z

What does this PR do?

This adds template helper functions for hashing a set of strings, output as hex or base64.

Why is it important?

This feature is needed for authentication with some APIs.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [filebeat][httpjson] Add support for setting hash in request header. #31613

Use cases

Screenshots

Logs

elasticmachine · 2022-05-16T04:32:48Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

CHANGELOG.next.asciidoc

elasticmachine · 2022-05-16T05:39:57Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-05-23T23:34:41.773+0000
Duration: 76 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2120
Skipped	166
Total	2286

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

efd6 · 2022-05-16T06:17:13Z

x-pack/filebeat/input/httpjson/value_tpl.go

@@ -62,13 +62,15 @@ func (t *valueTpl) Unpack(in string) error {
 			"mul":                 mul,
 			"div":                 div,
 			"hmac":                hmacStringHex,
+			"hash":                hashStringHex,
 			"base64Encode":        base64Encode,
 			"base64EncodeNoPad":   base64EncodeNoPad,
 			"base64Decode":        base64Decode,
 			"base64DecodeNoPad":   base64DecodeNoPad,
 			"join":                join,
 			"sprintf":             fmt.Sprintf,


I noticed this exists; it is already provided by the text/template packages global template functions via printf.

andrewkroh · 2022-05-16T18:51:25Z

x-pack/filebeat/input/httpjson/value_tpl.go

+		n += len(d)
+	}
+	if n == 0 {
+		return nil


Why is there special handling for empty values? I would have expected that these result in the hash value for 0 length input / empty string (like the same values listed in the test vectors for 0 len values).

The current behaviour for hmac treats "" specially (I think this is wrong, but I was keeping that behaviour). I can drop that if you want.

I think we should remove the special handling for hash.

I just looked at the hmac implementation and don't know why the special handling was added there. @marc-gr would there be a reason to handle empty values with special behavior?

The only reason I can remember of is since this will be used from templates an empty value would come as <no value>, but now that I try to think about anything else I am leaning more towards a bad outcome of trying to be too defensive. I'd rather change the way we deal with it in the other functions.

mergify · 2022-05-20T10:24:19Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httpjson-hash upstream/httpjson-hash
git merge upstream/main
git push upstream httpjson-hash

efd6 · 2022-05-23T00:23:28Z

Taking the opportunity to remove nolint directives for gosec linting that is in the process of being removed.

CHANGELOG.next.asciidoc

marc-gr · 2022-05-23T11:44:06Z

x-pack/filebeat/input/httpjson/value_tpl.go

+		n += len(d)
+	}
+	if n == 0 {
+		return nil


The only reason I can remember of is since this will be used from templates an empty value would come as <no value>, but now that I try to think about anything else I am leaning more towards a bad outcome of trying to be too defensive. I'd rather change the way we deal with it in the other functions.

mergify · 2022-05-23T13:41:52Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httpjson-hash upstream/httpjson-hash
git merge upstream/main
git push upstream httpjson-hash

CHANGELOG.next.asciidoc

efd6 added enhancement Filebeat Filebeat Team:Security-External Integrations backport-skip Skip notification from the automated backport with mergify 8.3-candidate labels May 16, 2022

efd6 requested a review from a team as a code owner May 16, 2022 04:32

efd6 self-assigned this May 16, 2022

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 16, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 16, 2022

efd6 commented May 16, 2022

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

efd6 commented May 16, 2022

View reviewed changes

andrewkroh reviewed May 16, 2022

View reviewed changes

efd6 and others added 3 commits May 23, 2022 09:36

x-pack/filebeat/input/httpjson: add hash template function

2df0b8c

Update CHANGELOG.next.asciidoc

c28c513

don't special-case empty

832c9b7

efd6 force-pushed the httpjson-hash branch from a90eca1 to 832c9b7 Compare May 23, 2022 00:06

marc-gr reviewed May 23, 2022

View reviewed changes

andrewkroh approved these changes May 23, 2022

View reviewed changes

Update CHANGELOG.next.asciidoc

47aadb6

efd6 commented May 23, 2022

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

efd6 added 2 commits May 24, 2022 09:02

Update CHANGELOG.next.asciidoc

bb1a22e

Merge branch 'main' into httpjson-hash

3c758ae

efd6 merged commit 6ae8856 into elastic:main May 24, 2022

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

x-pack/filebeat/input/httpjson: add hash template function (#31630)

5a32330

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x-pack/filebeat/input/httpjson: add hash template function #31630

x-pack/filebeat/input/httpjson: add hash template function #31630

efd6 commented May 16, 2022

elasticmachine commented May 16, 2022

elasticmachine commented May 16, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

efd6 May 16, 2022

andrewkroh May 16, 2022 •

edited

Loading

efd6 May 16, 2022 •

edited

Loading

andrewkroh May 20, 2022

marc-gr May 23, 2022 •

edited

Loading

mergify bot commented May 20, 2022

efd6 commented May 23, 2022

marc-gr May 23, 2022 •

edited

Loading

mergify bot commented May 23, 2022

x-pack/filebeat/input/httpjson: add hash template function #31630

x-pack/filebeat/input/httpjson: add hash template function #31630

Conversation

efd6 commented May 16, 2022

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented May 16, 2022

elasticmachine commented May 16, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

efd6 May 16, 2022

Choose a reason for hiding this comment

andrewkroh May 16, 2022 • edited Loading

Choose a reason for hiding this comment

efd6 May 16, 2022 • edited Loading

Choose a reason for hiding this comment

andrewkroh May 20, 2022

Choose a reason for hiding this comment

marc-gr May 23, 2022 • edited Loading

Choose a reason for hiding this comment

mergify bot commented May 20, 2022

efd6 commented May 23, 2022

marc-gr May 23, 2022 • edited Loading

Choose a reason for hiding this comment

mergify bot commented May 23, 2022

elasticmachine commented May 16, 2022 •

edited by jenkins-beats-ci bot

Loading

andrewkroh May 16, 2022 •

edited

Loading

efd6 May 16, 2022 •

edited

Loading

marc-gr May 23, 2022 •

edited

Loading

marc-gr May 23, 2022 •

edited

Loading