Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make yaml.Unmarshal strict #271

Merged
merged 2 commits into from
Aug 2, 2022
Merged

Conversation

ronensc
Copy link
Collaborator

@ronensc ronensc commented Jul 28, 2022

Using yaml.UnmarshalStrict() will prevent us from making typos in our yamls without being noticed.

@ronensc ronensc requested review from eranra, jotak and KalmanMeth July 28, 2022 11:19
@codecov-commenter
Copy link

codecov-commenter commented Jul 28, 2022

Codecov Report

Merging #271 (59612db) into main (6c1dae1) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main     #271   +/-   ##
=======================================
  Coverage   66.83%   66.83%           
=======================================
  Files          73       73           
  Lines        4149     4149           
=======================================
  Hits         2773     2773           
  Misses       1197     1197           
  Partials      179      179           
Flag Coverage Δ
unittests 66.83% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/confgen/config.go 33.33% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us.

@@ -37,6 +37,6 @@ func TestDuration_MarshalYAML(t *testing.T) {
func TestDuration_UnmarshalYAML(t *testing.T) {
expectedMsg := testMessage{Elapsed: Duration{2 * time.Hour}}
var actualMsg testMessage
require.NoError(t, yaml.Unmarshal([]byte("elapsed: 2h\n"), &actualMsg))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ronensc What is your general rule when you want to use UnmarshalStrict as opposed to Unmarshal?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assume we have a type like this:

type Person struct {
	Name string `yaml:"name"`
	Age  int    `yaml:"age"`
}

and data that we want to parse:

person:
  name: John
  Age: 42

Notice that the age field is misspelled (it should be age with lowercase a).

Using unstrict yaml.Unmarshal() won't raise an error on this. It will simply ignore the unrecognized field Age and set the default value for the missing field age.

I think in our use-case, where we parse config files, we want to make sure that all fields are recognized and emit an error otherwise.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So aren't there numerous places in the code where we Marshal the config data where we should also use UnmarshalStrict ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I replaced all the yaml.Unmarshal() invocations that I found (I found only 3).
Please let me know if I missed something.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are also lots of json.Unmarshal() calls.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, the json library doesn't have a strict version of Unmarshal() like the yaml has.
But, we can get the strict behavior with json.NewDecoder and DisallowUnknownFields() as suggested here:
https://groups.google.com/g/golang-nuts/c/MJuJDte5YHg/m/JMs0S-4uAQAJ

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't some of those be changed to to yaml.UnmarshalStrict?

Copy link
Member

@jotak jotak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm
Indeed I remember having seen hostname and hostName in some places and was confused about that

@ronensc
Copy link
Collaborator Author

ronensc commented Aug 2, 2022

Note: This PR deals only with confgen. That is, flowlogs-pipeline would start normally even if its config has unknown fields. I created an issue for it and will solve it in a later PR. #274

@ronensc ronensc force-pushed the yaml-unmarshal-strict branch from 90e775f to 3f5d194 Compare August 2, 2022 06:24
@ronensc ronensc merged commit 0184936 into netobserv:main Aug 2, 2022
@ronensc ronensc deleted the yaml-unmarshal-strict branch August 2, 2022 07:59
@jotak jotak added the breaking-change This pull request has breaking changes. They should be described in PR description. label Aug 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change This pull request has breaking changes. They should be described in PR description.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants