Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avoid type loss with JSON unmarshal/marshal #64

Merged
merged 5 commits into from
Feb 23, 2021
Merged

Conversation

grount
Copy link
Contributor

@grount grount commented Feb 22, 2021

issue #63

@grount grount marked this pull request as draft February 22, 2021 08:11
@knadh
Copy link
Owner

knadh commented Feb 22, 2021

This is cool. Just need to figure out why the tests are failing: https://travis-ci.com/github/knadh/koanf/jobs/484967206

Ints should avoid calling Ints64s to avoid unneeded conversion between
[]in64 and []int.

MergeAt unit test cannot be tested against JSON file since JSON treats
numbers as float64 and confmap.Provider/YAML treats them as integers.
@grount
Copy link
Contributor Author

grount commented Feb 22, 2021

See my latest commit, it seems that there were some issues.
Made a minor improvement as well.

  • Ints should avoid calling Ints64s to avoid unneeded conversion between
    []in64 and []int.

  • MergeAt unit test cannot be tested against JSON file since JSON treats
    numbers as float64 and confmap.Provider/YAML treats them as integers.

@grount grount marked this pull request as ready for review February 22, 2021 08:43
@knadh
Copy link
Owner

knadh commented Feb 22, 2021

MergeAt unit test cannot be tested against JSON file since JSON treats numbers as float64 and confmap.Provider/YAML treats them as integers.

Does that mean Load()ing a JSON numeric array into an instance and then loading a YAML numeric array for the same key will fail? Merges should be seamless unless strict mode is on.

@grount
Copy link
Contributor Author

grount commented Feb 22, 2021

With the current behavior, the latest loaded file will override the current type so I don't think it will be a problem.
It will make the strict mode a problem because loading [1,2,3] with JSON and YAML will fail even though they are the same.

Having said that:

  • When loading JSON and then YAML, intArray: [1,2,3] -> the type is []int. Doing k.Float64s will fail.
  • When loading YAML and then JSON, intArray: [1,2,3] -> the type is []float64. Doing k.Ints will fail.

This makes the behavior of getters somewhat inconsistent if someone loads multiple files and will try to get the value of an array.

With []int we can promote it to []float64 without loss, but doing []float64 to []int can cause a loss in case there is an actual float array [1.5,2.5,3.5] but in such case, in both usages, they should use k.Float64s and it should work.

WDYT?

@knadh
Copy link
Owner

knadh commented Feb 22, 2021

When loading JSON and then YAML, intArray: [1,2,3] -> the type is []int. Doing k.Float64s will fail.

toFloat64() and toInt64() can handle this. For float -> int lossiness, it can just be documented, in case someone really wants to call Ints() on a known float array. The MustInt64s() (all the Must*) aliases could panic for these cases, but that might involve a bit of internal rewiring.

@grount
Copy link
Contributor Author

grount commented Feb 22, 2021

It's possible to check if a []float64 includes only whole numbers to avoid data loss.
But the Must* functions will probably not work as intended, In MustInt64s when the user loaded [1.0,2.0,3.0] it should fail, but there is no way to know the intention since it could come either from the file or from the JSON.

@knadh
Copy link
Owner

knadh commented Feb 22, 2021

Yep, checking for whole numbers will work. toFloat64() and toInt64() can return errors. These errors can be propagated in the Must*() getters and ignored in the others.

@grount
Copy link
Contributor Author

grount commented Feb 22, 2021

I correct myself this is not an issue

When loading JSON and then YAML, intArray: [1,2,3] -> the type is []int. Doing k.Float64s will fail.
When loading YAML and then JSON, intArray: [1,2,3] -> the type is []float64. Doing k.Ints will fail.

Because toInt64 and toFloat64 try to cast at all cost using fmt.Sprintf and parsing, meaning no change in the existing behavior.

I think I will add another test of loading JSON and YAML and see that nothing is broken and you can merge it.

JSON treats int array as float64 array, this tests checks that there is
no issue between type conversions in Load Merge.
@knadh
Copy link
Owner

knadh commented Feb 22, 2021

Looks good. @rhnvrm could you also please review?

Copy link
Collaborator

@rhnvrm rhnvrm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, one minor doc change.

maps/maps.go Show resolved Hide resolved
@grount grount requested a review from rhnvrm February 23, 2021 06:23
getters.go Show resolved Hide resolved
@grount grount requested a review from knadh February 23, 2021 09:43
@knadh knadh merged commit e14a5b0 into knadh:master Feb 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants