Add support for reading and writing the .NET Half type #418
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adamreeve
commented
Feb 4, 2024
| } | ||
| else | ||
| { | ||
| // Float-16 values are always stored in little-endian order |
There was a problem hiding this comment.
For completeness I thought we should handle big-endian machines, as we have a similar check for the guid type, but we currently only build x64 and arm64 native binaries for the nuget package, and there doesn't seem to be an easy way to test this. Maybe we should just throw a not implemented exception for big endian instead? This didn't seem to affect performance at least.
There was a problem hiding this comment.
I think it is fine as it is now.
marcin-krystianc
approved these changes
Feb 5, 2024
| } | ||
| else | ||
| { | ||
| // Float-16 values are always stored in little-endian order |
There was a problem hiding this comment.
I think it is fine as it is now.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #413
This adds support for the new
Float16logical type added in Arrow 15. I've added a new .NET 6 target to the ParquetSharp project to allow using the newHalftype, which required fixing a few errors related to nullable reference type checking when building with the newer target.One thing to be aware of is that writing
Halfvalues can be a bit slower than floats because of the extra overhead of writing these as fixed-length byte arrays rather than having a dedicated physical type. We could possibly improve this in future if it turns out to be a problem. I did some quick benchmarking of reading and writing 1 million random floats, doubles and half values, with dictionary encoding disabled: