Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: test with the HalfFloatType from arrow #503

Merged
merged 4 commits into from
Jun 4, 2024

Conversation

cocoa-xu
Copy link
Contributor

@cocoa-xu cocoa-xu commented Jun 1, 2024

Hi this PR adds an extra test case that tests ArrowFloatToHalfFloat and ArrowArrayViewGet{Double,Int,UInt}Unsafe with the HalfFloatType from arrow.

https://github.com/apache/arrow/blob/255dbf990c3d3e5fb1270a2a11efe0af2be195ab/cpp/src/arrow/type.h#L704-L713

And sorry that I didn't know there's a HalfFloatType in arrow-cpp when I was doing #501 🥲.

Sadly that we cannot simply append TestGetFromNumericArrayView<HalfFloatType>(); at the end of the ArrayViewTestGetNumeric test suite because we have to convert floats to half-floats using ArrowFloatToHalfFloat before calling builder.Append (otherwise we'll get weird values back in the subsequent ArrowArrayViewGet{Double,Int,UInt}Unsafe calls).

Added some simple C++ magic and we can simply append TestGetFromNumericArrayView<HalfFloatType>(); at the end of the ArrayViewTestGetNumeric test suite now.

Copy link
Member

@paleolimbot paleolimbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

I think the repetition is OK...we have a lot of repetition in our tests and while it would be good to go through and reduce that where possible, I prefer getting the test coverage there first.

There is also a few places this could be enabled in the integration tests if you have the bandwidth (no pressure!). (This can also be another PR if you prefer)

case NANOARROW_TYPE_FLOAT:
case NANOARROW_TYPE_DOUBLE: {
// JSON number to float_precision_ decimal places
LocalizedStream local_stream_opt(out);
local_stream_opt.SetFixed(float_precision_);
WriteFloatMaybeNull(out, value, 0);
for (int64_t i = 1; i < value->length; i++) {
out << ", ";
WriteFloatMaybeNull(out, value, i);
}
break;
}

case NANOARROW_TYPE_FLOAT:
case NANOARROW_TYPE_DOUBLE: {
// JSON number to float_precision_ decimal places
LocalizedStream local_stream_opt(out);
local_stream_opt.SetFixed(float_precision_);
WriteFloatMaybeNull(out, value, 0);
for (int64_t i = 1; i < value->length; i++) {
out << ", ";
WriteFloatMaybeNull(out, value, i);
}
break;
}

case NANOARROW_TYPE_FLOAT:
return SetBufferFloatingPoint<float>(data, buffer, error);
case NANOARROW_TYPE_DOUBLE:
return SetBufferFloatingPoint<double>(data, buffer, error);

@cocoa-xu
Copy link
Contributor Author

cocoa-xu commented Jun 3, 2024

and reduce that where possible

Not a problem, I added a helper function and it should look better now. :)

There is also a few places this could be enabled in the integration tests if you have the bandwidth (no pressure!)

It's done now ;)

Copy link
Member

@paleolimbot paleolimbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@paleolimbot paleolimbot merged commit d917f29 into apache:main Jun 4, 2024
32 checks passed
@cocoa-xu cocoa-xu deleted the cx-add-more-float16-tests branch June 4, 2024 12:55
paleolimbot added a commit that referenced this pull request Jun 10, 2024
Tests for #507 and
#501 and/or
#503 either used C++17 or
features from Arrow C++ > 12. Our test suite still supports these
(although perhaps parts of this support should be dropped soon).

On Windows, formatting with `%lu` was doing some unexpected formatting.
We could do a better job formatting 64-bit integers in error messages
(e.g., using `PRId64` and the requisite defines to ensure it works on
mingw); however, we probably won't ever be able to support properly
formatting an unsigned 64-bit integer on every platform we support. I
changed the error message (and its test) slightly to reflect that.
@paleolimbot paleolimbot added this to the nanoarrow 0.6.0 milestone Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants