Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Spark-compatible CAST float/double to string #312

Closed
andygrove opened this issue Apr 24, 2024 · 3 comments · Fixed by #346
Closed

Implement Spark-compatible CAST float/double to string #312

andygrove opened this issue Apr 24, 2024 · 3 comments · Fixed by #346
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@andygrove
Copy link
Member

What is the problem the feature request solves?

We currently delegate to DataFusion when casting float/double to string and there are some differences in behavior compared to Spark.

  • Spark uses Infinity and -Infinity and DataFusion uses inf and -inf
  • Spark uses E for exponent and DataFusion uses e

There can also be differences in formatting but where the values are semantically equivalent. For example, for the input 1.0499721536516571E-4:

Spark produces "1.0499721536516571E-4"
DataFusion produces: "0.00010499721536516571"

We should create a compatibility guide and include this information.

Here are some suggested values to test:

val testData = Seq(Float.NegativeInfinity, Float.PositiveInfinity,
      Float.MinPositiveValue, Float.MinValue, Float.MaxValue, Float.NaN, -0.0f, 0.0f)

Describe the potential solution

No response

Additional context

No response

@andygrove andygrove added enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed labels Apr 24, 2024
@mattharder91
Copy link
Contributor

Can I try this one out?

@viirya
Copy link
Member

viirya commented Apr 27, 2024

Of course. Thanks @mattharder91. Please let us know if you have any question.

@mattharder91
Copy link
Contributor

Here you go #346

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants