Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-8621] [SQL] crosstab exception when one of the value is empty #7117

Closed
wants to merge 2 commits into from

Conversation

animeshbaranawal
Copy link

  1. Supported Empty String Analysis in parseAttributeName function.
  2. String Arguments containing two continuous " ` " character also supported.

cc @rxin @marmbrus

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@JoshRosen
Copy link
Contributor

Can you add a regression test?

@animeshbaranawal
Copy link
Author

Didn't get you @JoshRosen . Are not the crossTab tests already present in DataFrameStatSuite.scala?

@JoshRosen
Copy link
Contributor

Yes but it looks like this patch fixes a bug without adding a test that could have reproduced / exposed that bug. The fact that there was a bug in the old code but no test failed suggests that our test coverage was insufficient.

@JoshRosen
Copy link
Contributor

Maybe you have an unstaged test change that didn't get committed here?

@animeshbaranawal
Copy link
Author

Okay will add a suitable test.

@rxin
Copy link
Contributor

rxin commented Jun 30, 2015

cc @cloud-fan can you review this? thanks.

@cloud-fan
Copy link
Contributor

Hi @animeshbaranawal , thanks for fixing this!
Spark SQL should allow any kind of string as column name and that's my bad that I mistakenly deny empty string when wrote parseAttributeName. I have opened #7149 to fix this with less changing and added test, do you mind have a look?

asfgit pushed a commit that referenced this pull request Jul 1, 2015
improve the empty check in `parseAttributeName` so that we can allow empty string as column name.
Close #7117

Author: Wenchen Fan <cloud0fan@outlook.com>

Closes #7149 from cloud-fan/8621 and squashes the following commits:

efa9e3e [Wenchen Fan] support empty string
asfgit pushed a commit that referenced this pull request Jul 1, 2015
improve the empty check in `parseAttributeName` so that we can allow empty string as column name.
Close #7117

Author: Wenchen Fan <cloud0fan@outlook.com>

Closes #7149 from cloud-fan/8621 and squashes the following commits:

efa9e3e [Wenchen Fan] support empty string

(cherry picked from commit 31b4a3d)
Signed-off-by: Reynold Xin <rxin@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants