Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SparkSQL] allow UDF on struct #796

Closed
wants to merge 3 commits into from
Closed

[SparkSQL] allow UDF on struct #796

wants to merge 3 commits into from

Conversation

xiliu82
Copy link

@xiliu82 xiliu82 commented May 15, 2014

No description provided.

@xiliu82
Copy link
Author

xiliu82 commented May 15, 2014

This is to fix a bug where apply Hive UDF on a struct, it will throw NoMatch exception.

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@marmbrus
Copy link
Contributor

Can you add a test case? Or maybe find a HiveTest case that can be whitelisted in HiveCompatibilitySuite?

@xiliu82
Copy link
Author

xiliu82 commented May 19, 2014

I have a test but it has too much application information, so I will try to write a minimum test. Unfortunately named_struct also failed to work because it is expecting WritableConstantStringObjectInspector, and catalyst returned JavaStringObjectInspector, I am not sure I want to change that yet.

@marmbrus
Copy link
Contributor

test this please

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15324/

@marmbrus
Copy link
Contributor

Thanks for working on this!

Looks like adding the extra function is breaking the show_functions test. Maybe cleanup by deleting that new UDF at the end of your test?

@marmbrus
Copy link
Contributor

test this please

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15780/

@marmbrus
Copy link
Contributor

test this please

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15781/

@marmbrus
Copy link
Contributor

@pwendell, any idea what is going on here? I keep getting:

Building remotely on Ubuntu 14.04 on EC2 6 (ec2) in workspace /home/ubuntu/workspace/SparkPullRequestBuilder
java.io.IOException: remote file operation failed: /home/ubuntu/workspace/SparkPullRequestBuilder at hudson.remoting.Channel@7291f33e:Ubuntu 14.04 on EC2 6
    at hudson.FilePath.act(FilePath.java:916)
    at hudson.FilePath.act(FilePath.java:893)
    at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:66)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:566)
    at hudson.plugins.git.GitSCM.createClient(GitSCM.java:558)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:874)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
    at hudson.model.Run.execute(Run.java:1710)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:234)
Caused by: java.io.IOException: Remote call on Ubuntu 14.04 on EC2 6 failed
    at hudson.remoting.Channel.call(Channel.java:748)
    at hudson.FilePath.act(FilePath.java:909)
    ... 13 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy8
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at java.lang.reflect.Proxy.newInstance(Proxy.java:748)
    at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
    at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:100)
    at hudson.remoting.Channel.export(Channel.java:584)
    at hudson.remoting.Channel.export(Channel.java:553)
    at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:161)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1075)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at hudson.remoting.UserRequest._serialize(UserRequest.java:155)
    at hudson.remoting.UserRequest.serialize(UserRequest.java:164)
    at hudson.remoting.UserRequest.perform(UserRequest.java:126)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:328)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Recording test results
Finished: FAILURE

@marmbrus
Copy link
Contributor

test this please

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15782/

@rxin
Copy link
Contributor

rxin commented Jun 16, 2014

Jenkins, retest this please.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15830/

@asfgit asfgit closed this in f5a4049 Jun 17, 2014
asfgit pushed a commit that referenced this pull request Jun 17, 2014
@conviva-zz @marmbrus
[SPARK-2164][SQL] Allow Hive UDF on columns of type struct
8e6b77f 
Author: Xi Liu <xil@conviva.com>

Closes #796 from xiliu82/sqlbug and squashes the following commits:

328dfc4 [Xi Liu] [Spark SQL] remove a temporary function after test
354386a [Xi Liu] [Spark SQL] add test suite for UDF on struct
8fc6f51 [Xi Liu] [SparkSQL] allow UDF on struct

(cherry picked from commit f5a4049)
Signed-off-by: Michael Armbrust <michael@databricks.com>
@marmbrus
Copy link
Contributor

Thanks! I've merged this into master and 1.0 as https://issues.apache.org/jira/browse/SPARK-2164

pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
@conviva-zz @pdeyhim
[SPARK-2164][SQL] Allow Hive UDF on columns of type struct
f396726 
Author: Xi Liu <xil@conviva.com>

Closes apache#796 from xiliu82/sqlbug and squashes the following commits:

328dfc4 [Xi Liu] [Spark SQL] remove a temporary function after test
354386a [Xi Liu] [Spark SQL] add test suite for UDF on struct
8fc6f51 [Xi Liu] [SparkSQL] allow UDF on struct
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@conviva-zz
[SPARK-2164][SQL] Allow Hive UDF on columns of type struct
c7b4512 
Author: Xi Liu <xil@conviva.com>

Closes apache#796 from xiliu82/sqlbug and squashes the following commits:

328dfc4 [Xi Liu] [Spark SQL] remove a temporary function after test
354386a [Xi Liu] [Spark SQL] add test suite for UDF on struct
8fc6f51 [Xi Liu] [SparkSQL] allow UDF on struct
agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022
@ekrivokonmapr
MapR [SPARK-841] Backport SPARK-32723 (apache#796)
0b3319a
udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024
@ekrivokonmapr
MapR [SPARK-841] Backport SPARK-32723 (apache#796)
3ef6f19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@xiliu82 @AmplabJenkins @marmbrus @rxin @conviva-zz