-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Alternate column referencing syntax for TypeDataset #39
Comments
I was playing with this by trying implicit conversion from |
Hey guys, I think this is a great feature and it will make writing expressions much cleaner. I was able to get this working: def select[A](column: Witness.Lt[Symbol])(
implicit
exists: TypedColumn.Exists[T, column.T, A],
encoder: TypedEncoder[A]): TypedDataset[A] = select(col(column)) It combines what the With this you can do select('foo) and it works. Unfortunately this cause a strange issue when passing a Obviously, the solution is not perfect (since it's causing an issue), but the direction might be promising? What do you guys think? Any ideas? |
The biggest challenge with this syntax (besides IDE support) is the support full scope of Spark Early work on the lib made use of shapeless' |
I think we could make @kanterov idea of "implicit conversion from scala> :paste
// Entering paste mode (ctrl-D to finish)
trait TypedColumn[S <: Singleton]
implicit def lift[S <: Singleton](s: S): TypedColumn[S] = new TypedColumn[S] {}
def select[A <: Singleton](p: TypedColumn[A]) = p
implicit class AddIntToTypedColumn(i: Int) {
def plus[S <: Singleton](s: TypedColumn[S]) = s
}
// Exiting paste mode, now interpreting.
defined trait TypedColumn
lift: [S <: Singleton](s: S)TypedColumn[S]
select: [A <: Singleton](p: TypedColumn[A])TypedColumn[A]
defined class AddIntToTypedColumn
scala> select("hello")
res0: TypedColumn["hello"] = $anon$1@7e40c3aa
scala> select(1 plus "hello")
res1: TypedColumn["hello"] = $anon$1@5d864a5 |
@OlivierBlanvillain yes, supporting expressions with select should definetly be part of ay solution.
|
On my setup it works with the following: $ cat build.sbt
scalaVersion := "2.11.8"
scalaOrganization := "org.typelevel"
libraryDependencies ++= Seq(
"org.typelevel" %% "cats" % "0.7.2",
"com.chuusai" %% "shapeless" % "2.3.2")
scalacOptions := Seq(
"-deprecation",
"-encoding", "UTF-8",
"-feature",
"-language:implicitConversions",
"-unchecked",
"-Xfuture",
"-Xlint",
"-Yinline-warnings",
"-Yno-adapted-args",
"-Ywarn-dead-code",
"-Ywarn-numeric-widen",
"-Ypartial-unification",
"-Yliteral-types",
"-Ywarn-value-discard")
$ cat project/build.properties
sbt.version=0.13.13-RC2
$ sbt console
[...] |
@OlivierBlanvillain This is awesome. Does it require Typelevel Scala to compile user code? In this case, we might still want to investigate if we can reuse macro from shapeless somehow, it would be nice if we find cheap solution instead of forcing users to switch Scala compiler :). |
Yes, we would need this PR to be merged to have singelton types in Lightbend Scala. |
Wow, I'm looking forward to have this if one day Spark compiles with 2.13.0 |
Hi, closing it for now with #449 merged. |
It would be nice to add an alternate column referencing syntax to the
TypedDataset
API which is closer to the vanilla syntax, similarly to the way it's done in TypeDataFrame.Currently it looks like: (source)
I think it should be possible to change it to something like:
It would be also interesting to investigate an alternate syntax for
td.colMany('b, 'b)
(equivalent to accessing_.b.b
).The text was updated successfully, but these errors were encountered: