Skip to content

Commit

Permalink
#427 - Based on feedback on the pull, reverting test changes - no lon…
Browse files Browse the repository at this point in the history
…ger necessary
  • Loading branch information
chris-twiner committed Jul 27, 2020
1 parent 9e0de29 commit 69ae591
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 5 deletions.
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ This essentially allows you to use any version of Frameless with any version of
The aforementioned table simply provides the versions of Spark we officially compile
and test Frameless with, but other versions may probably work as well.

### Breaking changes in 0.9

* Spark 3 introduces a new ExpressionEncoder approach, the schema for single value DataFrame's is now ["value"](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala#L270) not "_1".

## Why?

Frameless introduces a new Spark API, called `TypedDataset`.
Expand All @@ -74,7 +78,7 @@ detailed comparison of `TypedDataset` with Spark's `Dataset` API.
* [Proof of Concept: TypedDataFrame](http://typelevel.org/frameless/TypedDataFrame.html)

## Quick Start
Frameless is compiled against Scala 2.12.x
Since the 0.9.x release, Frameless is compiled only against Scala 2.12.x.

To use Frameless in your project add the following in your `build.sbt` file as needed:

Expand Down
8 changes: 4 additions & 4 deletions dataset/src/test/scala/frameless/ops/PivotTest.scala
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,19 @@ import org.scalacheck.Prop._
import org.scalacheck.{Gen, Prop}

class PivotTest extends TypedDatasetSuite {
def withCustomGenX4: Gen[Vector[X4[String, String, Long, Boolean]]] = {
val kvPairGen: Gen[X4[String, String, Long, Boolean]] = for {
def withCustomGenX4: Gen[Vector[X4[String, String, Int, Boolean]]] = {
val kvPairGen: Gen[X4[String, String, Int, Boolean]] = for {
a <- Gen.oneOf(Seq("1", "2", "3", "4"))
b <- Gen.oneOf(Seq("a", "b", "c"))
c <- arbitrary[Long]
c <- arbitrary[Int]
d <- arbitrary[Boolean]
} yield X4(a, b, c, d)

Gen.listOfN(4, kvPairGen).map(_.toVector)
}

test("X4[Boolean, String, Int, Boolean] pivot on String") {
def prop(data: Vector[X4[String, String, Long, Boolean]]): Prop = {
def prop(data: Vector[X4[String, String, Int, Boolean]]): Prop = {
val d = TypedDataset.create(data)
val frameless = d.groupBy(d('a)).
pivot(d('b)).on("a", "b", "c").
Expand Down

0 comments on commit 69ae591

Please sign in to comment.