[SPARK-8117] [SQL] Push codegen implementation into each Expression

[rxin]

This PR move codegen implementation of expressions into Expression class itself, make it easy to manage.

It introduces two APIs in Expression:

def gen(ctx: CodeGenContext): GeneratedExpressionCode
def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode): Code

gen(ctx) will call genSource(ctx, ev) to generate Java source code for the current expression. A expression needs to override genSource().

Here are the types:

type Term String
type Code String

/**
 * Java source for evaluating an [[Expression]] given a [[Row]] of input.
 */
case class GeneratedExpressionCode(var code: Code,
                               nullTerm: Term,
                               primitiveTerm: Term,
                               objectTerm: Term)
/**
 * A context for codegen, which is used to bookkeeping the expressions those are not supported
 * by codegen, then they are evaluated directly. The unsupported expression is appended at the
 * end of `references`, the position of it is kept in the code, used to access and evaluate it.
 */
class CodeGenContext {
  /**
   * Holding all the expressions those do not support codegen, will be evaluated directly.
   */
  val references: Seq[Expression] = new mutable.ArrayBuffer[Expression]()
}

This is basically #6660, but fixed style violation and compilation failure.

[rxin]

cc @davies I have a big followup pr to move a lot of expression test code for better grouping.

[SparkQA]

Test build #34388 has finished for PR 6690 at commit 73db80e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class GeneratedExpressionCode(var code: Code, var isNull: Term, var primitive: Term)
  • class CodeGenContext
  • case class Pow(left: Expression, right: Expression)
  • case class Rint(child: Expression) extends UnaryMathExpression(math.rint, "ROUND")
  • case class ToDegrees(child: Expression) extends UnaryMathExpression(math.toDegrees, "DEGREES")
  • case class ToRadians(child: Expression) extends UnaryMathExpression(math.toRadians, "RADIANS")

[davies]

@rxin Should we merge this one first?

[rxin]

Yes -- except it is failing tests :(

[rxin]

cc @liancheng why is partitioning suites failing?

[rxin]

Jenkins, retest this please.

[SparkQA]

Test build #34394 has finished for PR 6690 at commit 73db80e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class GeneratedExpressionCode(var code: Code, var isNull: Term, var primitive: Term)
  • class CodeGenContext
  • case class Pow(left: Expression, right: Expression)
  • case class Rint(child: Expression) extends UnaryMathExpression(math.rint, "ROUND")
  • case class ToDegrees(child: Expression) extends UnaryMathExpression(math.toDegrees, "DEGREES")
  • case class ToRadians(child: Expression) extends UnaryMathExpression(math.toRadians, "RADIANS")

[rxin]

@liancheng The failure is due to double type vs float type under the hood. What I don't get is that how come this code was passing before?!

Should we parse 1.5 as double type, or float type?

[rxin]

I filed #6692 to always use DoubleType. It's much simpler to have only one type for floating point numbers, rather than having to reason about float vs double.

[SparkQA]

Test build #34395 has finished for PR 6690 at commit e1368c2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class GeneratedExpressionCode(var code: Code, var isNull: Term, var primitive: Term)
  • class CodeGenContext
  • case class Pow(left: Expression, right: Expression)
  • case class Rint(child: Expression) extends UnaryMathExpression(math.rint, "ROUND")
  • case class ToDegrees(child: Expression) extends UnaryMathExpression(math.toDegrees, "DEGREES")
  • case class ToRadians(child: Expression) extends UnaryMathExpression(math.toRadians, "RADIANS")

[liancheng]

@rxin The test failure was because this PR overrides Literal.equals which uses value.equals(o.value). A quick Scala REPL check:

scala> 1.5 == 1.5f
res2: Boolean = true

scala> 1.5.equals(1.5f)
res3: Boolean = false

[rxin]

It's a problem of the test case that manifests from this.

We shouldn't use a double value in the place where we intend to have a float value. Let's be more careful with that in the future.