[SPARK-5890][ML] Add feature discretizer

[yinxusen]

JIRA issue here.

I borrow the code of findSplits from RandomForest. I don't think it's good to call it from RandomForest directly.

[SparkQA]

Test build #31281 has finished for PR 5779 at commit f6be730.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class FeatureDiscretizer extends Transformer with HasInputCol with HasOutputCol
• This patch does not change any dependencies.

[jkbradley]

FeatureDiscretizer should produce a Bucketizer as its Model. Could you please refactor this accordingly? I'd be happy to help as needed, so please let me know.

Alternatively, we could do a PR for Bucketizer first and then update this PR to use the Bucketizer.

[yinxusen]

@jkbradley Sure, I'll refactor it ASAP.

[jkbradley]

Thanks!

[jkbradley]

@yinxusen It will be great if we can get this in by tomorrow. If it would be helpful, I'd be happy to do part, such as writing the Bucketizer PR.

Also, can we rename this to QuantileDiscretizer? There will likely be other feature discretization methods added in the future. Thanks!

[yinxusen]

I am writing the Bucketizer now. I will refactor this PR with Bucketizer soon afterwards. You can check it in tomorrow, I think. Sorry for the delay, because I just finished my vacation.

[jkbradley]

@yinxusen No problem, I know it's last minute & appreciate your efforts! I'll check tomorrow morning.

[SparkQA]

Test build #32810 has finished for PR 5779 at commit 5ffa167.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class QuantileDiscretizer(override val uid: String)

[jkbradley]

Reviewing now

[jkbradley]

Maximum number of buckets (quantiles, or categories) into which data points are grouped. Must be >= 2. (Please copy to IntParam doc below as well.)

[SparkQA]

Test build #820 has finished for PR 5779 at commit 5ffa167.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class QuantileDiscretizer(override val uid: String)

[jkbradley]

Ping. It'd be nice to get this into 1.6

[jkbradley]

Checking for updates here too : )

[jkbradley]

@yinxusen If you won't be able to keep working on this, please let me know. Someone else or I can take over. Thanks.

[yinxusen]

@jkbradley No, I'll keep work on this, and sorry for the delay.

[SparkQA]

Test build #42886 has finished for PR 5779 at commit 99d46ee.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class QuantileDiscretizer(override val uid: String)

[SparkQA]

Test build #42889 has finished for PR 5779 at commit 954d7b9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class QuantileDiscretizer(override val uid: String)

[jkbradley]

State default value here (in Scala doc only, not IntParam doc)

[jkbradley]

@yinxusen Thanks for the updates!

[SparkQA]

Test build #42983 has finished for PR 5779 at commit 0e807bc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class QuantileDiscretizer(override val uid: String)

[jkbradley]

organize imports

[jkbradley]

I just made minor comments. It looks good, though I'm curious about whether that test caught bugs in the previous version of this PR.

[yinxusen]

Yes, it can catch the previous bug. In the previous setting, the check

checkDiscretizedData(sc,
      Array[Double](1, 2, 3, 3, 3, 3, 3, 3, 3),
      3,
      Array[Double](0, 1, 2, 2, 2, 2, 2, 2, 2),
      Array("-Infinity, 2.0", "2.0, 3.0", "3.0, Infinity"))

throws error because it misses the split point of 3.0.

[SparkQA]

Test build #43159 has finished for PR 5779 at commit 9f878e9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class QuantileDiscretizer(override val uid: String)

[jkbradley]

OK thanks for confirming! LGTM, merging with master.