Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated CountVectorizer #3

Merged
merged 1 commit into from
Aug 14, 2015
Merged

Updated CountVectorizer #3

merged 1 commit into from
Aug 14, 2015

Conversation

jkbradley
Copy link

Updates:

  • Renamed "minCount" to "minTokenCount"
  • Added "minTermFreq" back, including unit test
  • Moved all Params to include in both Estimator and Model so that they can be viewed in either.

New thoughts:

  • Previously, we had decided to call "minTermFreq" a "frequency" even though it is a count. That was to follow other things such as sklearn and "TF-IDF."
  • Now, I would like to rename it to "minTermCount." Yuhao got me thinking along these lines, and it actually matches other libraries to some extent:
    • sklearn calls it "min_tf," but it accepts both integers (counts) and doubles (fractions or frequencies). Since we cannot take both the way Params are set up, the best we could do is have a minTermCount Param and a minTermFreq Param. Frequency might actually be a better way to specify this threshold.
    • TF-IDF arguably uses fractional frequencies (not counts)...depending on whether "TF-IDF" refers to the value or the normalizer.

Proposal: I propose we do 1 of the following:

  • (A) Rename "minTermFreq" to "minTermCount"
  • (B) Change "minTermFreq" to take a Double fraction
  • (C) Remove "minTermFreq" for now pending further discussion

CC: @hhbyyh @mengxr What do you think?

@hhbyyh Feel free to merge, take parts, or modify as you see fit. It may be easiest if you go ahead and merge/modify so that @mengxr can view your full PR.

Thanks!

@jkbradley
Updates:
0fe9f96 
* Renamed "minCount" to "minTokenCount"
* Added "minTermFreq" back, including unit test
* Moved all Params to include in both Estimator and Model so that they can be viewed in either.
@jkbradley
Copy link
Author

PS: I'm operating on low sleep, so please check my update!

@hhbyyh
Copy link
Owner

hhbyyh commented Aug 14, 2015

I see only few minor things. Thanks for the spending the time and efforts on this.

I plan to merge this for now and copy your comments to the PR apache#7388. We can have further discussion and updates there. Sounds good?

hhbyyh added a commit that referenced this pull request Aug 14, 2015
@hhbyyh
Merge pull request #3 from jkbradley/countvec
7d4743a 
Updated CountVectorizer
@hhbyyh hhbyyh merged commit 7d4743a into hhbyyh:cvEstimator Aug 14, 2015
@hhbyyh hhbyyh mentioned this pull request Aug 14, 2015
Closed
hhbyyh pushed a commit that referenced this pull request Aug 18, 2015
@rekhajoshm @srowen
[SPARK-8593] [CORE] Sort app attempts by start time.
42d8a01 
This makes sure attempts are listed in the order they were executed, and that the
app's state matches the state of the most current attempt.

Author: Joshi <rekhajoshm@gmail.com>
Author: Rekha Joshi <rekhajoshm@gmail.com>

Closes apache#7253 from rekhajoshm/SPARK-8593 and squashes the following commits:

874dd80 [Joshi] History Server: updated order for multiple attempts(logcleaner)
716e0b1 [Joshi] History Server: updated order for multiple attempts(descending start time works everytime)
548c753 [Joshi] History Server: updated order for multiple attempts(descending start time works everytime)
83306a8 [Joshi] History Server: updated order for multiple attempts(descending start time)
b0fc922 [Joshi] History Server: updated order for multiple attempts(updated comment)
cc0fda7 [Joshi] History Server: updated order for multiple attempts(updated test)
304cb0b [Joshi] History Server: updated order for multiple attempts(reverted HistoryPage)
85024e8 [Joshi] History Server: updated order for multiple attempts
a41ac4b [Joshi] History Server: updated order for multiple attempts
ab65fa1 [Joshi] History Server: some attempt completed to work with showIncomplete
0be142d [Rekha Joshi] Merge pull request #3 from apache/master
106fd8e [Rekha Joshi] Merge pull request #2 from apache/master
e3677c9 [Rekha Joshi] Merge pull request #1 from apache/master
hhbyyh pushed a commit that referenced this pull request Aug 18, 2015
@rekhajoshm @mengxr
[SPARK-9118] [ML] Implement IntArrayParam in mllib
1017908 
Implement IntArrayParam in mllib

Author: Rekha Joshi <rekhajoshm@gmail.com>
Author: Joshi <rekhajoshm@gmail.com>

Closes apache#7481 from rekhajoshm/SPARK-9118 and squashes the following commits:

d3b1766 [Joshi] Implement IntArrayParam
0be142d [Rekha Joshi] Merge pull request #3 from apache/master
106fd8e [Rekha Joshi] Merge pull request #2 from apache/master
e3677c9 [Rekha Joshi] Merge pull request #1 from apache/master
hhbyyh pushed a commit that referenced this pull request Jan 12, 2016
@yanboliang @mengxr
[SPARK-6267] [MLLIB] Python API for IsotonicRegression
384ac3c 
https://issues.apache.org/jira/browse/SPARK-6267

Author: Yanbo Liang <ybliang8@gmail.com>
Author: Xiangrui Meng <meng@databricks.com>

Closes apache#5890 from yanboliang/spark-6267 and squashes the following commits:

f20541d [Yanbo Liang] Merge pull request #3 from mengxr/SPARK-6267
7f202f9 [Xiangrui Meng] use Vector to have the best Python 2&3 compatibility
4bccfee [Yanbo Liang] fix doctest
ec09412 [Yanbo Liang] fix typos
8214bbb [Yanbo Liang] fix code style
5c8ebe5 [Yanbo Liang] Python API for IsotonicRegression

(cherry picked from commit 7b14578)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
hhbyyh pushed a commit that referenced this pull request Jan 12, 2016
@brkyvz @mengxr
[SPARK-7388] [SPARK-7383] wrapper for VectorAssembler in Python
6b9737a 
The wrapper required the implementation of the `ArrayParam`, because `Array[T]` is hard to obtain from Python. `ArrayParam` has an extra function called `wCast` which is an internal function to obtain `Array[T]` from `Seq[T]`

Author: Burak Yavuz <brkyvz@gmail.com>
Author: Xiangrui Meng <meng@databricks.com>

Closes apache#5930 from brkyvz/ml-feat and squashes the following commits:

73e745f [Burak Yavuz] Merge pull request #3 from mengxr/SPARK-7388
c221db9 [Xiangrui Meng] overload StringArrayParam.w
c81072d [Burak Yavuz] addressed comments
99c2ebf [Burak Yavuz] add to python_shared_params
39ecb07 [Burak Yavuz] fix scalastyle
7f7ea2a [Burak Yavuz] [SPARK-7388][SPARK-7383] wrapper for VectorAssembler in Python

(cherry picked from commit 9e2ffb1)
Signed-off-by: Xiangrui Meng <meng@databricks.com>
hhbyyh pushed a commit that referenced this pull request Jan 12, 2016
@pzzs @marmbrus
[SPARK-7437] [SQL] Fold "literal in (item1, item2, ..., literal, ...)…
c30982d 
…" into true or false directly

SQL
```
select key from src where 3 in (4, 5);
```
Before
```
== Optimized Logical Plan ==
Project [key#12]
 Filter 3 INSET (5,4)
  MetastoreRelation default, src, None
```

After
```
== Optimized Logical Plan ==
LocalRelation [key#228], []
```

Author: Zhongshuai Pei <799203320@qq.com>
Author: DoingDone9 <799203320@qq.com>

Closes apache#5972 from DoingDone9/InToFalse and squashes the following commits:

4c722a2 [Zhongshuai Pei] Update predicates.scala
abe2bbb [Zhongshuai Pei] Update Optimizer.scala
fa461a5 [Zhongshuai Pei] Update Optimizer.scala
e34c28a [Zhongshuai Pei] Update predicates.scala
24739bd [Zhongshuai Pei] Update ConstantFoldingSuite.scala
f4dbf50 [Zhongshuai Pei] Update ConstantFoldingSuite.scala
35ceb7a [Zhongshuai Pei] Update Optimizer.scala
36c194e [Zhongshuai Pei] Update Optimizer.scala
2e8f6ca [Zhongshuai Pei] Update Optimizer.scala
14952e2 [Zhongshuai Pei] Merge pull request apache#13 from apache/master
f03fe7f [Zhongshuai Pei] Merge pull request apache#12 from apache/master
f12fa50 [Zhongshuai Pei] Merge pull request apache#10 from apache/master
f61210c [Zhongshuai Pei] Merge pull request apache#9 from apache/master
34b1a9a [Zhongshuai Pei] Merge pull request apache#8 from apache/master
802261c [DoingDone9] Merge pull request apache#7 from apache/master
d00303b [DoingDone9] Merge pull request apache#6 from apache/master
98b134f [DoingDone9] Merge pull request apache#5 from apache/master
161cae3 [DoingDone9] Merge pull request #4 from apache/master
c87e8b6 [DoingDone9] Merge pull request #3 from apache/master
cb1852d [DoingDone9] Merge pull request #2 from apache/master
c3f046f [DoingDone9] Merge pull request #1 from apache/master

(cherry picked from commit 4b5e1fe)
Signed-off-by: Michael Armbrust <michael@databricks.com>
hhbyyh pushed a commit that referenced this pull request Jan 12, 2016
@rekhajoshm @sarutak
[SPARK-5768] [WEB UI] Fix for incorrect memory in Spark UI
74001db 
Fix for incorrect memory in Spark UI as per SPARK-5768

Author: Joshi <rekhajoshm@gmail.com>
Author: Rekha Joshi <rekhajoshm@gmail.com>

Closes apache#6972 from rekhajoshm/SPARK-5768 and squashes the following commits:

b678a91 [Joshi] Fix for incorrect memory in Spark UI
2fe53d9 [Joshi] Fix for incorrect memory in Spark UI
eb823b8 [Joshi] SPARK-5768: Fix for incorrect memory in Spark UI
0be142d [Rekha Joshi] Merge pull request #3 from apache/master
106fd8e [Rekha Joshi] Merge pull request #2 from apache/master
e3677c9 [Rekha Joshi] Merge pull request #1 from apache/master

(cherry picked from commit 085a721)
Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
hhbyyh pushed a commit that referenced this pull request Jan 12, 2016
@rekhajoshm @srowen
[SPARK-8593] [CORE] Sort app attempts by start time.
5983828 
This makes sure attempts are listed in the order they were executed, and that the
app's state matches the state of the most current attempt.

Author: Joshi <rekhajoshm@gmail.com>
Author: Rekha Joshi <rekhajoshm@gmail.com>

Closes apache#7253 from rekhajoshm/SPARK-8593 and squashes the following commits:

874dd80 [Joshi] History Server: updated order for multiple attempts(logcleaner)
716e0b1 [Joshi] History Server: updated order for multiple attempts(descending start time works everytime)
548c753 [Joshi] History Server: updated order for multiple attempts(descending start time works everytime)
83306a8 [Joshi] History Server: updated order for multiple attempts(descending start time)
b0fc922 [Joshi] History Server: updated order for multiple attempts(updated comment)
cc0fda7 [Joshi] History Server: updated order for multiple attempts(updated test)
304cb0b [Joshi] History Server: updated order for multiple attempts(reverted HistoryPage)
85024e8 [Joshi] History Server: updated order for multiple attempts
a41ac4b [Joshi] History Server: updated order for multiple attempts
ab65fa1 [Joshi] History Server: some attempt completed to work with showIncomplete
0be142d [Rekha Joshi] Merge pull request #3 from apache/master
106fd8e [Rekha Joshi] Merge pull request #2 from apache/master
e3677c9 [Rekha Joshi] Merge pull request #1 from apache/master

(cherry picked from commit 42d8a01)
Signed-off-by: Sean Owen <sowen@cloudera.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jkbradley @hhbyyh