vaex-ml: package centred around machine learning related tasks #254

JovanVeljanoski · 2019-05-12T22:50:16Z

This is a big PR in which we are introducing vaex-ml, a vaex package centred around machine learning related tasks and applications. The following describes the contents:

vaex.ml.transformations: methods related to preprocessing: scalers, categorical encoders, PCA
vaex.ml.cluster: provides an efficient KMeans clustering algorithm
vaex.ml.ui: provides means to construct an ipywidget for nearly any transformer in this package
vaex.ml.xgboost: a binding to the xgboost library
vaex.ml.lightgbm: a binding to the lightgbm library
vaex.ml.catboost: a binding to the catboost library
vaex.ml.sklearn: a binding to the scikit-learn library. At the moment, only the estimators are supported.
vaex.ml.incubator: a module housing various machine learning models. The bindings in the incubator are considered experimental and are under testing. The API, implementation or support may change without notice.
vaex.ml.datasets: contains datasets for experimentation and training. Currently contains the titanic and the iris classical datasets. It also contains methods for replicating the iris dataset such that it contains a total of 10^9 samples, creating a "big data" example.
vaex.ml.generate: module that auto-generates an alternative API for the transformers and ML models.
vaex.ml.pipeline: provides a pipeline object for the vaex-ml transformers and estimators
vaex.ml.state: methods for serialisation of vaex objects
vaex.ml.linear_model: provides an implementation of linear models that operate on a grid (binned data) instead of individual samples.

Everything comes with a full suite of tests handled by pytest.

NB: The contents above and this description itself may change slightly as vaex-ml is integrated with vaex.

maartenbreddels

Awesome! I'll fix travis.
packages/vaex-ml/vaex/ml/pycache/ should not go in

maartenbreddels · 2019-05-13T06:29:57Z

packages/vaex-meta/setup.py

@@ -21,7 +21,8 @@
      'vaex-server==0.2',
      'vaex-hdf5==0.4',
      'vaex-astro==0.4',
-      'vaex-arrow==0.3'
+      'vaex-arrow==0.3',
+      'vaex-ml==0.4'


I don't think vaex-ml should be installed by default when you install vaex, like vaex-ui, what do you think?

Especially since vaex-ml now still depends on numba.

Why not? I thought it is good to come as a default. I am afraid it will not get noticed or add extra complexity otherwise.

For production environments, people can of course choose what to install.

maartenbreddels · 2019-05-13T06:30:57Z

packages/vaex-ml/vaex/ml/.coverage

@@ -0,0 +1 @@
+!coverage.py: This is a private format, don't read it directly!{"lines":{}}


I don't think this file should go into the repo.

Agreed! I tried to exclude it.. i will check the gitignore file again

Same for the pycache/ like things.

…licitly

…en values

…xecute()

…aring vaex.ml to pure sklearn

…n are copied

…to test similarity of results up to a certain precision.

…th the binaries, we do run on windows/osx

…be too old

maartenbreddels · 2019-06-25T19:43:55Z

🎉 yeah!

JovanVeljanoski requested a review from maartenbreddels May 12, 2019 22:50

maartenbreddels reviewed May 13, 2019

View reviewed changes

JovanVeljanoski force-pushed the vaexml branch from bcac1bf to 846f3d3 Compare May 16, 2019 16:33

maartenbreddels force-pushed the vaexml branch from 7199076 to 3ebf672 Compare May 21, 2019 17:37

JovanVeljanoski added the major-addition label May 22, 2019

maartenbreddels force-pushed the vaexml branch 4 times, most recently from bacb96b to 1ed152d Compare May 26, 2019 14:56

maartenbreddels force-pushed the vaexml branch 3 times, most recently from 548aff4 to 9acd38c Compare May 31, 2019 13:46

JovanVeljanoski mentioned this pull request Jun 15, 2019

vaex-ml techniques #301

Closed

xdssio added the priority: high label Jun 17, 2019

maartenbreddels force-pushed the master branch from cf7f3fe to 7f6ac9a Compare June 24, 2019 17:18

JovanVeljanoski and others added 14 commits June 24, 2019 21:35

Machine learning support in vaex: adding the vaex-ml package

fcb3ad2

Adding tests for the vaex-ml package

ac83d4a

Update to .gitignore

7b627c6

ci: set up travis and appveyor for vaex-ml

01a3611

setup: make symlinks relative

3cfd689

get pygbm and annoy from pypi

5de1875

skip pygbm, get xgboost from conda-forge

23f7aaa

skip generated code for df.ml

4a3c1ab

added iris and titanic dataset

a09c7e7

try without xgboost

fdc5997

chore(gitignore): ignore caches and all

10e36ca

chore(ci): use install py-xgboost instead of just xgboost

17155e6

fix(py27,ml): exec for py27 needs to be passed locals and globals exp…

406ddde

…licitly

fix(ml,FrequenceEncoder): use the new supported nan and null and unse…

f12b819

…en values

maartenbreddels and others added 21 commits June 24, 2019 21:35

fix(ml): use df.execute() which does slightly more than df.executor.e…

1d0be48

…xecute()

test(ml,FrequenceEncoder): more edge cases

84f5140

test(ml,sklearn): used fixed seeds for reproducable results when comp…

e7fcaf4

…aring vaex.ml to pure sklearn

fix(ml,setup.py): include package data, so files listed in MANIFEST.i…

b15d2bd

…n are copied

Since LinearRegression does not support random_state, use np.testing …

d38fb29

…to test similarity of results up to a certain precision.

Skipping some tests if python version is less than 3.6

8b783de

Remove files that should be ignored anyway.

57f9b2d

test(ml): skip lightgbm and xgboost on linux/travis, something odd wi…

7d8df74

…th the binaries, we do run on windows/osx

test(ml): skip xgboost tests when not installed

c0c2b90

test(ml): skip lightgbm and xgboost on python27 for osx, it seems to …

f2cb7c3

…be too old

test(ml): skip sklearn for py27

7553e65

Import pytest module in test files as needed

f610e6a

Import pytest at the top of file

981ba81

fix: importskip at wrong module

f6003b6

fix: importerskip sklearn

048c38e

fix: py27 compat

daaf0fb

fix: importerskip sklearn

3cdf99a

do not run tests for vaex-ml on py27

9b575e4

do not run tests for vaex-ml on py27

dd8f6f8

fix(test): windows + pip gives issues

acdbf1b

fix(test): windows + pip gives issues (install geopandas)

7793073

maartenbreddels force-pushed the vaexml branch from b56a78f to 7793073 Compare June 24, 2019 19:37

fix(test,ml): assertion->array_almost_equal

8d13bdd

JovanVeljanoski merged commit 24bf089 into master Jun 25, 2019

maartenbreddels deleted the vaexml branch May 11, 2020 09:09

jameslamb mentioned this pull request Sep 8, 2023

[docs] add vaex-ml to list of external repositories microsoft/LightGBM#6085

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vaex-ml: package centred around machine learning related tasks #254

vaex-ml: package centred around machine learning related tasks #254

JovanVeljanoski commented May 12, 2019

maartenbreddels left a comment

maartenbreddels May 13, 2019

maartenbreddels May 13, 2019

JovanVeljanoski May 13, 2019

maartenbreddels May 13, 2019

JovanVeljanoski May 13, 2019

JovanVeljanoski May 13, 2019

maartenbreddels commented Jun 25, 2019

		@@ -0,0 +1 @@
		!coverage.py: This is a private format, don't read it directly!{"lines":{}}

vaex-ml: package centred around machine learning related tasks #254

vaex-ml: package centred around machine learning related tasks #254

Conversation

JovanVeljanoski commented May 12, 2019

maartenbreddels left a comment

Choose a reason for hiding this comment

maartenbreddels May 13, 2019

Choose a reason for hiding this comment

maartenbreddels May 13, 2019

Choose a reason for hiding this comment

JovanVeljanoski May 13, 2019

Choose a reason for hiding this comment

maartenbreddels May 13, 2019

Choose a reason for hiding this comment

JovanVeljanoski May 13, 2019

Choose a reason for hiding this comment

JovanVeljanoski May 13, 2019

Choose a reason for hiding this comment

maartenbreddels commented Jun 25, 2019