[optional] support loading and saving models with protobuf

[wxchan]

related to #372

• solve conflicts
• test windows support: mingw can work, vs not tested yet

[guolinke]

@wxchan
the license of protobuf? and is it cross platform ?

[wxchan]

@guolinke

LICENSE: https://github.com/google/protobuf/blob/master/LICENSE

I add a test and it passes on both linux and osx, I am not sure about windows.

test on osx: https://travis-ci.org/wxchan/LightGBM/jobs/274043104, it takes too long.

[guolinke]

@wxchan what is the compression ratio of using protobuf, compared with raw text format ?

I think using protobuf is too heavy, can we just use a simple binary format (like the dataset bin file) ?

[wxchan]

@guolinke I didn't test on big model yet. My computer takes too long to load big datasets. The good thing of protobuf is, it's readable through .proto file. We can easily know what features are saved.

[wxchan]

@guolinke I made a test on Higgs dataset

text ->proto
after 500 rounds
save: 0.612300s -> 0.023627s
load: 0.471999s -> 0.024712s
size: 13M -> 5.9M

[henry0312]

please check #904 (comment)

[wxchan]

@henry0312 I saw. Do you have an agreement yet? I can add it later, I think it's a waste of resource to run those python tests I didn't alter multiple time now.

[henry0312]

Do you have an agreement yet?

No, I don't know if @guolinke agrees with it yet, but I think we should run tests with Python 2.7.x and Python 3.6.x at least.

I think it's a waste of resource to run those python tests I didn't alter multiple time now.

I got it!
Yeah, Travis always faces with lacks of instances of OSX.
We have no choice but to reduce test cases for OSX 😢

[wxchan]

one similar thing: https://github.com/dmlc/treelite/blob/master/src/tree.proto

[guolinke]

@wxchan any updates of this ?
this could be merged. but we still need a cross platform solution.

[guolinke]

@wxchan it seems that the treelite is like our if-else solution ?

[wxchan]

@hcho3
have you used this proto https://github.com/dmlc/treelite/blob/master/src/tree.proto to compress LightGBM model? what's the compression ratio?

[hcho3]

@wxchan No, in treelite, protobuf is only for input not for output. (Protobuf is one of four methods to input a model into treelite compiler.) So LightGBM model isn't being translated into protobuf.

Not sure how effective protobuf is for compression.

[Tony-Y]

I found useful information: https://github.com/thekvs/cpp-serializers
I think cereal is better than protobuf.

[wxchan]

@guolinke I already finished this thread.

[guolinke]

@wxchan this PR will break the build of VS. I will revert it and merge back after fixing this.

[wxchan]

I think I forgot changing some path of file in windows/LightGBM.vcxproj after moving it, is this the reason?

[guolinke]

@wxchan it cannot find some ".h" and ".cpp" files, seems are some the auto-generated protobuf files.

[wxchan]

@guolinke maybe this one https://github.com/wxchan/LightGBM/blob/1e6091c01f75cbbc86b55b104ae3231d302a0a9f/src/proto/not_implemented/model.pb.h? what's the error message?

[guolinke]

@wxchan
yeah, it says cannot find model.pb.h ... .
Maybe forget to add it to these include path ? https://github.com/Microsoft/LightGBM/blob/master/windows/LightGBM.vcxproj#L63

[wxchan]

@guolinke yes, I forgot adding it. btw, there are some files for when proto=on, some files for when proto=off, they are controlled by cmake, should I add them all to windows/LightGBM.vcxproj?

[guolinke]

@wxchan it seems VS cannot support proto, so we can just disable it