Add quantized Fully Convolutional Network model #877

wuxun-zhang · 2019-07-22T06:04:17Z

This PR is to add quantized FCN models into GluonCV model-zoo. With this PR, INT8 FCN model can get more than ~4x speedup on AWS C5.12xlarge.

mli · 2019-07-22T06:24:18Z

Job PR-877-1 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-877/1/index.html
Code coverage of this PR: vs. Master:

xinyu-intel · 2019-07-22T08:57:36Z

scripts/segmentation/eval_segmentation.py

+            metric.update(targets, outputs)
+            pixAcc, mIoU = metric.get()
+            tbar.set_description( 'pixAcc: %.4f, mIoU: %.4f' % (pixAcc, mIoU))
+        else:


what will happen if not eval only

In the current script, args.eval=False is only required for test mode, which will convert the input RGB images into segmented images as outputs. However, there are still incompatible issues for INT8 models and I'll try to fix them.

xinyu-intel · 2019-07-22T08:58:28Z

scripts/segmentation/eval_segmentation.py

+    return args
+
+
+def test(args, model):


also add a dummy benchmark like ssd.

xinyu-intel

Add test case in test_modelzoo.py. Once #863 fix mkldnn ci, quantized fcn will be test automatically.

zhanghang1989 · 2019-07-23T00:55:57Z

scripts/segmentation/eval_segmentation.py

@@ -0,0 +1,172 @@
+import os


Did you rename test.py to eval_segmentation.py?

Could you add the quantization to the original test.py?
If it is not possible, please change keep the orginal test.py and create a seperate file. Do not deleate existing files.

Thanks for your comments. Will keep original test.py and add support for quantized model in this script.

wuxun-zhang · 2019-07-23T13:28:17Z

@xinyu-intel @zhanghang1989 Updated. Please take a review again. Thanks.

mli · 2019-07-23T13:47:20Z

Job PR-877-3 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-877/3/index.html
Code coverage of this PR: vs. Master:

mli · 2019-07-23T14:08:13Z

Job PR-877-4 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-877/4/index.html
Code coverage of this PR: vs. Master:

zhanghang1989 · 2019-07-23T18:52:50Z

scripts/segmentation/test.py

+            data = mx.gluon.utils.split_and_load(batch, ctx_list=args.ctx, batch_axis=0, even_split=False)
+            outputs = None
+            for x in data:
+                output = model.forward(x)


why not using output=model(x)

zhanghang1989 · 2019-07-23T18:54:42Z

scripts/segmentation/test.py

    print(model)
-    evaluator = MultiEvalModel(model, testset.num_class, ctx_list=args.ctx)


This would break the exisiting multi-size evaluation code. Could you change this script to other filename instead of overwriting test.py?

Since quantized models are generated based on fixed crop_size, so I cannot use MultiEvalModel class here. Can we add a new function or a new script for INT8 inference? @xinyu-intel

can use a separate function ‘def quantized_test’

xinyu-intel · 2019-07-23T22:10:32Z

tests/unittests/test_model_zoo.py

+@unittest.skip("temporarily disabled to fallback to non-mkl version")
+@with_cpu(0)
+def test_quantized_fcn_models():
+    model_list = ['fcn_resnet101_voc_int8', 'fcn_resnet101_coco_int8-symbol']


maybe model_list = ['fcn_resnet101_voc_int8', 'fcn_resnet101_coco_int8']

xinyu-intel

.

wuxun-zhang · 2019-07-24T06:53:17Z

@zhanghang1989 Already added a separated test function for quantized model in test.py. In eval mode, I can get the below error by using original script. So I did some changes for this.
Also, in test mode, I would expect the outputs are segemented images with different mask colors. I'm not sure if such changes meet your expectations. Thanks.

Traceback (most recent call last):
  File "/home/wuxunzha/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/wuxunzha/anaconda3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/wuxunzha/github/gluon-cv/gluoncv/utils/metrics/segmentation.py", line 34, in evaluate_worker
    pred, label, self.nclass)
  File "/home/wuxunzha/github/gluon-cv/gluoncv/utils/metrics/segmentation.py", line 98, in batch_intersection_union
    predict = predict * (target > 0).astype(predict.dtype)
ValueError: operands could not be broadcast together with shapes (21,480) (480,480)

wuxun-zhang · 2019-07-26T01:36:41Z

@zhanghang1989 Could you please help take a review at this PR again? Thanks

zhanghang1989 · 2019-07-26T03:05:37Z

@zhanghang1989 Could you please help take a review at this PR again? Thanks

Could you keep the test function the same as the original to avoid introducing potential issues?

wuxun-zhang · 2019-07-26T05:50:37Z

@zhanghang1989 I will just put get_model() function outside test function, which can be reused for test_quantization function. Removed other changes done for test function.

mli · 2019-07-26T06:48:16Z

Job PR-877-7 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-877/7/index.html
Code coverage of this PR: vs. Master:

xinyu-intel

LGTM. @zhanghang1989 Can you help review again?

zhanghang1989 · 2019-07-29T16:39:33Z

@zhreshold @hetong007 @Jerryzcn Do we need to put json files into GluonCV? This PR has 37K lines of files.

hetong007 · 2019-07-29T16:56:00Z

My two cents on the json issue:

It won't be a scalable solution if the community plans to support all models in GluonCV. The size of this repository will largely increase and that is going to harm the user experience.
In terms of implementation and maintenance convenience, the descending order of places to host json is: this repo > public S3 > another repo (e.g. dmlc/web-data).
The issue of uploading to the public S3 is basically security consideration. @zhreshold may have more experience with it.
The issue of putting into another repo is the stability of github, and the risk of using absolute paths that are hard to maintain.

zhreshold · 2019-07-29T18:37:26Z

there's indeed a controvercy between portability/availability against the maintainance difficulty.
As the latest json files are more than 500kb each, it will likely explode the loc in this repo quickly if we plan to add more.

However, I still prefer the model definition should remain in the gluon-cv repo, as this behavior is aligned with python definitions in model_zoo

My suggestion is to utilize inline python json.dumps and zlib.compress and as a general rule of thumb, the json files can be compressed to 20% the original sizes.

The workflow will be:

Encode: 
[original json] -> json.dumps() -> [\{\n"input":"data"\n}] -> zlib.compress() -> [b'x\x9c\xabVJT\xb2RPrT\xd2QPJ] -> b64encode() -> [b'eJyrVkpUslJQclTSUVBK] human readable string -> store in a python dict

Decode:
Lookup the model name in python dict -> b64decode() -> zlib.decompress -> json.loads

This way we will have compact int8 model definitions line by line and stored elegantly in code.
What do you guys think?

hetong007 · 2019-07-29T21:48:47Z

This proposal looks good to me at this stage.

zhanghang1989 · 2019-07-29T21:51:14Z

That sounds good.

hetong007 · 2019-07-30T23:12:07Z

My suggestion is to utilize inline python json.dumps and zlib.compress and as a general rule of thumb, the json files can be compressed to 20% the original sizes.

The workflow will be:
Encode: 
[original json] -> json.dumps() -> [\{\n"input":"data"\n}] -> zlib.compress() -> [b'x\x9c\xabVJT\xb2RPrT\xd2QPJ] -> b64encode() -> [b'eJyrVkpUslJQclTSUVBK] human readable string -> store in a python dict

Decode:
Lookup the model name in python dict -> b64decode() -> zlib.decompress -> json.loads

@wuxun-zhang @xinyu-intel Please refer to above discussion for our suggested modifications.
Basically we would like to have a compressed version for all json files in order to keep the repository size manageable.

This is the first PR that comes with a significantly large json file, so please implement the proposed internal compression and decompression apis into gluoncv.utils, and include the compressed json.

Once we have the APIs, please have the other existing json files compressed in another PR. Thanks!

xinyu-intel · 2019-07-31T00:48:06Z

@hetong007 Okay

wuxun-zhang · 2019-08-02T02:27:45Z

Thanks @zhreshold for help. Now use compressed string instead of raw json file. @zhanghang1989 Please also help take a review. Thanks.

mli · 2019-08-02T02:33:17Z

Job PR-877-10 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-877/10/index.html
Code coverage of this PR: vs. Master:

hetong007 · 2019-08-02T17:55:41Z

gluoncv/utils/compress_json.py

+import zlib
+import base64
+
+_compressed_int8_json = {


I feel we should have the compressed string in another file or other files under model_zoo. Utils are utils, and different from specific model definitions.

@zhreshold what do you suggest?

agree, model_zoo is a better place for compressed strings

@wuxun-zhang Let's put the strings under gluoncv/model_zoo/quantized/. In the next PR we can replace all the current .json files with compressed strings.

xinyu-intel · 2019-08-04T11:42:24Z

@wuxun-zhang Please rebase code and enable your skipped ut since #863 has been merged:)

mli · 2019-08-05T06:55:58Z

Job PR-877-16 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-877/16/index.html
Code coverage of this PR: vs. Master:

wuxun-zhang · 2019-08-05T06:56:46Z

@hetong007 @zhreshold Now CI has passed. Please take a look again. Thanks.

zhreshold · 2019-08-05T18:25:42Z

gluoncv/model_zoo/quantized/quantized.py

        with warnings.catch_warnings(record=True) as w:
            warnings.simplefilter("always")
+            import tempfile
+            if "fcn" in model_name:


Looking for elimination of json files completely!

zhreshold · 2019-08-05T18:27:04Z

Looks good to me as a transition PR.

xinyu-intel self-requested a review July 22, 2019 08:51

xinyu-intel reviewed Jul 22, 2019

View reviewed changes

scripts/segmentation/eval_segmentation.py Outdated

return args

def test(args, model):

Copy link

Member

xinyu-intel Jul 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also add a dummy benchmark like ssd.

xinyu-intel requested changes Jul 22, 2019

View reviewed changes

hetong007 requested a review from zhanghang1989 July 22, 2019 22:02

zhanghang1989 suggested changes Jul 23, 2019

View reviewed changes

wuxun-zhang force-pushed the int8-fcn branch 2 times, most recently from c39ea0c to e599ba0 Compare July 23, 2019 13:24

wuxun-zhang force-pushed the int8-fcn branch from e599ba0 to ab1e7b9 Compare July 23, 2019 13:47

zhanghang1989 reviewed Jul 23, 2019

View reviewed changes

xinyu-intel reviewed Jul 23, 2019

View reviewed changes

xinyu-intel requested changes Jul 23, 2019

View reviewed changes

wuxun-zhang force-pushed the int8-fcn branch from a743c04 to 0e35083 Compare July 26, 2019 06:26

xinyu-intel approved these changes Jul 29, 2019

View reviewed changes

wuxun-zhang force-pushed the int8-fcn branch from 8126725 to a40db9a Compare August 2, 2019 02:12

zhanghang1989 approved these changes Aug 2, 2019

View reviewed changes

hetong007 requested changes Aug 2, 2019

View reviewed changes

wuxun-zhang force-pushed the int8-fcn branch from 1c191a1 to cf4fe1e Compare August 4, 2019 07:45

wuxun-zhang added 8 commits August 4, 2019 20:53

add quantized FCN model

d3f4d45

address comments

c3599c2

add separate test function for quantized model

cd03df0

address comments

5682cbe

adopt compressed json

1a56936

fix pylint

99e8749

fix comments

3173318

enable fcn ut

c5fd833

wuxun-zhang force-pushed the int8-fcn branch from cf9604b to c5fd833 Compare August 4, 2019 12:58

wuxun-zhang added 2 commits August 4, 2019 22:18

trigger CI

e7e4adb

fix python2 issue

d81fec6

zhreshold reviewed Aug 5, 2019

View reviewed changes

zhreshold approved these changes Aug 5, 2019

View reviewed changes

hetong007 approved these changes Aug 5, 2019

View reviewed changes

hetong007 merged commit baedaf8 into dmlc:master Aug 5, 2019

wuxun-zhang deleted the int8-fcn branch August 6, 2019 09:12

wuxun-zhang mentioned this pull request Aug 20, 2019

fix segmentation testing and support CPU #888

Open

		print(model)
		evaluator = MultiEvalModel(model, testset.num_class, ctx_list=args.ctx)

Add quantized Fully Convolutional Network model #877

Add quantized Fully Convolutional Network model #877

Conversation

wuxun-zhang commented Jul 22, 2019

mli commented Jul 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinyu-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wuxun-zhang commented Jul 23, 2019

mli commented Jul 23, 2019

mli commented Jul 23, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinyu-intel left a comment

Choose a reason for hiding this comment

wuxun-zhang commented Jul 24, 2019

wuxun-zhang commented Jul 26, 2019

zhanghang1989 commented Jul 26, 2019

wuxun-zhang commented Jul 26, 2019

mli commented Jul 26, 2019

xinyu-intel left a comment

Choose a reason for hiding this comment

zhanghang1989 commented Jul 29, 2019 • edited Loading

hetong007 commented Jul 29, 2019

zhreshold commented Jul 29, 2019

hetong007 commented Jul 29, 2019

zhanghang1989 commented Jul 29, 2019

hetong007 commented Jul 30, 2019

xinyu-intel commented Jul 31, 2019

wuxun-zhang commented Aug 2, 2019

mli commented Aug 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinyu-intel commented Aug 4, 2019

mli commented Aug 5, 2019

wuxun-zhang commented Aug 5, 2019

Choose a reason for hiding this comment

zhreshold commented Aug 5, 2019

zhanghang1989 commented Jul 29, 2019 •

edited

Loading