INT8 calibration for efficientdet

[srihari-humbarwadi]

From the readme

Where --calib_input points to a directory with several thousands of images. For example,
this could be a subset of the training or validation datasets that were used for the model. 
It's important that this data represents the runtime data distribution relatively well,  therefore,
the more images that are used for calibration, the better accuracy that will be achieved in
INT8 precision. For models  trained for the COCO dataset, we have found that 5,000 images gives a good result.

It is not clear how exactly these 5000 images were sampled. Are these 5,000 random images from coco2017 train dataset
or is it the entire coco2017 val dataset which has exactly 5000 images?
Also, can you please share the FP32 vs INT8 mAP scores for the models?

[ttyio]

Hello @srihari-humbarwadi ,

It is not clear how exactly these 5000 images were sample

could you check source code https://github.com/NVIDIA/TensorRT/blob/release/8.0/samples/python/efficientdet/image_batcher.py

can you please share the FP32 vs INT8 mAP scores for the models?

The author of the sample has not provide the scores yet, could you follow the instruction to collect mAP?
https://github.com/NVIDIA/TensorRT/tree/release/8.0/samples/python/efficientdet#evaluate-map-metric

[srihari-humbarwadi]

TensorRT/samples/python/efficientdet/image_batcher.py

Lines 24 to 32 in 834cb6d

	class ImageBatcher:
	"""
	Creates batches of pre-processed images.
	"""
	def __init__(self, input, shape, dtype, max_num_images=None, exact_batches=False, preprocessor="EfficientDet"):
	"""
	:param input: The input directory to read images from.
	:param shape: The tensor shape of the batch to prepare, either in NCHW or NHWC format.

I went through the code, and also the readme which has a brief mention of it
"For models trained for the COCO dataset, we have found that 5,000 images gives a good result."
but it is not clear how are the 5,000 images for the calibration dataset are chosen!

[nvpohanh]

but it is not clear how are the 5,000 images for the calibration dataset are chosen!

It is usually by experiments. There is no general guideline on how many images is enough.

[ttyio]

closing due to no activity for more than 3 weeks, please reopen if you still have question, thanks!