QAT recommendation

[shuyuan-wang]

In the document docs, It recommends not change quantization representation (scale) during training, at least not too frequently. How exactly do I not change scale during QAT training?

[zerollzeng]

@ttyio ^ ^ I would also like to learn about this

[ttyio]

@shuyuan-wang ,
see https://github.com/NVIDIA/TensorRT/blob/main/tools/pytorch-quantization/examples/torchvision/classification_flow.py#L387, the scale is changed during calibration.

for name, module in model.named_modules():
    if isinstance(module, quant_nn.TensorQuantizer):
        if module._calibrator is not None:
            module.disable_quant()
            module.enable_calib()
        else:
            module.disable()

By call

   module.disable_calib()

The calibration is turned off, and you can call

   module.enable_quant()

to do fine-tuning without change scale.

[shuyuan-wang]

Thanks

[zsh4614]

@shuyuan-wang , see https://github.com/NVIDIA/TensorRT/blob/main/tools/pytorch-quantization/examples/torchvision/classification_flow.py#L387, the scale is changed during calibration.

for name, module in model.named_modules():
    if isinstance(module, quant_nn.TensorQuantizer):
        if module._calibrator is not None:
            module.disable_quant()
            module.enable_calib()
        else:
            module.disable()

By call

   module.disable_calib()

The calibration is turned off, and you can call

   module.enable_quant()

to do fine-tuning without change scale.

Sorry, I have a question. During the fine-tuning process of QAT, the model weights will change. If the scale of the weights remains unchanged, is the scale calculated during calibration still reasonable? Should the scale never be updated during fine-tuning? If it can be updated, how should the scale be updated during training? Are there any recommended update strategies?

[shuyuan-wang]

@shuyuan-wang , see https://github.com/NVIDIA/TensorRT/blob/main/tools/pytorch-quantization/examples/torchvision/classification_flow.py#L387, the scale is changed during calibration.

for name, module in model.named_modules():
    if isinstance(module, quant_nn.TensorQuantizer):
        if module._calibrator is not None:
            module.disable_quant()
            module.enable_calib()
        else:
            module.disable()

By call

   module.disable_calib()

The calibration is turned off, and you can call

   module.enable_quant()

to do fine-tuning without change scale.

Sorry, I have a question. During the fine-tuning process of QAT, the model weights will change. If the scale of the weights remains unchanged, is the scale calculated during calibration still reasonable? Should the scale never be updated during fine-tuning? If it can be updated, how should the scale be updated during training? Are there any recommended update strategies?

I would say during QAT, the weight is adjusted based on the the newly calculated scale during PTQ