Fix Inception transform_input to use Float tensors

[apivovarov]

This PR make inveption_v3 model to be compatible with torch_tensorrt compilation.
Related issue: pytorch/TensorRT#1096

Problem:
The following statements in _transform_input method

x_ch0 = torch.unsqueeze(x[:, 0], 1) * (0.229 / 0.5) + (0.485 - 0.5) / 0.5

is converted into Double tensor in the traced graph

%1512 : Double(requires_grad=0, device=cpu) = prim::Constant[value={0.458}]()

BUT Double is not supported by torch_tensorrt compiler - Error: Unsupported ATen data type Double

The fix changes Double to Float by wrapping constants with torch.tensor() (which is float32 by default)

x_ch0 = torch.unsqueeze(x[:, 0], 1) * torch.tensor(0.229 / 0.5) + torch.tensor((0.485 - 0.5) / 0.5)

As a result the traced graph has Float node now

%1502 : Float(requires_grad=0, device=cpu) = prim::Constant[value={0.458}]()

Now, traced inception_v3 model can be compiled by torch_tensorrt.

@datumbox @NicolasHug @soumith @BowenBao

[soumith]

I think the right thing to do is to add a strength reduction pass in torch_tensorrt.
How many models are you going to change without fixing the root cause?

[apivovarov]

I found how to replace Double with Float in traced model graph

tmodel = torch.jit.trace(model, x)

float_tensor_type=torch._C.TensorType.create_from_tensor(torch.tensor(2.33))
constant_nodes = tmodel.graph.findAllNodes("prim::Constant", recurse=True)
for node in constant_nodes:
    for out in node.outputs():
        if isinstance(out.type(), torch._C.TensorType) and out.type().scalarType() == "Double":
            out.setType(float_tensor_type)

But torch_tensorrt compiler uses tmodel._c instead of tmodel.graph torch_tensorrt/ts/_compiler.py

tmodel._c.code_with_constants still shows float64 tensors.

{'c0': tensor(0.4580, dtype=torch.float64), 'c1': tensor(-0.0300, dtype=torch.float64), 'c2': tensor(0.4480, dtype=torch.float64), 'c3': tensor(-0.0880, dtype=torch.float64), 'c4': tensor(0.4500, dtype=torch.float64), 'c5': tensor(-0.1880, dtype=torch.float64)})

How to properly replace float64 constants with float32 constants in traced model?

Get I get some info or links about strength reduction pass?

[datumbox]

@apivovarov Thanks for the PR. Apologies for the delayed response, there were a few bank holidays in the UK.

I have very limited experience working with TensorRT but I agree with @soumith that this looks like something that needs to be addressed outside of TorchVision. Note that the specific model is quite old and uses a few idioms that we no longer use (such as preprocessing inside the model). Nevertheless, being able to do math operations between tensors and python numbers is something extremely common and I don't think the solution is to modify the entire code-base and promote the use of tensors. This will lead to a rather verbose and potentially ugly code-base. Note that we already have to do tricks like that for JIT and FX and they often conflict with each other (ONNX, TensorRT etc). As you probably imagine, the more workarounds we add, the harder maintaining and developing on the code-base becomes.

[datumbox]

Note that here you might be combining GPU tensors with CPU tensors.