chore: Upgrade TensorRT version to TRT 10 EA #2699

peri044 · 2024-03-19T23:47:01Z

Description

Upgrade TensorRT version to TRT 10 EA

Type of change

Please delete options that are not relevant and/or add your own.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

My code follows the style guidelines of this project (You can use the linters)
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas and hacks
I have made corresponding changes to the documentation
I have added tests to verify my fix or my feature
New and existing unit tests pass locally with my changes
I have added the relevant labels to my PR in so that relevant reviewers are notified

chore: updates to trt api chore: trt 10 fixes chore: more fixes

author Dheeraj Peri <peri.dheeraj@gmail.com> 1711393059 -0700 committer Dheeraj Peri <peri.dheeraj@gmail.com> 1711393072 -0700 chore: minor updates chore: Fix save failures chore: minor fixes chore: remove duplicate bert test case chore: remove comments chore: add load api chore: minor updates chore: minor updates chore: minor updates chore: more updates

cpp/include/torch_tensorrt/ptq.h

peri044 · 2024-04-20T06:39:43Z

py/torch_tensorrt/dynamo/runtime/_PythonTorchTensorRTModule.py

+        # self.hidden_output_names: Sequence[str] = []
+        # for i in range(
+        #     self.engine.num_bindings // self.engine.num_optimization_profiles
+        # ):
+        #     if i not in primary_input_outputs:
+        #         self.hidden_output_binding_indices_in_order.append(i)
+        #         self.hidden_output_names.append(self.engine.get_binding_name(i))


review needed

peri044 · 2024-04-20T06:39:54Z

py/torch_tensorrt/dynamo/runtime/_PythonTorchTensorRTModule.py

            len(self.input_names)
            + len(self.output_names)
-            + len(self.hidden_output_names)
+            # + len(self.hidden_output_names) #TODO: Verify if this is required


review needed

peri044 · 2024-04-20T06:40:28Z

py/torch_tensorrt/dynamo/runtime/_PythonTorchTensorRTModule.py

+        # TODO: Verify what this is for ?
+        # self.hidden_output_dtypes = [
+        #     unified_dtype_converter(
+        #         self.engine.get_binding_dtype(idx), Frameworks.TORCH
+        #     )
+        #     for idx in self.hidden_output_binding_indices_in_order
+        # ]
+


review needed

peri044 · 2024-04-20T06:44:18Z

py/torch_tensorrt/dynamo/runtime/_PythonTorchTensorRTModule.py

+                # TODO: Check what is this for ?
+                # for i, idx in enumerate(self.hidden_output_binding_indices_in_order):
+                #     shape = tuple(self.context.get_binding_shape(idx))


review needed

narendasan · 2024-04-22T17:30:58Z

py/torch_tensorrt/_enums.py

@@ -107,7 +107,7 @@ def _from(
                return dtype.f16
            elif t == trt.float32:
                return dtype.f32
-            elif trt.__version__ >= "7.0" and t == trt.bool:


Why are you removing the version check?

This version check actually fails. If trt.__version == 10.0.0b6, then trt.__version__ >= "7.0" is False (which should actually be true) and hence the trt.bool type wouldn't be returned which results in type errors.
We also don't need these version checks because we only support strict TRT versions.

py/torch_tensorrt/dynamo/runtime/_PythonTorchTensorRTModule.py

gs-olive

Overall looks great - added a comment in the TRTEngine and wanted to note that the Docker build + Docker build jobs will likely fail since the TRT installation will need upgrading for the build, but that can be a separate PR.

core/runtime/TRTEngine.cpp

peri044 · 2024-04-22T22:59:33Z

Docker build + Docker build jobs will likely fail since the TRT installation will need upgrading for the build,

Yes, I'm working on docker updates in fp8_trt branch. So that will likely get merged with this PR #2763

Co-authored-by: Evan Li <zewenl@nvidia.com>

Co-authored-by: Evan Li <zewenl@nvidia.com> Co-authored-by: George S <113141689+gs-olive@users.noreply.github.com>

Co-authored-by: Evan Li <zewenl@nvidia.com>

zewenli98

Need some patches as comments.

zewenli98 · 2024-05-03T23:35:09Z

py/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py

@@ -312,7 +313,7 @@ def run(
        )
        timing_cache = self._create_timing_cache(builder_config, existing_cache)

-        engine = self.builder.build_engine(self.ctx.net, builder_config)
+        engine = self.builder.build_serialized_network(self.ctx.net, builder_config)


build_serialized_network returns a plan instead of engine, which is different from previous TRT. In TRT-10, we can do two steps to get an engine:

plan = builder.build_serialized_network(network, config) engine = runtime.deserialize_cuda_engine(plan)

If it is on purpose to get plan, maybe we can change the name to make it easy understand.

zewenli98 · 2024-05-03T23:39:44Z

py/torch_tensorrt/dynamo/runtime/_PythonTorchTensorRTModule.py

+        assert (
+            self.engine.num_io_tensors // self.engine.num_optimization_profiles
+        ) == (len(self.input_names) + len(self.output_names))


The assertion should be:

assert self.engine.num_io_tensors == (len(self.input_names) + len(self.output_names))

A bug was reported here: #2811

Co-authored-by: Evan Li <zewenl@nvidia.com> Co-authored-by: George S <113141689+gs-olive@users.noreply.github.com>

peri044 added 6 commits March 14, 2024 02:01

feat: Add save API for torch-trt compiled models

cd86660

chore: resolve merge conflicts

3ece71b

chore: Fix save failures

eab0dba

chore: update to 2.3 rc build

b191d62

chore: rebase with release/2.3 branch

ce606fe

chore: minor fixes

8674a3c

facebook-github-bot added the cla signed label Mar 19, 2024

github-actions bot requested a review from gs-olive March 19, 2024 23:47

peri044 added 3 commits March 19, 2024 17:04

chore: remove duplicate bert test case

f4e8fe9

chore: remove comments

4ae6ab9

chore: Upgrade to TRT 10.0

fff1b80

chore: updates to trt api chore: trt 10 fixes chore: more fixes

peri044 force-pushed the trt_10 branch from 8faf7ec to fff1b80 Compare March 20, 2024 22:56

github-actions bot added component: tests Issues re: Tests component: fx labels Mar 20, 2024

peri044 added 2 commits March 21, 2024 16:53

chore: more fixes

39ca77d

chore: update trt version

5431ee3

github-actions bot added the component: build system Issues re: Build system label Mar 25, 2024

peri044 force-pushed the save branch 3 times, most recently from ff95381 to 8980b7a Compare March 25, 2024 21:58

peri044 added 2 commits March 25, 2024 18:51

chore: more updates

0c03de5

peri044 force-pushed the save branch from 8980b7a to 982dbd2 Compare March 26, 2024 01:57

chore: revert layer_norm test

d78a846

peri044 force-pushed the trt_10 branch from d285d27 to d78a846 Compare April 16, 2024 01:40

peri044 and others added 9 commits April 15, 2024 20:51

chore: rebase

ba8a424

Merge branch 'trt_10' of github.com:pytorch/TensorRT into trt_10

097d887

chore: rebase with release/2.3

ffe7a52

chore: updates

38642bb

chore: updates

d15dd72

chore: updates

dee9aa0

chore: update stream in python runtime

c05d675

chore: update hw_compat.ts

2329657

chore: updates

b8a8709

peri044 commented Apr 20, 2024

View reviewed changes

cpp/include/torch_tensorrt/ptq.h Outdated Show resolved Hide resolved

peri044 commented Apr 20, 2024

View reviewed changes

chore: updates

44778e1

github-actions bot removed the component: api [C++] Issues re: C++ API label Apr 20, 2024

chore: updates

0dbbcd7

github-actions bot added the component: api [C++] Issues re: C++ API label Apr 20, 2024

narendasan reviewed Apr 22, 2024

View reviewed changes

gs-olive reviewed Apr 22, 2024

View reviewed changes

core/runtime/TRTEngine.cpp Show resolved Hide resolved

core/runtime/TRTEngine.cpp Show resolved Hide resolved

chore: updates

89c3d76

peri044 merged commit 32a6e67 into release/2.3 Apr 23, 2024
23 checks passed

peri044 added a commit that referenced this pull request Apr 23, 2024

chore: Upgrade TensorRT version to TRT 10 EA (#2699)

2d6ffb4

Co-authored-by: Evan Li <zewenl@nvidia.com>

peri044 added a commit that referenced this pull request Apr 25, 2024

chore: Upgrade TensorRT version to TRT 10 EA (#2699) (#2774)

1a11f86

Co-authored-by: Evan Li <zewenl@nvidia.com> Co-authored-by: George S <113141689+gs-olive@users.noreply.github.com>

zewenli98 added a commit that referenced this pull request Apr 26, 2024

chore: Upgrade TensorRT version to TRT 10 EA (#2699)

29e1380

Co-authored-by: Evan Li <zewenl@nvidia.com>

zewenli98 reviewed May 4, 2024

View reviewed changes

laikhtewari pushed a commit that referenced this pull request May 24, 2024

chore: Upgrade TensorRT version to TRT 10 EA (#2699) (#2774)

81e9f4d

Co-authored-by: Evan Li <zewenl@nvidia.com> Co-authored-by: George S <113141689+gs-olive@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Upgrade TensorRT version to TRT 10 EA #2699

chore: Upgrade TensorRT version to TRT 10 EA #2699

peri044 commented Mar 19, 2024 •

edited

Loading

peri044 Apr 20, 2024

peri044 Apr 20, 2024

peri044 Apr 20, 2024

peri044 Apr 20, 2024

narendasan Apr 22, 2024

peri044 Apr 22, 2024

gs-olive left a comment

peri044 commented Apr 22, 2024

zewenli98 left a comment

zewenli98 May 3, 2024

zewenli98 May 3, 2024

chore: Upgrade TensorRT version to TRT 10 EA #2699

chore: Upgrade TensorRT version to TRT 10 EA #2699

Conversation

peri044 commented Mar 19, 2024 • edited Loading

Description

Type of change

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gs-olive left a comment

Choose a reason for hiding this comment

peri044 commented Apr 22, 2024

zewenli98 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peri044 commented Mar 19, 2024 •

edited

Loading