Class DataType¶
+Class DataType¶
Defined in File torch_tensorrt.h
Class Documentation¶
+Class Documentation¶
-
-class torch_tensorrt::DataType¶
Supported Data Types that can be used with TensorRT engines
This class is compatable with c10::DataTypes (but will check for TRT support) so there should not be a reason that you need to use this type explictly.
@@ -514,7 +517,7 @@Class Documentation
- -inline operator Value() const¶
+inline operator Value() const¶
Get the enum value of the DataType object.
- Returns @@ -530,64 +533,68 @@
-
-inline constexpr bool operator==(DataType other) const¶
Comparision operator for DataType.
- Parameters
other –
- Returns -
true
-false
- +true
+- Returns +
false
Class Documentation
-
-inline constexpr bool operator==(DataType::Value other) const¶
Comparision operator for DataType.
- Parameters
other –
- Returns -
true
-false
- +true
+- Returns +
false
-
-inline constexpr bool operator!=(DataType other) const¶
Comparision operator for DataType.
- Parameters
other –
- Returns -
true
-false
- +true
+- Returns +
false
-
-inline constexpr bool operator!=(DataType::Value other) const¶
Comparision operator for DataType.
- Parameters
other –
- Returns -
true
-false
- +true
+- Returns +
false
Class Documentation + diff --git a/docs/_cpp_api/classtorch__tensorrt_1_1Device_1_1DeviceType.html b/docs/_cpp_api/classtorch__tensorrt_1_1Device_1_1DeviceType.html index ed6f849578..ea4225da19 100644 --- a/docs/_cpp_api/classtorch__tensorrt_1_1Device_1_1DeviceType.html +++ b/docs/_cpp_api/classtorch__tensorrt_1_1Device_1_1DeviceType.html @@ -10,7 +10,7 @@ -
Class Device::DeviceType — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Class Device::DeviceType — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion
+- Using Custom Kernels within TensorRT Engines with Torch-TensorRT
+- Wrapping Custom Kernels to use in TensorRT
+- Using Torch-TensorRT to Insert the Kernel
Python API Documenation
-
@@ -416,19 +419,19 @@
Defined in File torch_tensorrt.h
-
-class torch_tensorrt::Device::DeviceType¶
Supported Device Types that can be used with TensorRT engines
This class is compatable with c10::DeviceTypes (but will check for TRT support) but the only applicable value is at::kCUDA, which maps to DeviceType::kGPU
To use the DataType class itself, interface using the enum vs. normal instatination
@@ -484,7 +487,7 @@Class Documentation
- -inline operator Value() const¶
+inline operator Value() const¶
Get the internal value from the Device object.
- Returns @@ -500,32 +503,34 @@
-
-inline constexpr bool operator==(DeviceType other) const¶
Comparison operator for DeviceType.
- Parameters
other –
- Returns -
true
-false
- +true
+- Returns +
false
Class Documentation
-
-inline constexpr bool operator!=(DeviceType other) const¶
Comparison operator for DeviceType.
- Parameters
other –
- Returns -
true
-false
- +true
+- Returns +
false
Class Documentation + diff --git a/docs/_cpp_api/classtorch__tensorrt_1_1TensorFormat.html b/docs/_cpp_api/classtorch__tensorrt_1_1TensorFormat.html index b6178ac30f..29ad45b2f6 100644 --- a/docs/_cpp_api/classtorch__tensorrt_1_1TensorFormat.html +++ b/docs/_cpp_api/classtorch__tensorrt_1_1TensorFormat.html @@ -10,7 +10,7 @@ -
Class TensorFormat — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Class TensorFormat — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -39,7 +39,7 @@ - + + diff --git a/docs/_cpp_api/classtorch__tensorrt_1_1ptq_1_1Int8CacheCalibrator.html b/docs/_cpp_api/classtorch__tensorrt_1_1ptq_1_1Int8CacheCalibrator.html index 7fb52ad6d6..b2cf2df2e9 100644 --- a/docs/_cpp_api/classtorch__tensorrt_1_1ptq_1_1Int8CacheCalibrator.html +++ b/docs/_cpp_api/classtorch__tensorrt_1_1ptq_1_1Int8CacheCalibrator.html @@ -10,7 +10,7 @@ -Template Class Int8CacheCalibrator — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Template Class Int8CacheCalibrator — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion
+- Using Custom Kernels within TensorRT Engines with Torch-TensorRT
+- Wrapping Custom Kernels to use in TensorRT
+- Using Torch-TensorRT to Insert the Kernel
- -inline operator Value() const¶
- Class Device::DeviceType¶
+Class Device::DeviceType¶
- Nested Relationships¶
+Nested Relationships¶
This class is a nested type of Struct Device.
- Class Documentation¶
+Class Documentation¶
Python API Documenation
-
@@ -416,28 +419,28 @@
Defined in File ptq.h
private Algorithm-
-template<typename Algorithm>
class torch_tensorrt::ptq::Int8CacheCalibrator : private Algorithm¶
+template<typename Algorithm> Generic Int8Calibrator implementation based on a specified TensorRT calibration algorithm that only reads from a calibration file.
- Template Parameters -
Algorithm, : – class nvinfer1::IInt8Calibrator (Default: nvinfer1::IInt8EntropyCalibrator2) - Algorithm to use
+Algorithm – class nvinfer1::IInt8Calibrator (Default: nvinfer1::IInt8EntropyCalibrator2) - Algorithm to use
@@ -454,8 +457,8 @@Class Documentation -
- -inline int getBatchSize () const noexceptoverride
+- +inline int getBatchSize() const noexcept override¶
Get the Batch Size for the next batch (always 1 due to issues with TRT and explicit batch)
- Returns @@ -465,16 +468,16 @@
- -inline bool getBatch (void *bindings[], const char *names[], int nbBindings) noexceptoverride +
-
+inline bool getBatch(void *bindings[], const char *names[], int nbBindings) noexcept override¶
Get the next Batch.
Not used always returns false
- Parameters
-
-
bindings, : – void*[] - An array of binding pointers (fed in from TensorRT calibrator), these buffers should be filed with batch data for each input
-names, : – const char*[] - Names of bindings
-nbBindings, : – int - Number of bindings
+bindings – void*[] - An array of binding pointers (fed in from TensorRT calibrator), these buffers should be filed with batch data for each input
+names – const char*[] - Names of bindings
+nbBindings – int - Number of bindings
- Returns @@ -484,8 +487,8 @@
- -inline const void * readCalibrationCache (size_t &length) noexceptoverride +
-
+inline const void *readCalibrationCache(size_t &length) noexcept override¶
Read calibration cache.
How to read from the calibration cache, only enabled if use_cache is set
-
@@ -499,15 +502,15 @@
- -inline void writeCalibrationCache (const void *cache, size_t length) noexceptoverride +
-
+inline void writeCalibrationCache(const void *cache, size_t length) noexcept override¶
Write calibration cache.
Write a the calibration cache provided by TensorRT to a specified file
- Parameters
-
-
cache, : – const void* - cache data
-length, : – size_t - length of cache
+cache – const void* - cache data
+length – size_t - length of cache
Class Documentation + diff --git a/docs/_cpp_api/classtorch__tensorrt_1_1ptq_1_1Int8Calibrator.html b/docs/_cpp_api/classtorch__tensorrt_1_1ptq_1_1Int8Calibrator.html index b1166a3d68..ca9ce1f6d3 100644 --- a/docs/_cpp_api/classtorch__tensorrt_1_1ptq_1_1Int8Calibrator.html +++ b/docs/_cpp_api/classtorch__tensorrt_1_1ptq_1_1Int8Calibrator.html @@ -10,7 +10,7 @@ -
Template Class Int8Calibrator — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Template Class Int8Calibrator — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion
+- Using Custom Kernels within TensorRT Engines with Torch-TensorRT
+- Wrapping Custom Kernels to use in TensorRT
+- Using Torch-TensorRT to Insert the Kernel
Python API Documenation
-
@@ -416,30 +419,30 @@
Defined in File ptq.h
private Algorithm-
-template<typename Algorithm, typename DataLoaderUniquePtr>
class torch_tensorrt::ptq::Int8Calibrator : private Algorithm¶
+template<typename Algorithm, typename DataLoaderUniquePtr> Generic Int8Calibrator implementation based on a specified TensorRT calibration algorithm and a LibTorch DataLoader.
- Template Parameters
-
-
Algorithm, : – class nvinfer1::IInt8Calibrator (Default: nvinfer1::IInt8EntropyCalibrator2) - Algorithm to use
-DataLoaderUniquePtr, : – std::unique_ptr<torch::data::DataLoader> - DataLoader type
+Algorithm – class nvinfer1::IInt8Calibrator (Default: nvinfer1::IInt8EntropyCalibrator2) - Algorithm to use
+DataLoaderUniquePtr – std::unique_ptr<torch::data::DataLoader> - DataLoader type
Class Documentation
- Parameters
-
-
dataloader, : – std::unqiue_ptr<torch::data::DataLoader> - A unique pointer to the DataLoader, should be what is returned from the make_data_loader factory
-cache_file_path, : – const std::string& - A path to store / find the calibration cache
+dataloader – std::unqiue_ptr<torch::data::DataLoader> - A unique pointer to the DataLoader, should be what is returned from the make_data_loader factory
+cache_file_path – const std::string& - A path to store / find the calibration cache
use_cache – : bool - Whether to use the cache (if it exists)
Class Documentation -
- -inline int getBatchSize () const noexceptoverride
+- +inline int getBatchSize() const noexcept override¶
Get the Batch Size for the next batch (always 1 due to issues with TRT and explicit batch)
- Returns @@ -473,28 +476,29 @@
- -inline bool getBatch (void *bindings[], const char *names[], int nbBindings) noexceptoverride +
-
+inline bool getBatch(void *bindings[], const char *names[], int nbBindings) noexcept override¶
Get the next Batch.
- Parameters
-
-
bindings, : – void*[] - An array of binding pointers (fed in from TensorRT calibrator), these buffers should be filed with batch data for each input
-names, : – const char*[] - Names of bindings
-nbBindings, : – int - Number of bindings
+bindings – void*[] - An array of binding pointers (fed in from TensorRT calibrator), these buffers should be filed with batch data for each input
+names – const char*[] - Names of bindings
+nbBindings – int - Number of bindings
- Returns -
true - There is a new batch for the calibrator to consume
-false - There is not a new batch for the calibrator to consume
- +true - There is a new batch for the calibrator to consume
+- Returns +
false - There is not a new batch for the calibrator to consume
Class Documentation -
-
-
- -inline const void * readCalibrationCache (size_t &length) noexceptoverride +
-
+inline const void *readCalibrationCache(size_t &length) noexcept override¶
Read calibration cache.
How to read from the calibration cache, only enabled if use_cache is set
-
@@ -508,15 +512,15 @@
- -inline void writeCalibrationCache (const void *cache, size_t length) noexceptoverride +
-
+inline void writeCalibrationCache(const void *cache, size_t length) noexcept override¶
Write calibration cache.
Write a the calibration cache provided by TensorRT to a specified file
- Parameters
-
-
cache, : – const void* - cache data
-length, : – size_t - length of cache
+cache – const void* - cache data
+length – size_t - length of cache
Class Documentation + diff --git a/docs/_cpp_api/define_macros_8h_1a18d295a837ac71add5578860b55e5502.html b/docs/_cpp_api/define_macros_8h_1a18d295a837ac71add5578860b55e5502.html index 1eea3e2ee1..b8f8a2f1f5 100644 --- a/docs/_cpp_api/define_macros_8h_1a18d295a837ac71add5578860b55e5502.html +++ b/docs/_cpp_api/define_macros_8h_1a18d295a837ac71add5578860b55e5502.html @@ -10,7 +10,7 @@ -
Define STR — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Define STR — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -40,7 +40,7 @@ - + + diff --git a/docs/_cpp_api/define_macros_8h_1a282fd3c0b1c3a215148ae372070e1268.html b/docs/_cpp_api/define_macros_8h_1a282fd3c0b1c3a215148ae372070e1268.html index d325d6a44d..fa1470f208 100644 --- a/docs/_cpp_api/define_macros_8h_1a282fd3c0b1c3a215148ae372070e1268.html +++ b/docs/_cpp_api/define_macros_8h_1a282fd3c0b1c3a215148ae372070e1268.html @@ -10,7 +10,7 @@ -Define TORCH_TENSORRT_PATCH_VERSION — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Define TORCH_TENSORRT_PATCH_VERSION — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion
+- Using Custom Kernels within TensorRT Engines with Torch-TensorRT
+- Wrapping Custom Kernels to use in TensorRT
+- Using Torch-TensorRT to Insert the Kernel
Class Documentation -
- Template Class Int8Calibrator¶
+Template Class Int8Calibrator¶
- Inheritance Relationships¶
+Inheritance Relationships¶
- Base Type¶
+Base Type¶
- Class Documentation¶
+Class Documentation¶
class Int8Calibrator : private Algorithm¶
Python API Documenation
-
@@ -416,12 +419,12 @@
Defined in File macros.h
-
TORCH_TENSORRT_PATCH_VERSION¶
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Define TORCH_TENSORRT_PATCH_VERSION¶
+Define TORCH_TENSORRT_PATCH_VERSION¶
- Define Documentation¶
+Define Documentation¶
Define Documentation + diff --git a/docs/_cpp_api/define_macros_8h_1a31398a6d4d27e28817afb0f0139e909e.html b/docs/_cpp_api/define_macros_8h_1a31398a6d4d27e28817afb0f0139e909e.html index 1c638f30a7..7f7880d6d0 100644 --- a/docs/_cpp_api/define_macros_8h_1a31398a6d4d27e28817afb0f0139e909e.html +++ b/docs/_cpp_api/define_macros_8h_1a31398a6d4d27e28817afb0f0139e909e.html @@ -10,7 +10,7 @@ -
Define TORCH_TENSORRT_MAJOR_VERSION — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Define TORCH_TENSORRT_MAJOR_VERSION — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@Python API Documenation
-
@@ -416,12 +419,12 @@
Defined in File macros.h
-
TORCH_TENSORRT_MAJOR_VERSION¶
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Define TORCH_TENSORRT_MAJOR_VERSION¶
+Define TORCH_TENSORRT_MAJOR_VERSION¶
- Define Documentation¶
+Define Documentation¶
Define Documentation + diff --git a/docs/_cpp_api/define_macros_8h_1a35703561b26b1a9d2738ad7d58b27827.html b/docs/_cpp_api/define_macros_8h_1a35703561b26b1a9d2738ad7d58b27827.html index 957dcda9b8..0c01cfbbe0 100644 --- a/docs/_cpp_api/define_macros_8h_1a35703561b26b1a9d2738ad7d58b27827.html +++ b/docs/_cpp_api/define_macros_8h_1a35703561b26b1a9d2738ad7d58b27827.html @@ -10,7 +10,7 @@ -
Define TORCH_TENSORRT_MINOR_VERSION — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Define TORCH_TENSORRT_MINOR_VERSION — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@Python API Documenation
-
@@ -416,12 +419,12 @@
Defined in File macros.h
-
TORCH_TENSORRT_MINOR_VERSION¶
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Define TORCH_TENSORRT_MINOR_VERSION¶
+Define TORCH_TENSORRT_MINOR_VERSION¶
- Define Documentation¶
+Define Documentation¶
Define Documentation + diff --git a/docs/_cpp_api/define_macros_8h_1abd1465eb38256d3f22cc1426b23d516b.html b/docs/_cpp_api/define_macros_8h_1abd1465eb38256d3f22cc1426b23d516b.html index 325f9523b2..8bff184716 100644 --- a/docs/_cpp_api/define_macros_8h_1abd1465eb38256d3f22cc1426b23d516b.html +++ b/docs/_cpp_api/define_macros_8h_1abd1465eb38256d3f22cc1426b23d516b.html @@ -10,7 +10,7 @@ -
Define TORCHTRT_API — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Define TORCHTRT_API — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@Python API Documenation
-
@@ -416,12 +419,12 @@
Defined in File macros.h
-
TORCHTRT_API¶
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Define TORCHTRT_API¶
+Define TORCHTRT_API¶
- Define Documentation¶
+Define Documentation¶
Define Documentation + diff --git a/docs/_cpp_api/define_macros_8h_1abe87b341f562fd1cf40b7672e4d759da.html b/docs/_cpp_api/define_macros_8h_1abe87b341f562fd1cf40b7672e4d759da.html index 07c461d7c8..90996b0315 100644 --- a/docs/_cpp_api/define_macros_8h_1abe87b341f562fd1cf40b7672e4d759da.html +++ b/docs/_cpp_api/define_macros_8h_1abe87b341f562fd1cf40b7672e4d759da.html @@ -10,7 +10,7 @@ -
Define XSTR — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Define XSTR — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@Python API Documenation
-
@@ -416,12 +419,12 @@
Defined in File macros.h
-
XSTR(x)¶
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Define XSTR¶
+Define XSTR¶
- Define Documentation¶
+Define Documentation¶
Define Documentation + diff --git a/docs/_cpp_api/define_macros_8h_1ad19939408f7be171a74a89928b36eb59.html b/docs/_cpp_api/define_macros_8h_1ad19939408f7be171a74a89928b36eb59.html index 857e831c07..074a703776 100644 --- a/docs/_cpp_api/define_macros_8h_1ad19939408f7be171a74a89928b36eb59.html +++ b/docs/_cpp_api/define_macros_8h_1ad19939408f7be171a74a89928b36eb59.html @@ -10,7 +10,7 @@ -
Define TORCHTRT_HIDDEN — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Define TORCHTRT_HIDDEN — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@Python API Documenation
-
@@ -416,12 +419,12 @@
Python API Documenation
-
@@ -416,12 +419,12 @@
Defined in File macros.h
-
TORCH_TENSORRT_VERSION¶
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Define TORCH_TENSORRT_VERSION¶
+Define TORCH_TENSORRT_VERSION¶
- Define Documentation¶
+Define Documentation¶
Define Documentation + diff --git a/docs/_cpp_api/dir_cpp.html b/docs/_cpp_api/dir_cpp.html index 17680f9a32..105c165493 100644 --- a/docs/_cpp_api/dir_cpp.html +++ b/docs/_cpp_api/dir_cpp.html @@ -10,7 +10,7 @@ -
Directory cpp — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Directory cpp — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@Python API Documenation
-
@@ -412,10 +415,10 @@
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Directory cpp¶
+Directory cpp¶
Directory path:
cpp- Subdirectories¶
+Subdirectories¶
@@ -478,6 +481,7 @@Subdirectories + diff --git a/docs/_cpp_api/dir_cpp_include.html b/docs/_cpp_api/dir_cpp_include.html index e8d2c4279d..a885f6bea2 100644 --- a/docs/_cpp_api/dir_cpp_include.html +++ b/docs/_cpp_api/dir_cpp_include.html @@ -10,7 +10,7 @@ -
Directory include — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Directory include — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@Python API Documenation
-
@@ -412,11 +415,11 @@
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Directory include¶
+Directory include¶
↰ Parent directory (
cpp)Directory path:
cpp/include- Subdirectories¶
+Subdirectories¶
@@ -479,6 +482,7 @@Subdirectories + diff --git a/docs/_cpp_api/dir_cpp_include_torch_tensorrt.html b/docs/_cpp_api/dir_cpp_include_torch_tensorrt.html index c6d75ae40a..ce0bb54ca2 100644 --- a/docs/_cpp_api/dir_cpp_include_torch_tensorrt.html +++ b/docs/_cpp_api/dir_cpp_include_torch_tensorrt.html @@ -10,7 +10,7 @@ -
Directory torch_tensorrt — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Directory torch_tensorrt — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@Python API Documenation
-
@@ -412,11 +415,11 @@
- @@ -482,6 +485,7 @@
-
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Enum Level - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- +- ++ - - +- + - Enum Level¶
+Enum Level¶
Defined in File logging.h
- Enum Documentation¶
+Enum Documentation¶
- +- + @@ -475,23 +526,24 @@@@ -448,11 +499,11 @@
Enum DocumentationSphinx using a theme provided by Read the Docs.
Enum Documentation + + - - + @@ -502,7 +554,7 @@
Enum DocumentationTwitter
- YouTube
- LinkedIn
- +@@ -629,9 +681,9 @@- Terms @@ -590,9 +642,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -691,7 +743,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/enum_torch__tensorrt_8h_1a3fbe5d72e4fc624dbd038853079620eb.html b/docs/_cpp_api/enum_torch__tensorrt_8h_1a3fbe5d72e4fc624dbd038853079620eb.html
index 800fbd47db..9ee99a5eef 100644
--- a/docs/_cpp_api/enum_torch__tensorrt_8h_1a3fbe5d72e4fc624dbd038853079620eb.html
+++ b/docs/_cpp_api/enum_torch__tensorrt_8h_1a3fbe5d72e4fc624dbd038853079620eb.html
@@ -9,38 +9,48 @@
+
+
Enum EngineCapability — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Enum EngineCapability — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Enum EngineCapability - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- +- ++ - - +- + - Enum EngineCapability¶
+Enum EngineCapability¶
Defined in File torch_tensorrt.h
- Enum Documentation¶
+Enum Documentation¶
- +- + @@ -454,23 +505,24 @@@@ -427,11 +478,11 @@
Enum DocumentationSphinx using a theme provided by Read the Docs.
Enum Documentation + + - - + @@ -481,7 +533,7 @@
Enum DocumentationTwitter
- YouTube
- LinkedIn
- +@@ -608,9 +660,9 @@- Terms @@ -569,9 +621,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -670,7 +722,7 @@
- +Resources
Python API Documenation
-
@@ -412,7 +415,7 @@
- Program Listing for File logging.h @@ -432,25 +437,43 @@
stringtorch_tensorrt/macros.h(File macros.h)- +
- +
- +
- +
- +
- +
- +
- +
- Included By
- Namespaces +
- Enums +
- Functions
- @@ -492,6 +515,8 @@File logging.h¶
+File logging.h¶
↰ Parent directory (
cpp/include/torch_tensorrt)- Definition (
+cpp/include/torch_tensorrt/logging.h)¶Definition (
cpp/include/torch_tensorrt/logging.h)¶Definition (
cpp- Includes¶
+Includes¶
- Included By¶
+Included By¶
- +Namespaces¶
+Namespaces¶
+ +Enums¶
+-
+
+ Functions¶
+-
+
NamespacesIncludes
Namespaces + diff --git a/docs/_cpp_api/file_cpp_include_torch_tensorrt_macros.h.html b/docs/_cpp_api/file_cpp_include_torch_tensorrt_macros.h.html index fc44e9f539..5085008489 100644 --- a/docs/_cpp_api/file_cpp_include_torch_tensorrt_macros.h.html +++ b/docs/_cpp_api/file_cpp_include_torch_tensorrt_macros.h.html @@ -10,7 +10,7 @@ -
File macros.h — Torch-TensorRT v2.3.0.dev0+85971ff documentation +File macros.h — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion
+- Using Custom Kernels within TensorRT Engines with Torch-TensorRT
+- Wrapping Custom Kernels to use in TensorRT
+- Using Torch-TensorRT to Insert the Kernel
Python API Documenation
-
@@ -412,7 +415,7 @@
- Program Listing for File macros.h @@ -432,7 +435,7 @@
- @@ -520,6 +523,7 @@
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Program Listing for File ptq.h @@ -432,7 +436,7 @@
- Namespaces
- Classes +
- Functions
- File macros.h¶
+File macros.h¶
↰ Parent directory (
cpp/include/torch_tensorrt)Contents
@@ -424,7 +427,7 @@- Definition (
+cpp/include/torch_tensorrt/macros.h)¶Definition (
cpp/include/torch_tensorrt/macros.h)¶Definition (
cpp- Included By¶
+Included By¶
- Defines¶
+Defines¶
Defines + diff --git a/docs/_cpp_api/file_cpp_include_torch_tensorrt_ptq.h.html b/docs/_cpp_api/file_cpp_include_torch_tensorrt_ptq.h.html index 9b4b314e10..bdac8765a4 100644 --- a/docs/_cpp_api/file_cpp_include_torch_tensorrt_ptq.h.html +++ b/docs/_cpp_api/file_cpp_include_torch_tensorrt_ptq.h.html @@ -10,7 +10,7 @@ -
File ptq.h — Torch-TensorRT v2.3.0.dev0+85971ff documentation +File ptq.h — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@Python API Documenation
-
@@ -412,7 +415,7 @@
- @@ -502,6 +513,7 @@File ptq.h¶
+File ptq.h¶
↰ Parent directory (
cpp/include/torch_tensorrt)- Definition (
+cpp/include/torch_tensorrt/ptq.h)¶Definition (
cpp/include/torch_tensorrt/ptq.h)¶Definition (
cpp- Includes¶
+Includes¶
- +Classes¶
+Classes¶
+ Functions¶
+ +ClassesIncludes
Classes + diff --git a/docs/_cpp_api/file_cpp_include_torch_tensorrt_torch_tensorrt.h.html b/docs/_cpp_api/file_cpp_include_torch_tensorrt_torch_tensorrt.h.html index a4be8fad24..e8b37aeb69 100644 --- a/docs/_cpp_api/file_cpp_include_torch_tensorrt_torch_tensorrt.h.html +++ b/docs/_cpp_api/file_cpp_include_torch_tensorrt_torch_tensorrt.h.html @@ -10,7 +10,7 @@ -
File torch_tensorrt.h — Torch-TensorRT v2.3.0.dev0+85971ff documentation +File torch_tensorrt.h — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion
+- Using Custom Kernels within TensorRT Engines with Torch-TensorRT
+- Wrapping Custom Kernels to use in TensorRT
+- Using Torch-TensorRT to Insert the Kernel
Python API Documenation
-
@@ -412,7 +415,7 @@
- Program Listing for File torch_tensorrt.h @@ -432,7 +437,7 @@
- @@ -463,6 +468,24 @@
- Namespaces
- Classes +
- Enums +
- Functions
- @@ -504,6 +527,8 @@File torch_tensorrt.h¶
+File torch_tensorrt.h¶
↰ Parent directory (
cpp/include/torch_tensorrt)- Definition (
+cpp/include/torch_tensorrt/torch_tensorrt.h)¶Definition (
cpp/include/torch_tensorrt/torch_tensorrt.h)¶Definition (
cpp- Includes¶
+Includes¶
- +Classes¶
+Classes¶
ClassesClass TensorFormat
+ +Enums¶
+ ++ Functions¶
+ +ClassesIncludes
Classes + diff --git a/docs/_cpp_api/function_logging_8h_1a0593f776f469c20469e2f729fc7861a3.html b/docs/_cpp_api/function_logging_8h_1a0593f776f469c20469e2f729fc7861a3.html index c3e3443a38..e6d49b265a 100644 --- a/docs/_cpp_api/function_logging_8h_1a0593f776f469c20469e2f729fc7861a3.html +++ b/docs/_cpp_api/function_logging_8h_1a0593f776f469c20469e2f729fc7861a3.html @@ -9,38 +9,48 @@ + +
Function torch_tensorrt::logging::get_logging_prefix — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::logging::get_logging_prefix — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@- - Mobile +
+ + Edge + + +- @@ -196,9 +218,9 @@ + - - +
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::logging::get_logging_prefix - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -464,7 +516,7 @@@@ -437,23 +488,24 @@- + - Function torch_tensorrt::logging::get_logging_prefix¶
+Function torch_tensorrt::logging::get_logging_prefix¶
Defined in File logging.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -528,7 +580,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -591,9 +643,9 @@- Terms @@ -552,9 +604,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -653,7 +705,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_logging_8h_1a0c012cb374addd90eb1f42eaec570650.html b/docs/_cpp_api/function_logging_8h_1a0c012cb374addd90eb1f42eaec570650.html
index 9f31fa88fa..c65b19ecbd 100644
--- a/docs/_cpp_api/function_logging_8h_1a0c012cb374addd90eb1f42eaec570650.html
+++ b/docs/_cpp_api/function_logging_8h_1a0c012cb374addd90eb1f42eaec570650.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::logging::get_reportable_log_level — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::logging::get_reportable_log_level — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::logging::get_reportable_log_level - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -470,7 +522,7 @@@@ -443,23 +494,24 @@- + - Function torch_tensorrt::logging::get_reportable_log_level¶
+Function torch_tensorrt::logging::get_reportable_log_level¶
Defined in File logging.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -534,7 +586,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -597,9 +649,9 @@- Terms @@ -558,9 +610,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -659,7 +711,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_logging_8h_1a56e110feaaba2c3fd44bd201fd21a76a.html b/docs/_cpp_api/function_logging_8h_1a56e110feaaba2c3fd44bd201fd21a76a.html
index c9c7b6ad2e..37426e0f1b 100644
--- a/docs/_cpp_api/function_logging_8h_1a56e110feaaba2c3fd44bd201fd21a76a.html
+++ b/docs/_cpp_api/function_logging_8h_1a56e110feaaba2c3fd44bd201fd21a76a.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::logging::get_is_colored_output_on — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::logging::get_is_colored_output_on — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::logging::get_is_colored_output_on - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -470,7 +522,7 @@@@ -443,23 +494,24 @@- + - Function torch_tensorrt::logging::get_is_colored_output_on¶
+Function torch_tensorrt::logging::get_is_colored_output_on¶
Defined in File logging.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -534,7 +586,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -597,9 +649,9 @@- Terms @@ -558,9 +610,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -659,7 +711,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_logging_8h_1a7cb50492421ea9de4e3db895819df6f2.html b/docs/_cpp_api/function_logging_8h_1a7cb50492421ea9de4e3db895819df6f2.html
index aab16ad69d..4bcd330ba8 100644
--- a/docs/_cpp_api/function_logging_8h_1a7cb50492421ea9de4e3db895819df6f2.html
+++ b/docs/_cpp_api/function_logging_8h_1a7cb50492421ea9de4e3db895819df6f2.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::logging::set_reportable_log_level — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::logging::set_reportable_log_level — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::logging::set_reportable_log_level - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -470,7 +522,7 @@@@ -443,23 +494,24 @@- + - Function torch_tensorrt::logging::set_reportable_log_level¶
+Function torch_tensorrt::logging::set_reportable_log_level¶
Defined in File logging.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -534,7 +586,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -597,9 +649,9 @@- Terms @@ -558,9 +610,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -659,7 +711,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_logging_8h_1ac46ac0901cb97e3ae6e93b45f24e90b8.html b/docs/_cpp_api/function_logging_8h_1ac46ac0901cb97e3ae6e93b45f24e90b8.html
index 2222f7ade9..e894376e96 100644
--- a/docs/_cpp_api/function_logging_8h_1ac46ac0901cb97e3ae6e93b45f24e90b8.html
+++ b/docs/_cpp_api/function_logging_8h_1ac46ac0901cb97e3ae6e93b45f24e90b8.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::logging::log — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::logging::log — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::logging::log - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -473,7 +525,7 @@@@ -446,23 +497,24 @@- + - Function torch_tensorrt::logging::log¶
+Function torch_tensorrt::logging::log¶
Defined in File logging.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -537,7 +589,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -600,9 +652,9 @@- Terms @@ -561,9 +613,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -662,7 +714,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_logging_8h_1ad2efd47b6c3689e58ccc595680579ae5.html b/docs/_cpp_api/function_logging_8h_1ad2efd47b6c3689e58ccc595680579ae5.html
index 336753ea8a..a0d5ad1cce 100644
--- a/docs/_cpp_api/function_logging_8h_1ad2efd47b6c3689e58ccc595680579ae5.html
+++ b/docs/_cpp_api/function_logging_8h_1ad2efd47b6c3689e58ccc595680579ae5.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::logging::set_is_colored_output_on — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::logging::set_is_colored_output_on — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::logging::set_is_colored_output_on - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -470,7 +522,7 @@@@ -443,23 +494,24 @@- + - Function torch_tensorrt::logging::set_is_colored_output_on¶
+Function torch_tensorrt::logging::set_is_colored_output_on¶
Defined in File logging.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -534,7 +586,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -597,9 +649,9 @@- Terms @@ -558,9 +610,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -659,7 +711,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_logging_8h_1af8f3443813315af7901903d25dd495cc.html b/docs/_cpp_api/function_logging_8h_1af8f3443813315af7901903d25dd495cc.html
index a0d143d349..1a168f5281 100644
--- a/docs/_cpp_api/function_logging_8h_1af8f3443813315af7901903d25dd495cc.html
+++ b/docs/_cpp_api/function_logging_8h_1af8f3443813315af7901903d25dd495cc.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::logging::set_logging_prefix — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::logging::set_logging_prefix — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::logging::set_logging_prefix - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -464,7 +516,7 @@@@ -437,23 +488,24 @@- + - Function torch_tensorrt::logging::set_logging_prefix¶
+Function torch_tensorrt::logging::set_logging_prefix¶
Defined in File logging.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -528,7 +580,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -591,9 +643,9 @@- Terms @@ -552,9 +604,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -653,7 +705,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_ptq_8h_1a226e3c83379d1012cde8578c1c86b16c.html b/docs/_cpp_api/function_ptq_8h_1a226e3c83379d1012cde8578c1c86b16c.html
index 0a8c7a869b..6916caccea 100644
--- a/docs/_cpp_api/function_ptq_8h_1a226e3c83379d1012cde8578c1c86b16c.html
+++ b/docs/_cpp_api/function_ptq_8h_1a226e3c83379d1012cde8578c1c86b16c.html
@@ -9,38 +9,48 @@
+
+
Template Function torch_tensorrt::ptq::make_int8_cache_calibrator — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Template Function torch_tensorrt::ptq::make_int8_cache_calibrator — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Template Function torch_tensorrt::ptq::make_int8_cache_calibrator - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -479,7 +531,7 @@@@ -452,23 +503,24 @@- + - Template Function torch_tensorrt::ptq::make_int8_cache_calibrator¶
+Template Function torch_tensorrt::ptq::make_int8_cache_calibrator¶
Defined in File ptq.h
- Function Documentation¶
+Function Documentation¶
-
template<typename Algorithm = nvinfer1::IInt8EntropyCalibrator2>
inline Int8CacheCalibrator<Algorithm> torch_tensorrt::ptq::make_int8_cache_calibrator(const std::string &cache_file_path)¶
@@ -399,25 +450,25 @@
Function Documentation - +
Function Documentation
Function Documentation + @@ -543,7 +595,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -606,9 +658,9 @@- Terms @@ -567,9 +619,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -668,7 +720,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_ptq_8h_1a6186e305f47c1d94b6130ef6c7f7e178.html b/docs/_cpp_api/function_ptq_8h_1a6186e305f47c1d94b6130ef6c7f7e178.html
index 9430bcd14e..3f0b1bdc62 100644
--- a/docs/_cpp_api/function_ptq_8h_1a6186e305f47c1d94b6130ef6c7f7e178.html
+++ b/docs/_cpp_api/function_ptq_8h_1a6186e305f47c1d94b6130ef6c7f7e178.html
@@ -9,38 +9,48 @@
+
+
Template Function torch_tensorrt::ptq::make_int8_calibrator — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Template Function torch_tensorrt::ptq::make_int8_calibrator — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Template Function torch_tensorrt::ptq::make_int8_calibrator - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -485,7 +537,7 @@@@ -458,23 +509,24 @@- + - Template Function torch_tensorrt::ptq::make_int8_calibrator¶
+Template Function torch_tensorrt::ptq::make_int8_calibrator¶
Defined in File ptq.h
- Function Documentation¶
+Function Documentation¶
-
template<typename Algorithm = nvinfer1::IInt8EntropyCalibrator2, typename DataLoader>
inline Int8Calibrator<Algorithm, DataLoader> torch_tensorrt::ptq::make_int8_calibrator(DataLoader dataloader, const std::string &cache_file_path, bool use_cache)¶
@@ -405,25 +456,25 @@
Function Documentation - +
Function Documentation
Function Documentation + @@ -549,7 +601,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -612,9 +664,9 @@- Terms @@ -573,9 +625,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -674,7 +726,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_torch__tensorrt_8h_1a5b405fd3bf3c8fc2e2a54cbbab979797.html b/docs/_cpp_api/function_torch__tensorrt_8h_1a5b405fd3bf3c8fc2e2a54cbbab979797.html
index d0704c7891..9507e513ad 100644
--- a/docs/_cpp_api/function_torch__tensorrt_8h_1a5b405fd3bf3c8fc2e2a54cbbab979797.html
+++ b/docs/_cpp_api/function_torch__tensorrt_8h_1a5b405fd3bf3c8fc2e2a54cbbab979797.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::torchscript::check_method_operator_support — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::torchscript::check_method_operator_support — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::torchscript::check_method_operator_support - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -479,7 +531,7 @@@@ -452,23 +503,24 @@- + - Function torch_tensorrt::torchscript::check_method_operator_support¶
+Function torch_tensorrt::torchscript::check_method_operator_support¶
Defined in File torch_tensorrt.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -543,7 +595,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -606,9 +658,9 @@- Terms @@ -567,9 +619,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -668,7 +720,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_torch__tensorrt_8h_1a6e19490a08fb1553c9dd347a5ae79db9.html b/docs/_cpp_api/function_torch__tensorrt_8h_1a6e19490a08fb1553c9dd347a5ae79db9.html
index b2597f6b12..6a44b7ba5b 100644
--- a/docs/_cpp_api/function_torch__tensorrt_8h_1a6e19490a08fb1553c9dd347a5ae79db9.html
+++ b/docs/_cpp_api/function_torch__tensorrt_8h_1a6e19490a08fb1553c9dd347a5ae79db9.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::torchscript::compile — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::torchscript::compile — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::torchscript::compile - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -479,7 +531,7 @@@@ -452,23 +503,24 @@- + - Function torch_tensorrt::torchscript::compile¶
+Function torch_tensorrt::torchscript::compile¶
Defined in File torch_tensorrt.h
- Function Documentation¶
+Function Documentation¶
-
TORCHTRT_API torch::jit::Module torch_tensorrt::torchscript::compile(const torch::jit::Module &module, CompileSpec info)¶
Function Documentation - +
Function Documentation
Function Documentation + @@ -543,7 +595,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -606,9 +658,9 @@- Terms @@ -567,9 +619,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -668,7 +720,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_torch__tensorrt_8h_1a81f9783517335dda877d8cfcf38987c9.html b/docs/_cpp_api/function_torch__tensorrt_8h_1a81f9783517335dda877d8cfcf38987c9.html
index be11491716..c9dfa66db7 100644
--- a/docs/_cpp_api/function_torch__tensorrt_8h_1a81f9783517335dda877d8cfcf38987c9.html
+++ b/docs/_cpp_api/function_torch__tensorrt_8h_1a81f9783517335dda877d8cfcf38987c9.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::torchscript::embed_engine_in_new_module — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::torchscript::embed_engine_in_new_module — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::torchscript::embed_engine_in_new_module - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -485,7 +537,7 @@@@ -458,23 +509,24 @@- + - Function torch_tensorrt::torchscript::embed_engine_in_new_module¶
+Function torch_tensorrt::torchscript::embed_engine_in_new_module¶
Defined in File torch_tensorrt.h
- Function Documentation¶
+Function Documentation¶
-
TORCHTRT_API torch::jit::Module torch_tensorrt::torchscript::embed_engine_in_new_module(const std::string &engine, Device device, const std::vector<std::string> &input_binding_names = std::vector<std::string>(), const std::vector<std::string> &output_binding_names = std::vector<std::string>())¶
Function Documentation - +
Function Documentation
Function Documentation + @@ -549,7 +601,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -612,9 +664,9 @@- Terms @@ -573,9 +625,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -674,7 +726,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_torch__tensorrt_8h_1ac4ab8313ae72c2c899ea31548b528528.html b/docs/_cpp_api/function_torch__tensorrt_8h_1ac4ab8313ae72c2c899ea31548b528528.html
index e8822211b2..afd66aa9e5 100644
--- a/docs/_cpp_api/function_torch__tensorrt_8h_1ac4ab8313ae72c2c899ea31548b528528.html
+++ b/docs/_cpp_api/function_torch__tensorrt_8h_1ac4ab8313ae72c2c899ea31548b528528.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::get_build_info — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::get_build_info — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::get_build_info - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -470,7 +522,7 @@@@ -443,23 +494,24 @@- + - Function torch_tensorrt::get_build_info¶
+Function torch_tensorrt::get_build_info¶
Defined in File torch_tensorrt.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -534,7 +586,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -597,9 +649,9 @@- Terms @@ -558,9 +610,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -659,7 +711,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_torch__tensorrt_8h_1ad1acd06eaeaffbbcf6e7ebf426891384.html b/docs/_cpp_api/function_torch__tensorrt_8h_1ad1acd06eaeaffbbcf6e7ebf426891384.html
index 0c0525e939..52e8733bfa 100644
--- a/docs/_cpp_api/function_torch__tensorrt_8h_1ad1acd06eaeaffbbcf6e7ebf426891384.html
+++ b/docs/_cpp_api/function_torch__tensorrt_8h_1ad1acd06eaeaffbbcf6e7ebf426891384.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::set_device — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::set_device — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::set_device - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -470,7 +522,7 @@@@ -443,23 +494,24 @@- + - Function torch_tensorrt::set_device¶
+Function torch_tensorrt::set_device¶
Defined in File torch_tensorrt.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -534,7 +586,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -597,9 +649,9 @@- Terms @@ -558,9 +610,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -659,7 +711,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_torch__tensorrt_8h_1ad6a4ee8ca6c8f6e5519eb1128ec7f4a1.html b/docs/_cpp_api/function_torch__tensorrt_8h_1ad6a4ee8ca6c8f6e5519eb1128ec7f4a1.html
index 907262e284..020504cffa 100644
--- a/docs/_cpp_api/function_torch__tensorrt_8h_1ad6a4ee8ca6c8f6e5519eb1128ec7f4a1.html
+++ b/docs/_cpp_api/function_torch__tensorrt_8h_1ad6a4ee8ca6c8f6e5519eb1128ec7f4a1.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::dump_build_info — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::dump_build_info — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::dump_build_info - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -465,7 +517,7 @@@@ -438,23 +489,24 @@- + - Function torch_tensorrt::dump_build_info¶
+Function torch_tensorrt::dump_build_info¶
Defined in File torch_tensorrt.h
- Function Documentation¶
+Function Documentation¶
Function Documentation
Function Documentation + @@ -529,7 +581,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -592,9 +644,9 @@- Terms @@ -553,9 +605,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -654,7 +706,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/function_torch__tensorrt_8h_1ae8d56472106eeef37fbe51ff7f40c9b2.html b/docs/_cpp_api/function_torch__tensorrt_8h_1ae8d56472106eeef37fbe51ff7f40c9b2.html
index 227a4a3403..5a40d5be1d 100644
--- a/docs/_cpp_api/function_torch__tensorrt_8h_1ae8d56472106eeef37fbe51ff7f40c9b2.html
+++ b/docs/_cpp_api/function_torch__tensorrt_8h_1ae8d56472106eeef37fbe51ff7f40c9b2.html
@@ -9,38 +9,48 @@
+
+
Function torch_tensorrt::torchscript::convert_method_to_trt_engine — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Function torch_tensorrt::torchscript::convert_method_to_trt_engine — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,20 +407,24 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Function torch_tensorrt::torchscript::convert_method_to_trt_engine - - + +
-
-
-
+
+
-
-
+
+
-
+
- - + + + + +- ++ + - - - + + + - - + @@ -479,7 +531,7 @@@@ -452,23 +503,24 @@- + - Function torch_tensorrt::torchscript::convert_method_to_trt_engine¶
+Function torch_tensorrt::torchscript::convert_method_to_trt_engine¶
Defined in File torch_tensorrt.h
- Function Documentation¶
+Function Documentation¶
-
TORCHTRT_API std::string torch_tensorrt::torchscript::convert_method_to_trt_engine(const torch::jit::Module &module, std::string method_name, CompileSpec info)¶
Function Documentation - +
Function Documentation
Function Documentation + @@ -543,7 +595,7 @@
Resources
- Twitter
- YouTube
- LinkedIn
- +@@ -606,9 +658,9 @@- Terms @@ -567,9 +619,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -668,7 +720,7 @@
- +Resources
-
diff --git a/docs/_cpp_api/namespace_torch.html b/docs/_cpp_api/namespace_torch.html
index 3bb3a82699..2b634c6e48 100644
--- a/docs/_cpp_api/namespace_torch.html
+++ b/docs/_cpp_api/namespace_torch.html
@@ -9,38 +9,48 @@
+
+
Namespace torch — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Namespace torch — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@ -
- Mobile
+ + + Edge + + +
-
@@ -196,9 +218,9 @@
+
-
-
+
Table of Contents @@ -209,21 +231,21 @@@@ -304,7 +351,7 @@- +- + @@ -323,32 +370,32 @@@@ -360,38 +407,42 @@
-
-
+
- - + Docs - + > - +
- Torch-TensorRT C++ API > - +
- Namespace torch - - + +
-
-
-
+
+
-
-
+
+
-
+
@@ -577,9 +629,9 @@- Terms @@ -538,9 +590,9 @@
Resources
© Copyright The Linux Foundation. The PyTorch Foundation is a project of The Linux Foundation. For web site terms of use, trademark policy and other policies applicable to The PyTorch Foundation please see - www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source + www.linuxfoundation.org/policies/. The PyTorch Foundation supports the PyTorch open source project, which has been established as PyTorch Project a Series of LF Projects, LLC. For policies applicable to the PyTorch Project a Series of LF Projects, LLC, - please see www.lfprojects.org/policies/.
+ please see www.lfprojects.org/policies/.Resources
- Ecosystem
- +- - Mobile + Mobile
- @@ -639,7 +691,7 @@
- +Resources
Python API Documenation
-
@@ -416,7 +419,7 @@
- -
- -
- -
- -
- -
- -
- +
- +
- +
- +
- +
- +
- +
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- @@ -510,6 +513,7 @@Namespace torch_tensorrt::logging¶
+Namespace torch_tensorrt::logging¶
Contents
-
@@ -425,21 +428,21 @@
- Enums¶
+Enums¶
- Functions¶
+Functions¶
-
-
Functions + diff --git a/docs/_cpp_api/namespace_torch_tensorrt__ptq.html b/docs/_cpp_api/namespace_torch_tensorrt__ptq.html index cff576fdae..557c3fb4a3 100644 --- a/docs/_cpp_api/namespace_torch_tensorrt__ptq.html +++ b/docs/_cpp_api/namespace_torch_tensorrt__ptq.html @@ -10,7 +10,7 @@ -
Namespace torch_tensorrt::ptq — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Namespace torch_tensorrt::ptq — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@Python API Documenation
-
@@ -416,7 +419,7 @@
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- @@ -506,6 +509,7 @@Namespace torch_tensorrt::ptq¶
+Namespace torch_tensorrt::ptq¶
Contents
-
@@ -425,17 +428,17 @@
- Classes¶
+Classes¶
- Functions¶
+Functions¶
Functions + diff --git a/docs/_cpp_api/namespace_torch_tensorrt__torchscript.html b/docs/_cpp_api/namespace_torch_tensorrt__torchscript.html index bdf840f257..e3d5a1a307 100644 --- a/docs/_cpp_api/namespace_torch_tensorrt__torchscript.html +++ b/docs/_cpp_api/namespace_torch_tensorrt__torchscript.html @@ -10,7 +10,7 @@ -
Namespace torch_tensorrt::torchscript — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Namespace torch_tensorrt::torchscript — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@Python API Documenation
-
@@ -416,7 +419,7 @@
Function torch_tensorrt::torchscript::check_method_operator_support
-- -
Function torch_tensorrt::torchscript::convert_method_to_trt_engine
-Function torch_tensorrt::torchscript::embed_engine_in_new_module
+Function torch_tensorrt::torchscript::check_method_operator_support
+- +
Function torch_tensorrt::torchscript::convert_method_to_trt_engine
+Function torch_tensorrt::torchscript::embed_engine_in_new_module
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- @@ -507,6 +510,7 @@Namespace torch_tensorrt::torchscript¶
+Namespace torch_tensorrt::torchscript¶
Contents
-
@@ -425,18 +428,18 @@
- Classes¶
+Classes¶
- Functions¶
+Functions¶
-
-
Functions + diff --git a/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_logging.h.html b/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_logging.h.html index 129e0d1836..79155a2d11 100644 --- a/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_logging.h.html +++ b/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_logging.h.html @@ -10,7 +10,7 @@ -
Program Listing for File logging.h — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Program Listing for File logging.h — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@Python API Documenation
-
@@ -412,47 +415,47 @@
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
Defined in File torch_tensorrt.h
-
-struct torch_tensorrt::Device¶
Setting data structure for Target device.
Public Functions
@@ -527,7 +530,7 @@Struct Documentation
- -inline operator Value() const
+inline operator Value() const
Get the internal value from the Device object.
- Returns @@ -543,32 +546,34 @@
-
-inline constexpr bool operator==(DeviceType other) const
Comparison operator for DeviceType.
- Parameters
other –
- Returns -
true
-false
- +true
+- Returns +
false
Struct Documentation
-
-inline constexpr bool operator!=(DeviceType other) const
Comparison operator for DeviceType.
- Parameters
other –
- Returns -
true
-false
- +true
+- Returns +
false
Struct Documentation + diff --git a/docs/_cpp_api/structtorch__tensorrt_1_1GraphInputs.html b/docs/_cpp_api/structtorch__tensorrt_1_1GraphInputs.html index d431562c52..d14225bb27 100644 --- a/docs/_cpp_api/structtorch__tensorrt_1_1GraphInputs.html +++ b/docs/_cpp_api/structtorch__tensorrt_1_1GraphInputs.html @@ -10,7 +10,7 @@ -
Struct GraphInputs — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Struct GraphInputs — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion
+- Using Custom Kernels within TensorRT Engines with Torch-TensorRT
+- Wrapping Custom Kernels to use in TensorRT
+- Using Torch-TensorRT to Insert the Kernel
Python API Documenation
-
@@ -416,15 +419,15 @@
Defined in File torch_tensorrt.h
-
-struct torch_tensorrt::GraphInputs¶
A struct to hold complex inputs.
This struct can either hold a complex inputs of shape or a flattened one,
@@ -510,6 +513,7 @@Struct Documentation + diff --git a/docs/_cpp_api/structtorch__tensorrt_1_1Input.html b/docs/_cpp_api/structtorch__tensorrt_1_1Input.html index 97933897a5..8ee4ee655d 100644 --- a/docs/_cpp_api/structtorch__tensorrt_1_1Input.html +++ b/docs/_cpp_api/structtorch__tensorrt_1_1Input.html @@ -10,7 +10,7 @@ -
Struct Input — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Struct Input — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion
+- Using Custom Kernels within TensorRT Engines with Torch-TensorRT
+- Wrapping Custom Kernels to use in TensorRT
+- Using Torch-TensorRT to Insert the Kernel
Python API Documenation
-
@@ -416,24 +419,24 @@
Defined in File torch_tensorrt.h
public CustomClassHolder
+public torch::CustomClassHolder-
-struct torch_tensorrt::Input : public CustomClassHolder¶
A struct to hold an input range (used by TensorRT Optimization profile)
This struct can either hold a single vector representing an input shape, signifying a static input shape or a set of three input shapes representing the min, optiminal and max input shapes allowed for the engine.
@@ -445,7 +448,7 @@Struct Documentation
- -TORCHTRT_API Input(std::vector<int64_t> shape, TensorFormat format = TensorFormat::kContiguous)¶
+TORCHTRT_API Input(std::vector<int64_t> shape, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for static input size from vector, optional arguments allow the user to configure expected input shape tensor format. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters @@ -459,7 +462,7 @@
-
-TORCHTRT_API Input(std::vector<int64_t> shape, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters @@ -474,7 +477,7 @@
-
-TORCHTRT_API Input(std::vector<int64_t> shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for static input size from vector, optional arguments allow the user to configure expected input shape tensor format.
- Parameters @@ -489,7 +492,7 @@
-
-TORCHTRT_API Input(std::vector<int64_t> shape, DataType dtype, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for static input size from vector, optional arguments allow the user to configure expected input shape tensor format.
- Parameters @@ -505,7 +508,7 @@
-
-TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters @@ -519,7 +522,7 @@
-
-TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters @@ -534,7 +537,7 @@
-
-TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format.
- Parameters @@ -549,7 +552,7 @@
-
-TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, DataType dtype, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format.
- Parameters @@ -565,7 +568,7 @@
-
-TORCHTRT_API Input(std::vector<int64_t> min_shape, std::vector<int64_t> opt_shape, std::vector<int64_t> max_shape, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters @@ -581,7 +584,7 @@
-
-TORCHTRT_API Input(std::vector<int64_t> min_shape, std::vector<int64_t> opt_shape, std::vector<int64_t> max_shape, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters @@ -598,7 +601,7 @@
-
-TORCHTRT_API Input(std::vector<int64_t> min_shape, std::vector<int64_t> opt_shape, std::vector<int64_t> max_shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for a dynamic input size from vectors for minimum shape, optimal shape, and max shape supported sizes optional arguments allow the user to configure expected input shape tensor format.
- Parameters @@ -615,7 +618,7 @@
-
-TORCHTRT_API Input(std::vector<int64_t> min_shape, std::vector<int64_t> opt_shape, std::vector<int64_t> max_shape, DataType dtype, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object for a dynamic input size from vectors for minimum shape, optimal shape, and max shape supported sizes optional arguments allow the user to configure expected input shape tensor format.
- Parameters @@ -633,7 +636,7 @@
-
-TORCHTRT_API Input(c10::ArrayRef<int64_t> min_shape, c10::ArrayRef<int64_t> opt_shape, c10::ArrayRef<int64_t> max_shape, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters @@ -649,7 +652,7 @@
-
-TORCHTRT_API Input(c10::ArrayRef<int64_t> min_shape, c10::ArrayRef<int64_t> opt_shape, c10::ArrayRef<int64_t> max_shape, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)
- Parameters @@ -666,7 +669,7 @@
-
-TORCHTRT_API Input(c10::ArrayRef<int64_t> min_shape, c10::ArrayRef<int64_t> opt_shape, c10::ArrayRef<int64_t> max_shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes.
- Parameters @@ -683,7 +686,7 @@
-
-TORCHTRT_API Input(c10::ArrayRef<int64_t> min_shape, c10::ArrayRef<int64_t> opt_shape, c10::ArrayRef<int64_t> max_shape, DataType dtype, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)¶
Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes.
- Parameters @@ -831,6 +834,7 @@
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
Defined in File torch_tensorrt.h
-
-struct torch_tensorrt::torchscript::CompileSpec¶
Settings data structure for Torch-TensorRT TorchScript compilation
Public Functions
@@ -486,7 +489,7 @@Struct Documentation
- -std::set<DataType> enabled_precisions = {DataType::kFloat}¶
+std::set<DataType> enabled_precisions = {DataType::kFloat}¶
The set of precisions TensorRT is allowed to use for kernels during compilation.
Struct Documentation
- -EngineCapability capability = EngineCapability::kSTANDARD¶
+EngineCapability capability = EngineCapability::kSTANDARD¶
Sets the restrictions for the engine (CUDA Safety)
Struct Documentation + diff --git a/docs/_cpp_api/torch_tensort_cpp.html b/docs/_cpp_api/torch_tensort_cpp.html index d83f92797f..569714e649 100644 --- a/docs/_cpp_api/torch_tensort_cpp.html +++ b/docs/_cpp_api/torch_tensort_cpp.html @@ -10,7 +10,7 @@ -
Torch-TensorRT C++ API — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Torch-TensorRT C++ API — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -39,7 +39,7 @@ - + + diff --git a/docs/_cpp_api/unabridged_orphan.html b/docs/_cpp_api/unabridged_orphan.html index 02ae2a1869..03c6f74298 100644 --- a/docs/_cpp_api/unabridged_orphan.html +++ b/docs/_cpp_api/unabridged_orphan.html @@ -10,7 +10,7 @@ -Full API — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Full API — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion
+- Using Custom Kernels within TensorRT Engines with Torch-TensorRT
+- Wrapping Custom Kernels to use in TensorRT
+- Using Torch-TensorRT to Insert the Kernel
Python API Documenation
-
@@ -412,9 +415,9 @@
- Directory cpp
-
@@ -441,7 +444,7 @@
- Includes
- Namespaces
- Classes +
- Functions @@ -494,6 +500,8 @@
- Includes
- Namespaces
- Classes +
- Enums +
- Functions @@ -562,6 +570,7 @@
- Compiling a Transformer using torch.compile and TensorRT
- Torch Compile Advanced Usage
- Torch Compile Stable Diffusion +
- Using Custom Kernels within TensorRT Engines with Torch-TensorRT +
- Wrapping Custom Kernels to use in TensorRT +
- Using Torch-TensorRT to Insert the Kernel
Directories -
Files¶
+Files¶
@@ -480,6 +485,7 @@Files
Files
Files + diff --git a/docs/_downloads/0daf1d0af656cac7b808856b71e6616f/torch_compile_resnet_example.ipynb b/docs/_downloads/0daf1d0af656cac7b808856b71e6616f/torch_compile_resnet_example.ipynb index 1c4b8134a9..4ef7182d48 100644 --- a/docs/_downloads/0daf1d0af656cac7b808856b71e6616f/torch_compile_resnet_example.ipynb +++ b/docs/_downloads/0daf1d0af656cac7b808856b71e6616f/torch_compile_resnet_example.ipynb @@ -150,7 +150,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.0" + "version": "3.11.7" } }, "nbformat": 4, diff --git a/docs/_downloads/0e30a6276601af7e5fc4d5166e2e3d37/torch_compile_advanced_usage.py b/docs/_downloads/0e30a6276601af7e5fc4d5166e2e3d37/torch_compile_advanced_usage.py index 96146a43d8..8ebedab111 100644 --- a/docs/_downloads/0e30a6276601af7e5fc4d5166e2e3d37/torch_compile_advanced_usage.py +++ b/docs/_downloads/0e30a6276601af7e5fc4d5166e2e3d37/torch_compile_advanced_usage.py @@ -43,7 +43,7 @@ def forward(self, x: torch.Tensor, y: torch.Tensor): # For the default settings, we can simply call torch.compile # with the backend "torch_tensorrt", and run the model on an # input to cause compilation, as so: -optimized_model = torch.compile(model, backend="torch_tensorrt") +optimized_model = torch.compile(model, backend="torch_tensorrt", dynamic=False) optimized_model(*sample_inputs) # %% @@ -81,7 +81,10 @@ def forward(self, x: torch.Tensor, y: torch.Tensor): # Run the model on an input to cause compilation, as so: optimized_model_custom = torch.compile( - model_half, backend="torch_tensorrt", options=backend_kwargs + model_half, + backend="torch_tensorrt", + options=backend_kwargs, + dynamic=False, ) optimized_model_custom(*sample_inputs_half) diff --git a/docs/_downloads/46b3e6febaab06324aa2715896895544/torch_compile_stable_diffusion.py b/docs/_downloads/46b3e6febaab06324aa2715896895544/torch_compile_stable_diffusion.py index 0511e5a363..a0b725572b 100644 --- a/docs/_downloads/46b3e6febaab06324aa2715896895544/torch_compile_stable_diffusion.py +++ b/docs/_downloads/46b3e6febaab06324aa2715896895544/torch_compile_stable_diffusion.py @@ -18,9 +18,8 @@ # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ import torch -from diffusers import DiffusionPipeline - import torch_tensorrt +from diffusers import DiffusionPipeline model_id = "CompVis/stable-diffusion-v1-4" device = "cuda:0" @@ -39,7 +38,7 @@ backend=backend, options={ "truncate_long_and_double": True, - "precision": torch.float16, + "enabled_precisions": {torch.float32, torch.float16}, }, dynamic=False, ) diff --git a/docs/_downloads/6a6052d9668b2cb8332d349d328e21c1/_rendered_examples_jupyter.zip b/docs/_downloads/6a6052d9668b2cb8332d349d328e21c1/_rendered_examples_jupyter.zip index 0bbc029cee..507661ba9d 100644 Binary files a/docs/_downloads/6a6052d9668b2cb8332d349d328e21c1/_rendered_examples_jupyter.zip and b/docs/_downloads/6a6052d9668b2cb8332d349d328e21c1/_rendered_examples_jupyter.zip differ diff --git a/docs/_downloads/798cda8f83bd9f5e2cc93f329a04332c/_rendered_examples_python.zip b/docs/_downloads/798cda8f83bd9f5e2cc93f329a04332c/_rendered_examples_python.zip index 6dbb2d437c..6301a433d4 100644 Binary files a/docs/_downloads/798cda8f83bd9f5e2cc93f329a04332c/_rendered_examples_python.zip and b/docs/_downloads/798cda8f83bd9f5e2cc93f329a04332c/_rendered_examples_python.zip differ diff --git a/docs/_downloads/b35883282793ac3413933fdb22d00d81/torch_compile_advanced_usage.ipynb b/docs/_downloads/b35883282793ac3413933fdb22d00d81/torch_compile_advanced_usage.ipynb index 57e26cd5ed..343b129a3b 100644 --- a/docs/_downloads/b35883282793ac3413933fdb22d00d81/torch_compile_advanced_usage.ipynb +++ b/docs/_downloads/b35883282793ac3413933fdb22d00d81/torch_compile_advanced_usage.ipynb @@ -62,7 +62,7 @@ }, "outputs": [], "source": [ - "# Next, we compile the model using torch.compile\n# For the default settings, we can simply call torch.compile\n# with the backend \"torch_tensorrt\", and run the model on an\n# input to cause compilation, as so:\noptimized_model = torch.compile(model, backend=\"torch_tensorrt\")\noptimized_model(*sample_inputs)" + "# Next, we compile the model using torch.compile\n# For the default settings, we can simply call torch.compile\n# with the backend \"torch_tensorrt\", and run the model on an\n# input to cause compilation, as so:\noptimized_model = torch.compile(model, backend=\"torch_tensorrt\", dynamic=False)\noptimized_model(*sample_inputs)" ] }, { @@ -91,7 +91,7 @@ }, "outputs": [], "source": [ - "# If we want to customize certain options in the backend,\n# but still use the torch.compile call directly, we can provide\n# custom options to the backend via the \"options\" keyword\n# which takes in a dictionary mapping options to values.\n#\n# For accepted backend options, see the CompilationSettings dataclass:\n# py/torch_tensorrt/dynamo/_settings.py\nbackend_kwargs = {\n \"enabled_precisions\": {torch.half},\n \"debug\": True,\n \"min_block_size\": 2,\n \"torch_executed_ops\": {\"torch.ops.aten.sub.Tensor\"},\n \"optimization_level\": 4,\n \"use_python_runtime\": False,\n}\n\n# Run the model on an input to cause compilation, as so:\noptimized_model_custom = torch.compile(\n model_half, backend=\"torch_tensorrt\", options=backend_kwargs\n)\noptimized_model_custom(*sample_inputs_half)" + "# If we want to customize certain options in the backend,\n# but still use the torch.compile call directly, we can provide\n# custom options to the backend via the \"options\" keyword\n# which takes in a dictionary mapping options to values.\n#\n# For accepted backend options, see the CompilationSettings dataclass:\n# py/torch_tensorrt/dynamo/_settings.py\nbackend_kwargs = {\n \"enabled_precisions\": {torch.half},\n \"debug\": True,\n \"min_block_size\": 2,\n \"torch_executed_ops\": {\"torch.ops.aten.sub.Tensor\"},\n \"optimization_level\": 4,\n \"use_python_runtime\": False,\n}\n\n# Run the model on an input to cause compilation, as so:\noptimized_model_custom = torch.compile(\n model_half,\n backend=\"torch_tensorrt\",\n options=backend_kwargs,\n dynamic=False,\n)\noptimized_model_custom(*sample_inputs_half)" ] }, { @@ -136,7 +136,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.0" + "version": "3.11.7" } }, "nbformat": 4, diff --git a/docs/_downloads/b776287bc876f7ce24942b82a66beb05/torch_compile_stable_diffusion.ipynb b/docs/_downloads/b776287bc876f7ce24942b82a66beb05/torch_compile_stable_diffusion.ipynb index d48320c8b1..24cbbefea1 100644 --- a/docs/_downloads/b776287bc876f7ce24942b82a66beb05/torch_compile_stable_diffusion.ipynb +++ b/docs/_downloads/b776287bc876f7ce24942b82a66beb05/torch_compile_stable_diffusion.ipynb @@ -22,7 +22,7 @@ }, "outputs": [], "source": [ - "import torch\nfrom diffusers import DiffusionPipeline\n\nimport torch_tensorrt\n\nmodel_id = \"CompVis/stable-diffusion-v1-4\"\ndevice = \"cuda:0\"\n\n# Instantiate Stable Diffusion Pipeline with FP16 weights\npipe = DiffusionPipeline.from_pretrained(\n model_id, revision=\"fp16\", torch_dtype=torch.float16\n)\npipe = pipe.to(device)\n\nbackend = \"torch_tensorrt\"\n\n# Optimize the UNet portion with Torch-TensorRT\npipe.unet = torch.compile(\n pipe.unet,\n backend=backend,\n options={\n \"truncate_long_and_double\": True,\n \"precision\": torch.float16,\n },\n dynamic=False,\n)" + "import torch\nimport torch_tensorrt\nfrom diffusers import DiffusionPipeline\n\nmodel_id = \"CompVis/stable-diffusion-v1-4\"\ndevice = \"cuda:0\"\n\n# Instantiate Stable Diffusion Pipeline with FP16 weights\npipe = DiffusionPipeline.from_pretrained(\n model_id, revision=\"fp16\", torch_dtype=torch.float16\n)\npipe = pipe.to(device)\n\nbackend = \"torch_tensorrt\"\n\n# Optimize the UNet portion with Torch-TensorRT\npipe.unet = torch.compile(\n pipe.unet,\n backend=backend,\n options={\n \"truncate_long_and_double\": True,\n \"enabled_precisions\": {torch.float32, torch.float16},\n },\n dynamic=False,\n)" ] }, { @@ -60,7 +60,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.0" + "version": "3.11.7" } }, "nbformat": 4, diff --git a/docs/_downloads/ce102e287ddb5744f0a1364e8c0c7f68/torch_compile_transformers_example.ipynb b/docs/_downloads/ce102e287ddb5744f0a1364e8c0c7f68/torch_compile_transformers_example.ipynb index 98c872cfb9..9d2c6dcebd 100644 --- a/docs/_downloads/ce102e287ddb5744f0a1364e8c0c7f68/torch_compile_transformers_example.ipynb +++ b/docs/_downloads/ce102e287ddb5744f0a1364e8c0c7f68/torch_compile_transformers_example.ipynb @@ -69,7 +69,7 @@ }, "outputs": [], "source": [ - "# Define backend compilation keyword arguments\ncompilation_kwargs = {\n \"enabled_precisions\": enabled_precisions,\n \"debug\": debug,\n \"workspace_size\": workspace_size,\n \"min_block_size\": min_block_size,\n \"torch_executed_ops\": torch_executed_ops,\n}\n\n# Build and compile the model with torch.compile, using Torch-TensorRT backend\noptimized_model = torch.compile(\n model,\n backend=\"torch_tensorrt\",\n options=compilation_kwargs,\n)\noptimized_model(*inputs)" + "# Define backend compilation keyword arguments\ncompilation_kwargs = {\n \"enabled_precisions\": enabled_precisions,\n \"debug\": debug,\n \"workspace_size\": workspace_size,\n \"min_block_size\": min_block_size,\n \"torch_executed_ops\": torch_executed_ops,\n}\n\n# Build and compile the model with torch.compile, using Torch-TensorRT backend\noptimized_model = torch.compile(\n model,\n backend=\"torch_tensorrt\",\n dynamic=False,\n options=compilation_kwargs,\n)\noptimized_model(*inputs)" ] }, { @@ -150,7 +150,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.0" + "version": "3.11.7" } }, "nbformat": 4, diff --git a/docs/_downloads/dfa60e8f9850fd7761f3e7da81304d32/torch_compile_transformers_example.py b/docs/_downloads/dfa60e8f9850fd7761f3e7da81304d32/torch_compile_transformers_example.py index 5422f9cc1d..01d46e96f6 100644 --- a/docs/_downloads/dfa60e8f9850fd7761f3e7da81304d32/torch_compile_transformers_example.py +++ b/docs/_downloads/dfa60e8f9850fd7761f3e7da81304d32/torch_compile_transformers_example.py @@ -61,6 +61,7 @@ optimized_model = torch.compile( model, backend="torch_tensorrt", + dynamic=False, options=compilation_kwargs, ) optimized_model(*inputs) diff --git a/docs/_modules/index.html b/docs/_modules/index.html index 8a50771b6c..5ced90b5d8 100644 --- a/docs/_modules/index.html +++ b/docs/_modules/index.html @@ -9,7 +9,7 @@ -
Overview: module code — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Overview: module code — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -234,7 +234,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -301,6 +301,9 @@Python API Documenation
-
@@ -407,12 +410,10 @@
- Full API¶
+Full API¶
- Directories¶
+Directories¶
- -std::set<DataType> enabled_precisions = {DataType::kFloat}¶
Struct Documentation + diff --git a/docs/_cpp_api/structtorch__tensorrt_1_1torchscript_1_1CompileSpec.html b/docs/_cpp_api/structtorch__tensorrt_1_1torchscript_1_1CompileSpec.html index 4d9cad0fba..5276580537 100644 --- a/docs/_cpp_api/structtorch__tensorrt_1_1torchscript_1_1CompileSpec.html +++ b/docs/_cpp_api/structtorch__tensorrt_1_1torchscript_1_1CompileSpec.html @@ -10,7 +10,7 @@ -
Struct CompileSpec — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Struct CompileSpec — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@Python API Documenation
-
@@ -416,15 +419,15 @@
- Struct CompileSpec¶
+Struct CompileSpec¶
- Struct Documentation¶
+Struct Documentation¶
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
Struct Documentation
- -TORCHTRT_API Input(std::vector<int64_t> shape, TensorFormat format = TensorFormat::kContiguous)¶
- Struct Input¶
+Struct Input¶
- Inheritance Relationships¶
+Inheritance Relationships¶
- Base Type¶
+Base Type¶
-
-
- Struct Documentation¶
+Struct Documentation¶
- Struct GraphInputs¶
+Struct GraphInputs¶
- Struct Documentation¶
+Struct Documentation¶
- -inline operator Value() const
- Program Listing for File logging.h¶
+Program Listing for File logging.h¶
↰ Return to documentation for file (
-cpp/include/torch_tensorrt/logging.h)/* - * Copyright (c) NVIDIA Corporation. - * All rights reserved. - * - * This library is licensed under the BSD-style license found in the - * LICENSE file in the root directory of this source tree. - */ -#pragma once - -#include <string> -#include "torch_tensorrt/macros.h" - -namespace torch_tensorrt { -namespace logging { -enum Level { - kINTERNAL_ERROR, - kERROR, - kWARNING, - kINFO, - kDEBUG, - kGRAPH, -}; - -// Are these ones necessary for the user? -TORCHTRT_API std::string get_logging_prefix(); -TORCHTRT_API void set_logging_prefix(std::string prefix); - -TORCHTRT_API void set_reportable_log_level(Level lvl); - -TORCHTRT_API void set_is_colored_output_on(bool colored_output_on); - -TORCHTRT_API Level get_reportable_log_level(); - -TORCHTRT_API bool get_is_colored_output_on(); - -// Dont know if we want this? -TORCHTRT_API void log(Level lvl, std::string msg); -} // namespace logging -} // namespace torch_tensorrt +@@ -510,6 +513,7 @@ + diff --git a/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_macros.h.html b/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_macros.h.html index c733e0bb7a..47f77b4c0e 100644 --- a/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_macros.h.html +++ b/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_macros.h.html @@ -10,7 +10,7 @@ -/* + * Copyright (c) NVIDIA Corporation. + * All rights reserved. + * + * This library is licensed under the BSD-style license found in the + * LICENSE file in the root directory of this source tree. + */ +#pragma once + +#include <string> +#include "torch_tensorrt/macros.h" + +namespace torch_tensorrt { +namespace logging { +enum Level { + kINTERNAL_ERROR, + kERROR, + kWARNING, + kINFO, + kDEBUG, + kGRAPH, +}; + +// Are these ones necessary for the user? +TORCHTRT_API std::string get_logging_prefix(); +TORCHTRT_API void set_logging_prefix(std::string prefix); + +TORCHTRT_API void set_reportable_log_level(Level lvl); + +TORCHTRT_API void set_is_colored_output_on(bool colored_output_on); + +TORCHTRT_API Level get_reportable_log_level(); + +TORCHTRT_API bool get_is_colored_output_on(); + +// Dont know if we want this? +TORCHTRT_API void log(Level lvl, std::string msg); +} // namespace logging +} // namespace torch_tensorrt
Program Listing for File macros.h — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Program Listing for File macros.h — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@Python API Documenation
-
@@ -412,46 +415,46 @@
- Program Listing for File macros.h¶
+Program Listing for File macros.h¶
↰ Return to documentation for file (
-cpp/include/torch_tensorrt/macros.h)/* - * Copyright (c) NVIDIA Corporation. - * All rights reserved. - * - * This library is licensed under the BSD-style license found in the - * LICENSE file in the root directory of this source tree. - */ -#pragma once - -#if defined(USE_CMAKE_GENERATED_EXPORT_HEADER) -#include <torch_tensorrt_export.h> -#else -#if defined(__GNUC__) -#define TORCHTRT_API __attribute__((__visibility__("default"))) -#define TORCHTRT_HIDDEN __attribute__((__visibility__("hidden"))) -#else -#define TORCHTRT_API -#define TORCHTRT_HIDDEN -#endif // defined(__GNUC__) -#endif // defined(USE_CMAKE_GENERATED_EXPORT_HEADER) - -// Does this need to be gaurded or something? -#define XSTR(x) #x -#define STR(x) XSTR(x) - -#define TORCH_TENSORRT_MAJOR_VERSION 2 -#define TORCH_TENSORRT_MINOR_VERSION 3 -#define TORCH_TENSORRT_PATCH_VERSION 0 -#define TORCH_TENSORRT_VERSION \ - STR(TORCH_TENSORRT_MAJOR_VERSION) \ - "." STR(TORCH_TENSORRT_MINOR_VERSION) "." STR(TORCH_TENSORRT_PATCH_VERSION) - -// Setup namespace aliases for ease of use -namespace torch_tensorrt { -namespace torchscript {} -namespace ts = torchscript; -} // namespace torch_tensorrt -namespace torchtrt = torch_tensorrt; +@@ -509,6 +512,7 @@ + diff --git a/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_ptq.h.html b/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_ptq.h.html index 5277714042..223204dc60 100644 --- a/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_ptq.h.html +++ b/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_ptq.h.html @@ -10,7 +10,7 @@ -/* + * Copyright (c) NVIDIA Corporation. + * All rights reserved. + * + * This library is licensed under the BSD-style license found in the + * LICENSE file in the root directory of this source tree. + */ +#pragma once + +#if defined(USE_CMAKE_GENERATED_EXPORT_HEADER) +#include <torch_tensorrt_export.h> +#else +#if defined(__GNUC__) +#define TORCHTRT_API __attribute__((__visibility__("default"))) +#define TORCHTRT_HIDDEN __attribute__((__visibility__("hidden"))) +#else +#define TORCHTRT_API +#define TORCHTRT_HIDDEN +#endif // defined(__GNUC__) +#endif // defined(USE_CMAKE_GENERATED_EXPORT_HEADER) + +// Does this need to be gaurded or something? +#define XSTR(x) #x +#define STR(x) XSTR(x) + +#define TORCH_TENSORRT_MAJOR_VERSION 2 +#define TORCH_TENSORRT_MINOR_VERSION 4 +#define TORCH_TENSORRT_PATCH_VERSION 0 +#define TORCH_TENSORRT_VERSION \ + STR(TORCH_TENSORRT_MAJOR_VERSION) \ + "." STR(TORCH_TENSORRT_MINOR_VERSION) "." STR(TORCH_TENSORRT_PATCH_VERSION) + +// Setup namespace aliases for ease of use +namespace torch_tensorrt { +namespace torchscript {} +namespace ts = torchscript; +} // namespace torch_tensorrt +namespace torchtrt = torch_tensorrt;
Program Listing for File ptq.h — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Program Listing for File ptq.h — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@Python API Documenation
-
@@ -412,186 +415,181 @@
- Program Listing for File ptq.h¶
+Program Listing for File ptq.h¶
↰ Return to documentation for file (
-cpp/include/torch_tensorrt/ptq.h)/* - * Copyright (c) NVIDIA Corporation. - * All rights reserved. - * - * This library is licensed under the BSD-style license found in the - * LICENSE file in the root directory of this source tree. - */ -#pragma once - -#include <fstream> -#include <iostream> -#include <iterator> -#include <memory> -#include <sstream> -#include <string> -#include <vector> - -#include "NvInfer.h" -#include "torch/torch.h" -#include "torch_tensorrt/logging.h" -#include "torch_tensorrt/macros.h" - -#ifndef DOXYGEN_SHOULD_SKIP_THIS -namespace nvinfer1 { -class IInt8Calibrator; -class IInt8EntropyCalibrator2; -} // namespace nvinfer1 - -namespace torch_tensorrt { -namespace ptq { -TORCHTRT_API bool get_batch_impl(void* bindings[], const char* names[], int nbBindings, torch::Tensor& data); -} -} // namespace torch_tensorrt -#endif // DOXYGEN_SHOULD_SKIP_THIS - -namespace torch_tensorrt { -namespace ptq { - -template <typename Algorithm, typename DataLoaderUniquePtr> -class Int8Calibrator : Algorithm { - using DataLoader = typename DataLoaderUniquePtr::element_type; - using Batch = typename DataLoader::super::BatchType; - - public: - Int8Calibrator(DataLoaderUniquePtr dataloader, const std::string& cache_file_path, bool use_cache) - : dataloader_(dataloader.get()), cache_file_path_(cache_file_path), use_cache_(use_cache) { - for (auto batch : *dataloader_) { - batched_data_.push_back(batch.data); - } - it_ = batched_data_.begin(); - } - - int getBatchSize() const noexcept override { - // HACK: Torch-TensorRT only uses explict batch sizing, INT8 Calibrator does not - // work when reporting the batch size here and having explicity batching. - // So we just report batch size 1 (warnings will still be printed out). - return 1; - // return static_cast<int>(dataloader_->options().batch_size); - } - - bool getBatch(void* bindings[], const char* names[], int nbBindings) noexcept override { - if (it_ != batched_data_.end()) { - auto status = get_batch_impl(bindings, names, nbBindings, *it_); - it_ = ++it_; - return status; - } else { - // Reset iterator if incase calibrator is going to be used again - it_ = batched_data_.begin(); - return false; - } - } - - const void* readCalibrationCache(size_t& length) noexcept override { - if (use_cache_) { - std::stringstream ss; - ss << "Reading Calibration Cache from " << cache_file_path_; - logging::log(logging::Level::kINFO, ss.str()); - - cache_.clear(); - std::ifstream input(cache_file_path_, std::ios::binary); - input >> std::noskipws; - if (input.good()) { - std::copy(std::istream_iterator<char>(input), std::istream_iterator<char>(), std::back_inserter(cache_)); - logging::log(logging::Level::kDEBUG, "Cache read"); - } - length = cache_.size(); - return length ? cache_.data() : nullptr; - } - return nullptr; - } - - void writeCalibrationCache(const void* cache, size_t length) noexcept override { - std::ofstream cache_file(cache_file_path_, std::ios::binary); - cache_file.write(reinterpret_cast<const char*>(cache), length); - std::stringstream ss; - ss << "Saved Calibration Cache to " << cache_file_path_; - logging::log(logging::Level::kINFO, ss.str()); - } - - operator nvinfer1::IInt8Calibrator*() { - return reinterpret_cast<nvinfer1::IInt8Calibrator*>(this); - } - - private: - DataLoader* dataloader_; - const std::string& cache_file_path_; - size_t cache_size_ = 0; - bool use_cache_; - std::vector<char> cache_; - std::vector<torch::Tensor> batched_data_; - std::vector<torch::Tensor>::iterator it_; -}; - -template <typename Algorithm> -class Int8CacheCalibrator : Algorithm { - public: - Int8CacheCalibrator(const std::string& cache_file_path) : cache_file_path_(cache_file_path) {} - - int getBatchSize() const noexcept override { - // HACK: Torch-TensorRT only uses explict batch sizing, INT8 Calibrator does not - // work when reporting the batch size here and having explicity batching. - // So we just report batch size 1 (warnings will still be printed out). - return 1; - } - - bool getBatch(void* bindings[], const char* names[], int nbBindings) noexcept override { - return false; - } - - const void* readCalibrationCache(size_t& length) noexcept override { - std::stringstream ss; - ss << "Reading Calibration Cache from " << cache_file_path_; - logging::log(logging::Level::kINFO, ss.str()); - - cache_.clear(); - std::ifstream input(cache_file_path_, std::ios::binary); - input >> std::noskipws; - if (input.good()) { - std::copy(std::istream_iterator<char>(input), std::istream_iterator<char>(), std::back_inserter(cache_)); - logging::log(logging::Level::kDEBUG, "Cache read"); - } - length = cache_.size(); - return length ? cache_.data() : nullptr; - } - - void writeCalibrationCache(const void* cache, size_t length) noexcept override { - std::ofstream cache_file(cache_file_path_, std::ios::binary); - cache_file.write(reinterpret_cast<const char*>(cache), length); - std::stringstream ss; - ss << "Saved Calibration Cache to " << cache_file_path_; - logging::log(logging::Level::kINFO, ss.str()); - } - - operator nvinfer1::IInt8Calibrator*() { - return reinterpret_cast<nvinfer1::IInt8Calibrator*>(this); - } - - private: - const std::string& cache_file_path_; - size_t cache_size_ = 0; - std::vector<char> cache_; -}; - -template <typename Algorithm = nvinfer1::IInt8EntropyCalibrator2, typename DataLoader> -inline Int8Calibrator<Algorithm, DataLoader> make_int8_calibrator( - DataLoader dataloader, - const std::string& cache_file_path, - bool use_cache) { - return Int8Calibrator<Algorithm, DataLoader>(std::move(dataloader), cache_file_path, use_cache); -} - -template <typename Algorithm = nvinfer1::IInt8EntropyCalibrator2> -inline Int8CacheCalibrator<Algorithm> make_int8_cache_calibrator(const std::string& cache_file_path) { - return Int8CacheCalibrator<Algorithm>(cache_file_path); -} - -} // namespace ptq -} // namespace torch_tensorrt +@@ -649,6 +647,7 @@ + diff --git a/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_torch_tensorrt.h.html b/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_torch_tensorrt.h.html index c769ce9e6f..87f2ab086c 100644 --- a/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_torch_tensorrt.h.html +++ b/docs/_cpp_api/program_listing_file_cpp_include_torch_tensorrt_torch_tensorrt.h.html @@ -10,7 +10,7 @@ -/* + * Copyright (c) NVIDIA Corporation. + * All rights reserved. + * + * This library is licensed under the BSD-style license found in the + * LICENSE file in the root directory of this source tree. + */ +#pragma once + +#include <fstream> +#include <iostream> +#include <iterator> +#include <memory> +#include <sstream> +#include <string> +#include <vector> + +#include "NvInfer.h" +#include "torch/torch.h" +#include "torch_tensorrt/logging.h" +#include "torch_tensorrt/macros.h" + +#ifndef DOXYGEN_SHOULD_SKIP_THIS +namespace torch_tensorrt { +namespace ptq { +TORCHTRT_API bool get_batch_impl(void* bindings[], const char* names[], int nbBindings, torch::Tensor& data); +} +} // namespace torch_tensorrt +#endif // DOXYGEN_SHOULD_SKIP_THIS + +namespace torch_tensorrt { +namespace ptq { + +template <typename Algorithm, typename DataLoaderUniquePtr> +class Int8Calibrator : Algorithm { + using DataLoader = typename DataLoaderUniquePtr::element_type; + using Batch = typename DataLoader::super::BatchType; + + public: + Int8Calibrator(DataLoaderUniquePtr dataloader, const std::string& cache_file_path, bool use_cache) + : dataloader_(dataloader.get()), cache_file_path_(cache_file_path), use_cache_(use_cache) { + for (auto batch : *dataloader_) { + batched_data_.push_back(batch.data); + } + it_ = batched_data_.begin(); + } + + int getBatchSize() const noexcept override { + // HACK: Torch-TensorRT only uses explict batch sizing, INT8 Calibrator does not + // work when reporting the batch size here and having explicity batching. + // So we just report batch size 1 (warnings will still be printed out). + return 1; + // return static_cast<int>(dataloader_->options().batch_size); + } + + bool getBatch(void* bindings[], const char* names[], int nbBindings) noexcept override { + if (it_ != batched_data_.end()) { + auto status = get_batch_impl(bindings, names, nbBindings, *it_); + it_ = ++it_; + return status; + } else { + // Reset iterator if incase calibrator is going to be used again + it_ = batched_data_.begin(); + return false; + } + } + + const void* readCalibrationCache(size_t& length) noexcept override { + if (use_cache_) { + std::stringstream ss; + ss << "Reading Calibration Cache from " << cache_file_path_; + logging::log(logging::Level::kINFO, ss.str()); + + cache_.clear(); + std::ifstream input(cache_file_path_, std::ios::binary); + input >> std::noskipws; + if (input.good()) { + std::copy(std::istream_iterator<char>(input), std::istream_iterator<char>(), std::back_inserter(cache_)); + logging::log(logging::Level::kDEBUG, "Cache read"); + } + length = cache_.size(); + return length ? cache_.data() : nullptr; + } + return nullptr; + } + + void writeCalibrationCache(const void* cache, size_t length) noexcept override { + std::ofstream cache_file(cache_file_path_, std::ios::binary); + cache_file.write(reinterpret_cast<const char*>(cache), length); + std::stringstream ss; + ss << "Saved Calibration Cache to " << cache_file_path_; + logging::log(logging::Level::kINFO, ss.str()); + } + + operator nvinfer1::IInt8Calibrator*() { + return reinterpret_cast<nvinfer1::IInt8Calibrator*>(this); + } + + private: + DataLoader* dataloader_; + const std::string& cache_file_path_; + size_t cache_size_ = 0; + bool use_cache_; + std::vector<char> cache_; + std::vector<torch::Tensor> batched_data_; + std::vector<torch::Tensor>::iterator it_; +}; + +template <typename Algorithm> +class Int8CacheCalibrator : Algorithm { + public: + Int8CacheCalibrator(const std::string& cache_file_path) : cache_file_path_(cache_file_path) {} + + int getBatchSize() const noexcept override { + // HACK: Torch-TensorRT only uses explict batch sizing, INT8 Calibrator does not + // work when reporting the batch size here and having explicity batching. + // So we just report batch size 1 (warnings will still be printed out). + return 1; + } + + bool getBatch(void* bindings[], const char* names[], int nbBindings) noexcept override { + return false; + } + + const void* readCalibrationCache(size_t& length) noexcept override { + std::stringstream ss; + ss << "Reading Calibration Cache from " << cache_file_path_; + logging::log(logging::Level::kINFO, ss.str()); + + cache_.clear(); + std::ifstream input(cache_file_path_, std::ios::binary); + input >> std::noskipws; + if (input.good()) { + std::copy(std::istream_iterator<char>(input), std::istream_iterator<char>(), std::back_inserter(cache_)); + logging::log(logging::Level::kDEBUG, "Cache read"); + } + length = cache_.size(); + return length ? cache_.data() : nullptr; + } + + void writeCalibrationCache(const void* cache, size_t length) noexcept override { + std::ofstream cache_file(cache_file_path_, std::ios::binary); + cache_file.write(reinterpret_cast<const char*>(cache), length); + std::stringstream ss; + ss << "Saved Calibration Cache to " << cache_file_path_; + logging::log(logging::Level::kINFO, ss.str()); + } + + operator nvinfer1::IInt8Calibrator*() { + return reinterpret_cast<nvinfer1::IInt8Calibrator*>(this); + } + + private: + const std::string& cache_file_path_; + size_t cache_size_ = 0; + std::vector<char> cache_; +}; + +template <typename Algorithm = nvinfer1::IInt8EntropyCalibrator2, typename DataLoader> +inline Int8Calibrator<Algorithm, DataLoader> make_int8_calibrator( + DataLoader dataloader, + const std::string& cache_file_path, + bool use_cache) { + return Int8Calibrator<Algorithm, DataLoader>(std::move(dataloader), cache_file_path, use_cache); +} + +template <typename Algorithm = nvinfer1::IInt8EntropyCalibrator2> +inline Int8CacheCalibrator<Algorithm> make_int8_cache_calibrator(const std::string& cache_file_path) { + return Int8CacheCalibrator<Algorithm>(cache_file_path); +} + +} // namespace ptq +} // namespace torch_tensorrt
Program Listing for File torch_tensorrt.h — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Program Listing for File torch_tensorrt.h — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -235,7 +235,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -302,6 +302,9 @@Python API Documenation
-
@@ -412,350 +415,350 @@
- Program Listing for File torch_tensorrt.h¶
+Program Listing for File torch_tensorrt.h¶
↰ Return to documentation for file (
-cpp/include/torch_tensorrt/torch_tensorrt.h)/* - * Copyright (c) NVIDIA Corporation. - * All rights reserved. - * - * This library is licensed under the BSD-style license found in the - * LICENSE file in the root directory of this source tree. - */ - -#pragma once - -#include <cuda_runtime.h> -#include <iostream> -#include <memory> -#include <set> -#include <string> -#include <vector> -#include "torch/custom_class.h" - -#include "torch_tensorrt/macros.h" - -// Just include the .h? -#ifndef DOXYGEN_SHOULD_SKIP_THIS -namespace torch { -namespace jit { -struct Graph; -struct Module; -} // namespace jit -} // namespace torch - -namespace c10 { -enum class DeviceType : int8_t; -enum class ScalarType : int8_t; -template <class> -class ArrayRef; -} // namespace c10 - -namespace nvinfer1 { -class IInt8Calibrator; -} -#endif // DOXYGEN_SHOULD_SKIP_THIS - -namespace torch_tensorrt { -class DataType { - public: - enum Value : int8_t { - kLong, - kDouble, - kFloat, - kHalf, - kChar, - kInt, - kBool, - kUnknown - }; - - DataType() = default; - constexpr DataType(Value t) : value(t) {} - TORCHTRT_API DataType(c10::ScalarType t); - operator Value() const { - return value; - } - explicit operator bool() = delete; - constexpr bool operator==(DataType other) const { - return value == other.value; - } - constexpr bool operator==(DataType::Value other) const { - return value == other; - } - constexpr bool operator!=(DataType other) const { - return value != other.value; - } - constexpr bool operator!=(DataType::Value other) const { - return value != other; - } - - private: - friend TORCHTRT_API std::ostream& operator<<(std::ostream& os, const DataType& dtype); - Value value; -}; - -struct Device { - class DeviceType { - public: - enum Value : int8_t { - kGPU, - kDLA, - }; - - DeviceType() = default; - constexpr DeviceType(Value t) : value(t) {} - DeviceType(c10::DeviceType t); - operator Value() const { - return value; - } - explicit operator bool() = delete; - constexpr bool operator==(DeviceType other) const { - return value == other.value; - } - constexpr bool operator!=(DeviceType other) const { - return value != other.value; - } - - private: - Value value; - }; - - DeviceType device_type; - - /* - * Target gpu id - */ - int64_t gpu_id; - - /* - * When using DLA core on NVIDIA AGX platforms gpu_id should be set as Xavier device - */ - int64_t dla_core; - - bool allow_gpu_fallback; - - Device() : device_type(DeviceType::kGPU), gpu_id(0), dla_core(0), allow_gpu_fallback(false) {} -}; - -enum class EngineCapability : int8_t { - kSTANDARD, - kSAFETY, - kDLA_STANDALONE, -}; - -class TensorFormat { - public: - enum Value : int8_t { - kContiguous, - kChannelsLast, - kUnknown, - }; - - TensorFormat() = default; - constexpr TensorFormat(Value t) : value(t) {} - TORCHTRT_API TensorFormat(at::MemoryFormat t); - operator Value() const { - return value; - } - explicit operator bool() = delete; - constexpr bool operator==(TensorFormat other) const { - return value == other.value; - } - constexpr bool operator==(TensorFormat::Value other) const { - return value == other; - } - constexpr bool operator!=(TensorFormat other) const { - return value != other.value; - } - constexpr bool operator!=(TensorFormat::Value other) const { - return value != other; - } - - private: - friend TORCHTRT_API std::ostream& operator<<(std::ostream& os, const TensorFormat& format); - Value value; -}; - -struct Input : torch::CustomClassHolder { - std::vector<int64_t> min_shape; - std::vector<int64_t> opt_shape; - std::vector<int64_t> max_shape; - std::vector<int64_t> shape; - DataType dtype; - TensorFormat format; - std::vector<double> tensor_domain; - - Input() {} - TORCHTRT_API Input(std::vector<int64_t> shape, TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input( - std::vector<int64_t> shape, - std::vector<double> tensor_domain, - TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input(std::vector<int64_t> shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input( - std::vector<int64_t> shape, - DataType dtype, - std::vector<double> tensor_domain, - TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input( - c10::ArrayRef<int64_t> shape, - std::vector<double> tensor_domain, - TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input( - c10::ArrayRef<int64_t> shape, - DataType dtype, - std::vector<double> tensor_domain, - TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input( - std::vector<int64_t> min_shape, - std::vector<int64_t> opt_shape, - std::vector<int64_t> max_shape, - TensorFormat format = TensorFormat::kContiguous); - TORCHTRT_API Input( - std::vector<int64_t> min_shape, - std::vector<int64_t> opt_shape, - std::vector<int64_t> max_shape, - std::vector<double> tensor_domain, - TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input( - std::vector<int64_t> min_shape, - std::vector<int64_t> opt_shape, - std::vector<int64_t> max_shape, - DataType dtype, - TensorFormat format = TensorFormat::kContiguous); - - TORCHTRT_API Input( - std::vector<int64_t> min_shape, - std::vector<int64_t> opt_shape, - std::vector<int64_t> max_shape, - DataType dtype, - std::vector<double> tensor_domain, - TensorFormat format = TensorFormat::kContiguous); +@@ -813,6 +816,7 @@ + diff --git a/docs/_cpp_api/structtorch__tensorrt_1_1Device.html b/docs/_cpp_api/structtorch__tensorrt_1_1Device.html index 1ebec72914..ca8c5dc4d2 100644 --- a/docs/_cpp_api/structtorch__tensorrt_1_1Device.html +++ b/docs/_cpp_api/structtorch__tensorrt_1_1Device.html @@ -10,7 +10,7 @@ -/* + * Copyright (c) NVIDIA Corporation. + * All rights reserved. + * + * This library is licensed under the BSD-style license found in the + * LICENSE file in the root directory of this source tree. + */ + +#pragma once + +#include <cuda_runtime.h> +#include <iostream> +#include <memory> +#include <set> +#include <string> +#include <vector> +#include "torch/custom_class.h" + +#include "torch_tensorrt/macros.h" + +// Just include the .h? +#ifndef DOXYGEN_SHOULD_SKIP_THIS +namespace torch { +namespace jit { +struct Graph; +struct Module; +} // namespace jit +} // namespace torch + +namespace c10 { +enum class DeviceType : int8_t; +enum class ScalarType : int8_t; +template <class> +class ArrayRef; +} // namespace c10 + +namespace nvinfer1 { +class IInt8Calibrator; +} +#endif // DOXYGEN_SHOULD_SKIP_THIS + +namespace torch_tensorrt { +class DataType { + public: + enum Value : int8_t { + kLong, + kDouble, + kFloat, + kHalf, + kChar, + kInt, + kBool, + kUnknown + }; + + DataType() = default; + constexpr DataType(Value t) : value(t) {} + TORCHTRT_API DataType(c10::ScalarType t); + operator Value() const { + return value; + } + explicit operator bool() = delete; + constexpr bool operator==(DataType other) const { + return value == other.value; + } + constexpr bool operator==(DataType::Value other) const { + return value == other; + } + constexpr bool operator!=(DataType other) const { + return value != other.value; + } + constexpr bool operator!=(DataType::Value other) const { + return value != other; + } + + private: + friend TORCHTRT_API std::ostream& operator<<(std::ostream& os, const DataType& dtype); + Value value; +}; + +struct Device { + class DeviceType { + public: + enum Value : int8_t { + kGPU, + kDLA, + }; + + DeviceType() = default; + constexpr DeviceType(Value t) : value(t) {} + DeviceType(c10::DeviceType t); + operator Value() const { + return value; + } + explicit operator bool() = delete; + constexpr bool operator==(DeviceType other) const { + return value == other.value; + } + constexpr bool operator!=(DeviceType other) const { + return value != other.value; + } + + private: + Value value; + }; + + DeviceType device_type; + + /* + * Target gpu id + */ + int64_t gpu_id; + + /* + * When using DLA core on NVIDIA AGX platforms gpu_id should be set as Xavier device + */ + int64_t dla_core; + + bool allow_gpu_fallback; + + Device() : device_type(DeviceType::kGPU), gpu_id(0), dla_core(0), allow_gpu_fallback(false) {} +}; + +enum class EngineCapability : int8_t { + kSTANDARD, + kSAFETY, + kDLA_STANDALONE, +}; + +class TensorFormat { + public: + enum Value : int8_t { + kContiguous, + kChannelsLast, + kUnknown, + }; + + TensorFormat() = default; + constexpr TensorFormat(Value t) : value(t) {} + TORCHTRT_API TensorFormat(at::MemoryFormat t); + operator Value() const { + return value; + } + explicit operator bool() = delete; + constexpr bool operator==(TensorFormat other) const { + return value == other.value; + } + constexpr bool operator==(TensorFormat::Value other) const { + return value == other; + } + constexpr bool operator!=(TensorFormat other) const { + return value != other.value; + } + constexpr bool operator!=(TensorFormat::Value other) const { + return value != other; + } + + private: + friend TORCHTRT_API std::ostream& operator<<(std::ostream& os, const TensorFormat& format); + Value value; +}; + +struct Input : torch::CustomClassHolder { + std::vector<int64_t> min_shape; + std::vector<int64_t> opt_shape; + std::vector<int64_t> max_shape; + std::vector<int64_t> shape; + DataType dtype; + TensorFormat format; + std::vector<double> tensor_domain; + + Input() {} + TORCHTRT_API Input(std::vector<int64_t> shape, TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input( + std::vector<int64_t> shape, + std::vector<double> tensor_domain, + TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input(std::vector<int64_t> shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input( + std::vector<int64_t> shape, + DataType dtype, + std::vector<double> tensor_domain, + TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input( + c10::ArrayRef<int64_t> shape, + std::vector<double> tensor_domain, + TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input( + c10::ArrayRef<int64_t> shape, + DataType dtype, + std::vector<double> tensor_domain, + TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input( + std::vector<int64_t> min_shape, + std::vector<int64_t> opt_shape, + std::vector<int64_t> max_shape, + TensorFormat format = TensorFormat::kContiguous); + TORCHTRT_API Input( + std::vector<int64_t> min_shape, + std::vector<int64_t> opt_shape, + std::vector<int64_t> max_shape, + std::vector<double> tensor_domain, + TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input( + std::vector<int64_t> min_shape, + std::vector<int64_t> opt_shape, + std::vector<int64_t> max_shape, + DataType dtype, + TensorFormat format = TensorFormat::kContiguous); + + TORCHTRT_API Input( + std::vector<int64_t> min_shape, + std::vector<int64_t> opt_shape, + std::vector<int64_t> max_shape, + DataType dtype, + std::vector<double> tensor_domain, + TensorFormat format = TensorFormat::kContiguous); - TORCHTRT_API Input( - c10::ArrayRef<int64_t> min_shape, - c10::ArrayRef<int64_t> opt_shape, - c10::ArrayRef<int64_t> max_shape, - TensorFormat format = TensorFormat::kContiguous); + TORCHTRT_API Input( + c10::ArrayRef<int64_t> min_shape, + c10::ArrayRef<int64_t> opt_shape, + c10::ArrayRef<int64_t> max_shape, + TensorFormat format = TensorFormat::kContiguous); - TORCHTRT_API Input( - c10::ArrayRef<int64_t> min_shape, - c10::ArrayRef<int64_t> opt_shape, - c10::ArrayRef<int64_t> max_shape, - std::vector<double> tensor_domain, - TensorFormat format = TensorFormat::kContiguous); + TORCHTRT_API Input( + c10::ArrayRef<int64_t> min_shape, + c10::ArrayRef<int64_t> opt_shape, + c10::ArrayRef<int64_t> max_shape, + std::vector<double> tensor_domain, + TensorFormat format = TensorFormat::kContiguous); - TORCHTRT_API Input( - c10::ArrayRef<int64_t> min_shape, - c10::ArrayRef<int64_t> opt_shape, - c10::ArrayRef<int64_t> max_shape, - DataType dtype, - TensorFormat format = TensorFormat::kContiguous); + TORCHTRT_API Input( + c10::ArrayRef<int64_t> min_shape, + c10::ArrayRef<int64_t> opt_shape, + c10::ArrayRef<int64_t> max_shape, + DataType dtype, + TensorFormat format = TensorFormat::kContiguous); - TORCHTRT_API Input( - c10::ArrayRef<int64_t> min_shape, - c10::ArrayRef<int64_t> opt_shape, - c10::ArrayRef<int64_t> max_shape, - DataType dtype, - std::vector<double> tensor_domain, - TensorFormat format = TensorFormat::kContiguous); + TORCHTRT_API Input( + c10::ArrayRef<int64_t> min_shape, + c10::ArrayRef<int64_t> opt_shape, + c10::ArrayRef<int64_t> max_shape, + DataType dtype, + std::vector<double> tensor_domain, + TensorFormat format = TensorFormat::kContiguous); - TORCHTRT_API Input(at::Tensor tensor); + TORCHTRT_API Input(at::Tensor tensor); - private: - friend TORCHTRT_API std::ostream& operator<<(std::ostream& os, const Input& input); - bool input_is_dynamic; -}; + private: + friend TORCHTRT_API std::ostream& operator<<(std::ostream& os, const Input& input); + bool input_is_dynamic; +}; -struct GraphInputs { - torch::jit::IValue input_signature; // nested Input, full input spec - std::vector<Input> inputs; // flatten input spec -}; +struct GraphInputs { + torch::jit::IValue input_signature; // nested Input, full input spec + std::vector<Input> inputs; // flatten input spec +}; -TORCHTRT_API std::string get_build_info(); +TORCHTRT_API std::string get_build_info(); -TORCHTRT_API void dump_build_info(); +TORCHTRT_API void dump_build_info(); -TORCHTRT_API void set_device(const int gpu_id); +TORCHTRT_API void set_device(const int gpu_id); -namespace torchscript { -struct CompileSpec { - TORCHTRT_API CompileSpec(std::vector<std::vector<int64_t>> fixed_sizes); +namespace torchscript { +struct CompileSpec { + TORCHTRT_API CompileSpec(std::vector<std::vector<int64_t>> fixed_sizes); - TORCHTRT_API CompileSpec(std::vector<c10::ArrayRef<int64_t>> fixed_sizes); + TORCHTRT_API CompileSpec(std::vector<c10::ArrayRef<int64_t>> fixed_sizes); - TORCHTRT_API CompileSpec(std::vector<Input> inputs); + TORCHTRT_API CompileSpec(std::vector<Input> inputs); - TORCHTRT_API CompileSpec(torch::jit::IValue input_signature); - // Defaults should reflect TensorRT defaults for BuilderConfig + TORCHTRT_API CompileSpec(torch::jit::IValue input_signature); + // Defaults should reflect TensorRT defaults for BuilderConfig - GraphInputs graph_inputs; - std::set<DataType> enabled_precisions = {DataType::kFloat}; + GraphInputs graph_inputs; + std::set<DataType> enabled_precisions = {DataType::kFloat}; - bool disable_tf32 = false; + bool disable_tf32 = false; - bool sparse_weights = false; + bool sparse_weights = false; - bool refit = false; + bool refit = false; - bool debug = false; + bool debug = false; - bool truncate_long_and_double = false; + bool truncate_long_and_double = false; - bool allow_shape_tensors = false; + bool allow_shape_tensors = false; - Device device; + Device device; - EngineCapability capability = EngineCapability::kSTANDARD; + EngineCapability capability = EngineCapability::kSTANDARD; - uint64_t num_avg_timing_iters = 1; + uint64_t num_avg_timing_iters = 1; - uint64_t workspace_size = 0; + uint64_t workspace_size = 0; - uint64_t dla_sram_size = 1048576; + uint64_t dla_sram_size = 1048576; - uint64_t dla_local_dram_size = 1073741824; + uint64_t dla_local_dram_size = 1073741824; - uint64_t dla_global_dram_size = 536870912; + uint64_t dla_global_dram_size = 536870912; - nvinfer1::IInt8Calibrator* ptq_calibrator = nullptr; + nvinfer1::IInt8Calibrator* ptq_calibrator = nullptr; - bool require_full_compilation = false; + bool require_full_compilation = false; - uint64_t min_block_size = 3; + uint64_t min_block_size = 3; - std::vector<std::string> torch_executed_ops; + std::vector<std::string> torch_executed_ops; - std::vector<std::string> torch_executed_modules; -}; + std::vector<std::string> torch_executed_modules; +}; -TORCHTRT_API bool check_method_operator_support(const torch::jit::Module& module, std::string method_name); +TORCHTRT_API bool check_method_operator_support(const torch::jit::Module& module, std::string method_name); -TORCHTRT_API torch::jit::Module compile(const torch::jit::Module& module, CompileSpec info); +TORCHTRT_API torch::jit::Module compile(const torch::jit::Module& module, CompileSpec info); -TORCHTRT_API std::string convert_method_to_trt_engine( - const torch::jit::Module& module, - std::string method_name, - CompileSpec info); +TORCHTRT_API std::string convert_method_to_trt_engine( + const torch::jit::Module& module, + std::string method_name, + CompileSpec info); -TORCHTRT_API torch::jit::Module embed_engine_in_new_module( - const std::string& engine, - Device device, - const std::vector<std::string>& input_binding_names = std::vector<std::string>(), - const std::vector<std::string>& output_binding_names = std::vector<std::string>()); -} // namespace torchscript -} // namespace torch_tensorrt +TORCHTRT_API torch::jit::Module embed_engine_in_new_module( + const std::string& engine, + Device device, + const std::vector<std::string>& input_binding_names = std::vector<std::string>(), + const std::vector<std::string>& output_binding_names = std::vector<std::string>()); +} // namespace torchscript +} // namespace torch_tensorrt
Struct Device — Torch-TensorRT v2.3.0.dev0+85971ff documentation +Struct Device — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation @@ -237,7 +237,7 @@- v2.3.0.dev0+85971ff + v2.4.0.dev0+4dc9acfc9@@ -304,6 +304,9 @@Python API Documenation
-
@@ -416,24 +419,24 @@
- Struct Device¶
+Struct Device¶
- Nested Relationships¶
+Nested Relationships¶
- Nested Types¶
+Nested Types¶
- Struct Documentation¶
+Struct Documentation¶
- Directory torch_tensorrt¶
+Directory torch_tensorrt¶
↰ Parent directory (
cpp/include)Directory path:
cpp/include/torch_tensorrt- Files¶
+Files¶
Files + diff --git a/docs/_cpp_api/enum_logging_8h_1a130f65408ad8cbaee060f05e8db69558.html b/docs/_cpp_api/enum_logging_8h_1a130f65408ad8cbaee060f05e8db69558.html index 70aef1d65a..fed57faf1b 100644 --- a/docs/_cpp_api/enum_logging_8h_1a130f65408ad8cbaee060f05e8db69558.html +++ b/docs/_cpp_api/enum_logging_8h_1a130f65408ad8cbaee060f05e8db69558.html @@ -9,38 +9,48 @@ + +
Enum Level — Torch-TensorRT v2.4.0.dev0+4dc9acfc9 documentation + -Enum Level — Torch-TensorRT v1.4.0+7d1d80773 documentation - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - + + + + + + @@ -81,7 +91,19 @@
Class Documentation -
Class Documentation -
Class Documentation -
- Template Class Int8CacheCalibrator¶
+Template Class Int8CacheCalibrator¶
- Inheritance Relationships¶
+Inheritance Relationships¶
- Base Type¶
+Base Type¶
- Class Documentation¶
+Class Documentation¶
class Int8CacheCalibrator : private Algorithm¶
- -inline operator Value() const¶
