Merge branch 'master' of https://github.com/NVIDIA/trtorch into ptq

CONTRIBUTING.md

@@ -0,0 +1,76 @@
+# Contribution Guidelines
+
+### Developing TRTorch
+
+Do try to fill an issue with your feature or bug before filling a PR (op support is generally an exception as long as you provide tests to prove functionality). There is also a backlog (https://github.com/NVIDIA/TRTorch/issues) of issues which are tagged with the area of focus, a coarse priority level and whether the issue may be accessible to new contributors. Let us know if you are interested in working on a issue. We are happy to provide guidance and mentorship for new contributors. Though note, there is no claiming of issues, we prefer getting working code quickly vs. addressing concerns about "wasted work".
+
+#### Communication
+
+The primary location for discussion is GitHub issues. This is the best place for questions about the project and discussion about specific issues.
+
+We use the PyTorch Slack for communication about core development, integration with PyTorch and other communication that doesn't make sense in GitHub issues. If you need an invite, take a look at the [PyTorch README](https://github.com/pytorch/pytorch/blob/master/README.md) for instructions on requesting one.
+
+### Coding Guidelines
+
+- We generally follow the coding guidelines used in PyTorch, right now this is not strictly enforced, but match the style used in the code already
+
+- Avoid introducing unnecessary complexity into existing code so that maintainability and readability are preserved
+
+- Try to avoid commiting commented out code
+
+- Minimize warnings (and no errors) from the compiler
+
+- Make sure all converter tests and the core module testsuite pass
+
+- New features should have corresponding tests or if its a difficult feature to test in a testing framework, your methodology for testing.
+
+- Comment subtleties and design decisions
+
+- Document hacks, we can discuss it only if we can find it
+
+### Commits and PRs
+
+- Try to keep pull requests focused (multiple pull requests are okay). Typically PRs should focus on a single issue or a small collection of closely related issue.
+
+- Typically we try to follow the guidelines set by https://www.conventionalcommits.org/en/v1.0.0/ for commit messages for clarity. Again not strictly enforced.
+
+- #### Sign Your Work
+    We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
+
+    Any contribution which contains commits that are not Signed-Off will not be accepted.
+
+    To sign off on a commit you simply use the `--signoff` (or `-s`) option when committing your changes:
+
+        $ git commit -s -m "Add cool feature."
+
+    This will append the following to your commit message:
+
+        Signed-off-by: Your Name <your@email.com>
+
+    By doing this you certify the below:
+
+        Developer Certificate of Origin
+        Version 1.1
+
+        Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
+        1 Letterman Drive
+        Suite D4700
+        San Francisco, CA, 94129
+
+        Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
+
+
+        Developer's Certificate of Origin 1.1
+
+        By making a contribution to this project, I certify that:
+
+        (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or
+
+        (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or
+
+        (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.
+
+        (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.
+
+
+Thanks in advance for your patience as we review your contributions; we do appreciate them!

README.md

@@ -1,33 +1,70 @@
-# TRTorch 
+# TRTorch
 
 > Ahead of Time (AOT) compiling for PyTorch JIT
 
-## Compiling TRTorch 
+TRTorch is a compiler for PyTorch/TorchScript, targeting NVIDIA GPUs via NVIDIA's TensorRT Deep Learning Optimizer and Runtime. Unlike PyTorch's Just-In-Time (JIT) compiler, TRTorch is an Ahead-of-Time (AOT) compiler, meaning that before you deploy your TorchScript code, you go through an explicit compile step to convert a standard TorchScript program into an module targeting a TensorRT engine. TRTorch operates as a PyTorch extention and compiles modules that integrate into the JIT runtime seamlessly. After compilation using the optimized graph should feel no different than running a TorchScript module. You also have access to TensorRT's suite of configurations at compile time, so you are able to specify operating precision (FP32/F16) and other settings for your module.
+
+More Information / System Architecture:
+
+- [GTC 2020 Talk](https://developer.nvidia.com/gtc/2020/video/s21671)
+
+## Example Usage
+
+```c++
+#include "torch/script.h"
+#include "trtorch/trtorch.h"
+
+...
+auto compile_settings = trtorch::ExtraInfo(dims);
+// FP16 execution
+compile_settings.op_precision = torch::kHalf;
+// Compile module
+auto trt_mod = trtorch::CompileGraph(ts_mod, compile_settings);
+// Run like normal
+auto results = trt_mod.forward({in_tensor});
+...
+```
+
+## Platform Support
+
+| Platform | Support |
+| -------- | ------- |
+| Linux AMD64 / GPU   | **Supported** |
+| Linux aarch64 / GPU | **Planned/Possible with Native Compiation and small modifications to the build system** |
+| Linux aarch64 / DLA | **Planned/Possible with Native Compilation but untested** |
+| Windows / GPU       | - |
+| Linux ppc64le / GPU | - |
 
 ### Dependencies
 
 - Libtorch 1.4.0
 - CUDA 10.1
 - cuDNN 7.6
-- TensorRT 6.0.1.5
+- TensorRT 6.0.1
 
-Install TensorRT, CUDA and cuDNN on the system before starting to compile.
+## Prebuilt Binaries
+
+Releases: https://github.com/NVIDIA/TRTorch/releases
 
+## Compiling TRTorch
+
+Install TensorRT, CUDA and cuDNN on the system before starting to compile.
 
 ``` shell
-bazel build //:libtrtorch --cxxopt="-DNDEBUG"
+bazel build //:libtrtorch --compilation_mode=opt
 ```
 
-### Debug build 
+### Debug build
 ``` shell
 bazel build //:libtrtorch --compilation_mode=dbg
 ```
 
-A tarball with the include files and library can then be found in bazel-bin 
+A tarball with the include files and library can then be found in bazel-bin
 
-### Running TRTorch on a JIT Graph 
+### Running TRTorch on a JIT Graph
 
-> Make sure to add LibTorch's version of CUDA 10.1 to your LD_LIBRARY_PATH `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/bazel-TRTorch/external/libtorch/lib`
+> Make sure to add LibTorch to your LD_LIBRARY_PATH <br>
+>`export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/bazel-TRTorch/external/libtorch/lib`
 
 
 ``` shell
@@ -38,22 +75,28 @@ bazel run //cpp/trtorchexec -- $(realpath <PATH TO GRAPH>) <input-size>
 
 ### In TRTorch?
 
-Thanks for wanting to contribute! There are two main ways to handle supporting a new op. Either you can write a converter for the op from scratch and register it in the NodeConverterRegistry or if you can map the op to a set of ops that already have converters you can write a graph rewrite pass which will replace your new op with an equivalent subgraph of supported ops. Its preferred to use graph rewriting because then we do not need to maintain a large library of op converters. 
+Thanks for wanting to contribute! There are two main ways to handle supporting a new op. Either you can write a converter for the op from scratch and register it in the NodeConverterRegistry or if you can map the op to a set of ops that already have converters you can write a graph rewrite pass which will replace your new op with an equivalent subgraph of supported ops. Its preferred to use graph rewriting because then we do not need to maintain a large library of op converters. Also do look at the various op support trackers in the [issues](https://github.com/NVIDIA/TRTorch/issues) for information on the support status of various operators.
 
 ### In my application?
 
-> The Node Converter Registry is not exposed currently in the public API but you can try using internal headers. 
+> The Node Converter Registry is not exposed in the top level API but you can try using the internal headers shipped with the tarball.
 
-You can register a converter for your op using the NodeConverterRegistry inside your application. 
+You can register a converter for your op using the NodeConverterRegistry inside your application.
 
 ## Structure of the repo
 
 | Component     | Description                                                  |
 | ------------- | ------------------------------------------------------------ |
-| [**core**]()  | Main JIT ingest, lowering, conversion and execution implementations |
-| [**cpp**]()   | C++ API for TRTorch                                          |
-| [**tests**]() | Unit test for TRTorch                                        |
+| [**core**](core)  | Main JIT ingest, lowering, conversion and execution implementations |
+| [**cpp**](cpp)   | C++ specific components including API and example applications |
+| [**cpp/api**](cpp/api)   | C++ API for TRTorch |
+| [**tests**](tests) | Unit test for TRTorch |
+
+## Contributing
+
+Take a look at the [CONTRIBUTING.md](CONTRIBUTING.md)
+
 
 ## License
 
-The TRTorch license can be found in the LICENSE file.
+The TRTorch license can be found in the LICENSE file. It is licensed with a BSD Style licence

tests/modules/hub.py

@@ -10,7 +10,7 @@
           "inception": models.inception_v3(pretrained=True),
           "googlenet": models.googlenet(pretrained=True),
           "shufflenet": models.shufflenet_v2_x1_0(pretrained=True),
-          "mobilenet": models.mobilenet_v2(pretrained=True),
+          "mobilenet_v2": models.mobilenet_v2(pretrained=True),
           "resnext50_32x4d": models.resnext50_32x4d(pretrained=True),
           "wideresnet50_2": models.wide_resnet50_2(pretrained=True),
           "mnasnet": models.mnasnet1_0(pretrained=True),

tests/modules/test_compiled_modules.cpp

@@ -28,11 +28,9 @@ TEST_P(ModuleTests, CompiledModuleIsClose) {
 INSTANTIATE_TEST_SUITE_P(CompiledModuleForwardIsCloseSuite,
                          ModuleTests,
                          testing::Values(
-                             PathAndInSize({"tests/modules/lenet.jit.pt",
-                                            {{1,1,28,28}}}),
-                             PathAndInSize({"tests/modules/resnet18.jit.pt",
-                                             {{1,3,224,224}}}),
-                             PathAndInSize({"tests/modules/resnet50.jit.pt",
-                                            {{1,3,224,224}}}),
-                             PathAndInSize({"tests/modules/mobilenet_v2.jit.pt",
-                                            {{1,3,224,224}}})));
+                            PathAndInSize({"tests/modules/resnet18.jit.pt",
+                                          {{1,3,224,224}}}),
+                            PathAndInSize({"tests/modules/resnet50.jit.pt",
+                                          {{1,3,224,224}}}),
+                            PathAndInSize({"tests/modules/mobilenet_v2.jit.pt",
+                                          {{1,3,224,224}}})));

tests/modules/test_modules_as_engines.cpp

@@ -19,11 +19,9 @@ TEST_P(ModuleTests, ModuleAsEngineIsClose) {
 INSTANTIATE_TEST_SUITE_P(ModuleAsEngineForwardIsCloseSuite,
                          ModuleTests,
                          testing::Values(
-                             PathAndInSize({"tests/modules/lenet.jit.pt",
-                                            {{1,1,28,28}}}),
-                             PathAndInSize({"tests/modules/resnet18.jit.pt",
-                                            {{1,3,224,224}}}),
-                             PathAndInSize({"tests/modules/resnet50.jit.pt",
-                                            {{1,3,224,224}}}),
-                             PathAndInSize({"tests/modules/mobilenet_v2.jit.pt",
-                                            {{1,3,224,224}}})));
+                            PathAndInSize({"tests/modules/resnet18.jit.pt",
+                                          {{1,3,224,224}}}),
+                            PathAndInSize({"tests/modules/resnet50.jit.pt",
+                                          {{1,3,224,224}}}),
+                            PathAndInSize({"tests/modules/mobilenet_v2.jit.pt",
+                                          {{1,3,224,224}}})));