pytorch: 1.0.0 -> 1.2.0 #65041

stites · 2019-07-18T22:36:46Z

Motivation for this change

Closes #63073
Closes #65041
Closes #67468

This depends on @tbenst 's #61347 (thank you!). See #63073 and this PR thread for more details.

Work can be previewed at stites/pytorch-world. Building from source takes a long time, please check out the hosted cachix binaries

Changelog:

updates to pytorch-1.2.0
runs all tests except for utils
adds a cudaArchList argument that allows users to test on the latest nvidia hardware without waiting for this package.
adds mkl support to magma-2.5.0
adds cuda support to openMPI
adds openMPI support
allows users to build NamedTensors
allows users to get access to generated binaries from build process
adds .dev output which gives top-level access to generated C/C++ code (libtorch).
adds darwin support if cudaSupport is disabled.

Things done

tbenst

Thanks for working on this! Hopefully we can get this as feature complete as possible.

tbenst · 2019-07-21T09:41:45Z

pkgs/development/python-modules/pytorch/default.nix

@@ -72,6 +67,12 @@ in buildPythonPackage rec {
  PYTORCH_BUILD_VERSION = version;
  PYTORCH_BUILD_NUMBER = 0;

+  USE_FBGEMM = 0; # this can't build because of CMAKE downloads
+  USE_MKLDNN = 0; # mkl with cuda is broken
+  USE_NCCL = 0; # multigpu looks broken broken


Did you try including NCCL?

Yes, unfortunately. I reviewed setup.py and will try using their bundled NCCL (via USE_SYSTEM_NCCL=0), also setting NCCL variables manually (NCCL_ROOT_DIR, NCCL_LIB_DIR, NCCL_INCLUDE_DIR), and making sure that nccl.out and nccl.dev are included.

None of these seem to work. You will quickly (<10m) get the following errors:

nvlink error : Undefined reference to '_Z25ncclAllReduceRing_min_f32P14CollectiveArgs' in '/build/source/build/nccl/obj/collectives/device/functions.o' (target: sm_30) nvlink error : Undefined reference to '_Z27ncclAllReduceRingLL_min_f32P14CollectiveArgs' in '/build/source/build/nccl/obj/collectives/device/functions.o' (target: sm_30) nvlink error : Undefined reference to '_Z25ncclAllReduceTree_min_f32P14CollectiveArgs' in '/build/source/build/nccl/obj/collectives/device/functions.o' (target: sm_30) nvlink error : Undefined reference to '_Z27ncclAllReduceTreeLL_min_f32P14CollectiveArgs' in '/build/source/build/nccl/obj/collectives/device/functions.o' (target: sm_30) nvlink error : Undefined reference to '_Z25ncclAllReduceRing_min_f64P14CollectiveArgs' in '/build/source/build/nccl/obj/collectives/device/functions.o' (target: sm_30) nvlink error : Undefined reference to '_Z27ncclAllReduceRingLL_min_f64P14CollectiveArgs' in '/build/source/build/nccl/obj/collectives/device/functions.o' (target: sm_30) nvlink error : Undefined reference to '_Z25ncclAllReduceTree_min_f64P14CollectiveArgs' in '/build/source/build/nccl/obj/collectives/device/functions.o' (target: sm_30) nvlink error : Undefined reference to '_Z27ncclAllReduceTreeLL_min_f64P14CollectiveArgs' in '/build/source/build/nccl/obj/collectives/device/functions.o' (target: sm_30) make[5]: *** [Makefile:68: /build/source/build/nccl/obj/collectives/device/devlink.o] Error 255 make[4]: *** [Makefile:44: /build/source/build/nccl/obj/collectives/device/colldevice.a] Error 2 make[4]: *** Waiting for unfinished jobs....

There are a lot of these undefined references, all coming from /build/source/build/nccl/obj/collectives/device/functions.o.

~~Maybe this is because nvlink is attempting to perform checks in the GPU runtime?~~

Update: I found the symbols in "${nccl.dev}"/lib/libnccl_static.a but they seem to fail lookups on build, even when they are included in $NIX_LDFLAGS. Manually setting NCCL_* environment variables seems to fix this!

tbenst · 2019-07-21T09:43:58Z

pkgs/development/python-modules/pytorch/default.nix

@@ -36,14 +39,6 @@ in buildPythonPackage rec {
    sha256 = "076cpbig4sywn9vv674c0xdg832sdrd5pk1d0725pjkm436kpvlm";


Is this SHA correct? Looks the same to me as 1.0.0?

ah! Yes, I'm primarily working off of https://github.com/stites/libtorch-nix and missed this when transferring over details. Good catch -- I'm a bit surprised this slipped through since I was building off of my fork (using the command from the PR message).

tbenst · 2019-07-21T09:47:17Z

pkgs/development/python-modules/pytorch/default.nix

@@ -72,6 +67,12 @@ in buildPythonPackage rec {
  PYTORCH_BUILD_VERSION = version;
  PYTORCH_BUILD_NUMBER = 0;

+  USE_FBGEMM = 0; # this can't build because of CMAKE downloads


Worth reviewing the instructions at https://github.com/pytorch/FBGEMM. Looks like we can avoid downloads by setting ASMJIT_SRC_DIR, CPUINFO_SRC_DIR, GOOGLETEST_SOURCE_DIR.

So I did set these as environment variables, using pointers from your gist: https://gist.github.com/tbenst/15495f97ffe2a71dc18c49e247af8f51#file-pytorch_11-nix-L62

Unfortunately, these didn't seem to propagate through to CMake. I'll do a round-two later.

stites · 2019-07-24T20:03:34Z

pkgs/development/python-modules/pytorch/default.nix

@@ -72,6 +67,12 @@ in buildPythonPackage rec {
  PYTORCH_BUILD_VERSION = version;
  PYTORCH_BUILD_NUMBER = 0;

+  USE_FBGEMM = 0; # this can't build because of CMAKE downloads
+  USE_MKLDNN = 0; # mkl with cuda is broken


MKL with cuda is not broken, actually -- just MKLDNN. The problem seems to be a warning which is treated as an error:

/build/source/third_party/ideep/mkl-dnn/src/cpu/jit_uni_reorder_utils.cpp: In function ‘void mkldnn::impl::cpu::tr::prb_normalize(mkldnn::impl::cpu::tr::prb_t&)’: /build/source/third_party/ideep/mkl-dnn/src/cpu/jit_uni_reorder_utils.cpp:273:29: error: array subscript is above array bounds [-Werror=array-bounds] || p.nodes[j].os < p.nodes[min_pos].os ~~~~~~~~~^ /build/source/third_party/ideep/mkl-dnn/src/cpu/jit_uni_reorder_utils.cpp:276:37: error: array subscript is above array bounds [-Werror=array-bounds] && p.nodes[j].n < p.nodes[min_pos].n); ~~~~~~~~~^ cc1plus: all warnings being treated as errors make[2]: *** [third_party/ideep/mkl-dnn/src/CMakeFiles/mkldnn.dir/build.make:1272: third_party/ideep/mkl-dnn/src/CMakeFiles/mkldnn.dir/cpu/jit_uni_reorder_utils.cpp.o] Error 1 make[2]: *** Waiting for unfinished jobs....

Found the issue: pytorch/pytorch#22346

I missed the workaround at https://github.com/NixOS/nixpkgs/blob/fb76f093c9ab204fbec9dc9c56cb9b4a81f43bc7/pkgs/development/python-modules/pytorch/default.nix#L79

I'll be adding a mklSupport ? false which will swap out the numpy BLAS implementation so that this is more explicit. The working code is similar to my_magma:

my_numpy = if mklSupport && numpy.blasImplementation != "mkl" then numpy.override { blas = mkl; } else numpy;

pkgs/development/python-modules/pytorch/default.nix

stites · 2019-07-25T16:53:26Z

I've just rebased this on top of master, which means I need to cherry-pick and remove the #61347 commit (65494f2) before this is finalized. I don't think this should be a problem since I believe everything needs to be squashed anyways.

I've updated the PR with the following:

NCCL support is back
DISTRIBUTED is turned back on
OpenMPI support is now available (I needed to get the latest master so that I could update it's default.nix with CUDA support). I am tempted to change openMPISupport to distributedSupport since they seem synonymous.
SHA has been corrected
mkl support is synced with numpy and correctly builds MKL_DNN
ninja and setuptools have been added to the build (pytorch makes them optional, but I think this speeds up the build a bit).

Things that still need to be worked on (a check means that I have these working locally and/or in libtorch-nix).

FBGEMM support
syncing mkl support with Magma (mentioned in magma: 2.0.2 -> 2.5.0 #61347 (comment)).
multiprocessing tests fail with OpenMPI+CUDA due to a failed runtime lookup of libmpi

For updates on this work please check out https://github.com/stites/libtorch-nix. @tscholak is also experimentally building the C++ libtorch on darwin and might also be able to verify pytorch compatibility on OSX (possibly disabling cuda support).

stites · 2019-07-26T16:53:55Z

FBGEMM works in https://github.com/stites/libtorch-nix/blob/master/deps/fbgemm.nix but, unfortunately, setup.py seems to hide arguments passed into cmakeFlags from cmake.

The problem, found in #61820, is that pytorch didn't have the fbgemm dependencies under submodule control until a commit in May (1.1.0 was released in April). Unfortunately, git submodules cannot be patched. Furthermore $src is read-only, so hard-coding cmake variables (via a preconfigure command: sed -i '10a\\\nset(SOURCE_DIR ${source})' $src/CMakeLists.txt) is also not an option.

Since FBGEMM is optional, I think the most simple solution is to put a version bound that disables FBGEMM unless pytorch is >1.1.0. This way FBGEMM will automatically turn on if someone bumps the version and forgets about this workaround. According to the milestones, it looks like 1.1.1 might be cut for release soon.

stites · 2019-07-30T15:41:05Z

I asked around on the pytorch slack and the 1.1.1 release is going to be cut this week, so I think waiting is the best course of action.

For posterity, if anyone is trying to build pytorch-1.1.0, this PR is fully-functional without FBGEMM -- which provides optional, low-precision general matrix multiplication for small batch sizes, various MM-based optimizations, and fused operators. If you are working off of an x86 CPU backend, however, having FBGEMM might be a bit of a nessecity.

stites · 2019-08-08T16:21:15Z

https://github.com/pytorch/pytorch/releases/tag/v1.2.0

Looks like they skipped the 1.1.1 release. I'll update this PR for pytorch-1.2.0.

Update: unfortunately, pytorch-1.2.0 brings back the NCCL issues.

Here is the build matrix I am working with:

Python-3.6.9 support	pytorch-v1.1.0	pytorch-v1.2.0
vanilla	✔️	✔️
mkl+mkldnn	✔️	✔️
openMPI	✔️	✔️
FBGEMM	❌	✔️
Full CPU (all above)*	✔️	✔️
Full CPU + NamedTensors + Binaries	❕	✔️
CUDA	✔️	✔️
CUDA+NCCL	✔️	✔️
CUDA+NCCL+openMPI	✔️	✔️
CUDA+NCCL+openMPI+mkl	✔️	✔️
Full CUDA (all above)*	✔️	✔️
Full CUDA + NamedTensors + Binaries	❕	✔️

The ":grey_exclamation:" means that namedtensors and binaries weren't attempted in pytorch-v1.1.0.

The "*" implies that FBGEMM is not included in any of the "Full" builds in pytorch-v1.1.0, but is included for all of pytorch-v1.2.0

I've also started including previously ignored tests for pytorch-1.2.0. It looks like we can get away with only disabling utils which relies on making a git-clone.

stites · 2019-08-14T16:09:47Z

Still working on this, slowly, in pytorch-world and am updating the above build matrix to cut down on github notifications -- I'll make a final commit and message eventually. I am also going to set platforms = with lib.platforms; [linux] ++ lib.optionals (!cudaSupport) [darwin]; unless there are any objections.

stites · 2019-08-14T22:49:04Z

I made an update-worthy breakthrough for pytorch-1.2.0. The errors I saw weren't with NCCL, but with the 3.0 CUDA architectures, documented in pytorch/pytorch#23573. I'm currently trying to figure out how to polish this up and did notice that cmake was throwing this warning:

CMake Warning at cmake/public/utils.cmake:172 (message):
  In the future we will require one to explicitly pass TORCH_CUDA_ARCH_LIST
  to cmake instead of implicitly setting it as an env variable.  This will
  become a FATAL_ERROR in future version of pytorch.

Which is quite troubling. So far, we have been using buildPythonPackage to install pytorch, but I think we could can use the Adjust Build Options instructions for more flexibility around cmake (so this would have cracked the FBGEMM problem in 1.1.0), as well as future proof the pytorch derivation.

I'll be looking at adjusting buildPythonPackage next.

NOTE: Because of sandboxing, the build can't auto-detect the hardware's cuda architecture, so there is also now a problem around new architectures not being supported until explicitly added to this derivation.

stites · 2019-08-20T16:22:10Z

buildPythonPackage is actually a bit too much for me right now, this has been added as a comment in https://github.com/stites/pytorch-world and will be slotted for "future work." All variants of pytorch-1.2 have been built and tested on python-3.6 -- cachix binaries are also available.

I'll update this PR this week -- I've also added myself as a maintainer for this package.

stites · 2019-08-29T20:00:13Z

I've gone ahead and pushed up the latest changes, pytorch-1.2.0 is fully functional! Note that this is still blocked by #61347 and I can squash the commits if desired, but this PR now touches openmpi (adding cuda support to it) and magma (adding mkl support to it).

Final notes:

I've added a cudaArchList because of notes above regarding sandboxing -- this should allow users to test on the latest gpu hardware without waiting for this package. The arch lists haven't been vetted and I just notice a minute ago that cuda-9 was broken, so I've split it up into -9 and -10 versions. This logic feels dirty.
I've added dev outputs -- this is a quick hack that gives users like hasktorch (@tscholak) the generated C/C++ backend (thanks @cleverca22!). I haven't tested this, but you have to use python36Packages.callPackage (example here). There's probably a better way to do this (possibly promoting pytorch to top-level and using the "Adjust Build Options" instructions).
Because the original derivation mutates numpy via passthrough, it is very easy to get a "duplicate numpy package found." I am thinking it would be safer to assert numpy.blasImplementation == "mkl" if mklSupport is present, but I also used the same trick for magma-mkl support and openmpi-cuda support, so ¯_(ツ)_/¯.

This is ready for review. Again, not sure if I need to squash all of this or split up the PR into parts. Because I just found out how to properly support cuda-9, cachix binaries are pending -- I'll comment one last time to ping folks who want to test-drive this (although only the vanilla cuda-9 build is in there).

stites · 2019-10-17T21:19:08Z

Rebased! After this PR, I'll start on 1.3.0

teh · 2019-10-20T16:22:53Z

@jonringer good to merge?

jonringer · 2019-10-20T17:01:04Z

even with my 3900X, this takes a while to build maxing out all 24 threads >.>

jonringer

is caffe2 supposed to be bundled with pytorch?

[nix-shell:/home/jon/.cache/nix-review/pr-65041-1]$ tree -L 4 ./results/python37Packages.pytorchWithCuda/
./results/python37Packages.pytorchWithCuda/
├── bin
│   ├── convert-caffe2-to-onnx
│   └── convert-onnx-to-caffe2
└── lib
    └── python3.7
        └── site-packages
            ├── caffe2
            ├── torch
            └── torch-1.2.0.dist-info

if so, please add click to the propagatedBuildInputs:

[nix-shell:/home/jon/.cache/nix-review/pr-65041-1]$ ./results/python37Packages.pytorchWithCuda/bin/convert-caffe2-to-onnx --help
Traceback (most recent call last):
  File "/nix/store/cbrg1ldydrl4wb6v4xpkkfh2jbmhwaiv-python3.7-pytorch-1.2.0/bin/.convert-caffe2-to-onnx-wrapped", line 7, in <module>
    from caffe2.python.onnx.bin.conversion import caffe2_to_onnx
  File "/nix/store/5ck5ncbzq8pr9aynzyp3j2y0bhk0v42i-python3.7-pytorch-1.2.0/lib/python3.7/site-packages/caffe2/python/onnx/bin/conversion.py", line 11, in <module>
    import click
ModuleNotFoundError: No module named 'click'

jonringer · 2019-10-20T18:09:16Z

closes #69424

stites · 2019-10-20T18:36:10Z

I believe caffe2 is used for the "production-ready" features of pytorch.

I've added click to the propagatedBuildInputs. My ML box is currently down (just moved offices), so this is a bit of an optimistic push.

jonringer · 2019-10-20T18:37:55Z

I'm building and testing now. If it seems to work in the trivial case, I'll merge :)

jonringer · 2019-10-20T18:41:18Z

@GrahamcOfBorg eval

jonringer · 2019-10-20T18:44:44Z

not sure how much you use onnx, but I added a cpu-only version to nixpkgs #67542

jonringer

nix-review passes on NixOS (python38Packages.pytest is broken)
diff LGTM

[nix-shell:/home/jon/.cache/nix-review/pr-65041-2]$ python
Python 3.7.4 (default, Jul  8 2019, 18:31:06)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.2.0'

[3 built (2 failed), 0.0 MiB DL]
error: build of '/nix/store/zqv58qrag64v4db7ljcmk62r3w4vjjd7-env.drv' failed
https://github.com/NixOS/nixpkgs/pull/65041
3 package failed to build:
python38Packages.pytorch python38Packages.pytorchWithCuda python38Packages.torchvision

4 package were build:
magma python37Packages.pytorch python37Packages.pytorchWithCuda python37Packages.torchvision

the caffe2 executables still need some love, but I'll do that in a different PR seeing as this one has been going on for close to 3 months

jonringer

ooops, to conform to contirbuting.md, please break out the work into the following commits:

maintainers: add tscholak
openmpi: enable cuda support
magma: use cudatoolkit
python3Packages.pytorch: 1.0.0 -> 1.2.0

stites · 2019-10-23T15:56:29Z

quick ping to mention that this change was made!

jonringer · 2019-10-23T16:35:57Z

I might wait for #71780 to get in, as it has significant improvements to python38. And it's really hard to not have 480 commits bit rot fast

jonringer · 2019-10-23T17:57:25Z

fridh said he would de-conflict the merge at a later time.

stites · 2019-10-24T01:03:26Z

Heads-up: an initial build of pytorch-1.3.0 succeeds on python3.7 with and without cuda. Test are failing do to some dependencies issues.

I'm also about to be very busy for the next week -- expect a PR after.

jonringer · 2019-10-24T01:04:30Z

no sweat, take your time man. We appreciate your effort :)

stites · 2019-10-24T03:55:57Z

~~vanilla pytorch-1.3.0 builds~~ with a bumped tensorboard-2.0 (incoming PR). Triggering a full dependency grid-sweep now with tests.

Looks like there is still an issue in the grid sweep. pytorch-1.3 will be delayed a bit.

danieldk · 2019-12-06T09:14:33Z

I think 2947c23 should also be backported to 19.09? magma builds with CUDA on 19.09 fail with:

In file included from /nix/store/lzwx11791zlw77swqakrzjrkm5mz84cq-cudatoolkit-10.0.130/include/cuda_runtime.h:83,
                 from <command-line>:
/nix/store/lzwx11791zlw77swqakrzjrkm5mz84cq-cudatoolkit-10.0.130/include/crt/host_config.h:129:2: error: #error -- unsupported GNU version! gcc versions later than 7 are not supported!
 #error -- unsupported GNU version! gcc versions later than 7 are not supported!
  ^~~~~

stites requested a review from FRidh as a code owner July 18, 2019 22:36

ofborg bot added the 6.topic: python label Jul 18, 2019

ofborg bot requested a review from thoughtpolice July 18, 2019 22:48

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100 labels Jul 18, 2019

tbenst reviewed Jul 21, 2019

View reviewed changes

stites commented Jul 24, 2019

View reviewed changes

pkgs/development/python-modules/pytorch/default.nix Outdated Show resolved Hide resolved

stites mentioned this pull request Jul 25, 2019

magma: 2.0.2 -> 2.5.0 #61347

Merged

10 tasks

stites force-pushed the pytorch_11 branch from fb76f09 to 215c8bd Compare July 25, 2019 16:45

stites changed the title ~~pytorch: 1.0.0 -> 1.1.0~~ pytorch: 1.0.0 -> 1.1.0 (WIP, depends on #61347) Jul 25, 2019

stites mentioned this pull request Jul 26, 2019

fetchgit: fetchSubmodules is not recursive #61820

Closed

samueldr added the 2.status: work-in-progress This PR isn't done label Jul 27, 2019

This was referenced Aug 4, 2019

Pin nix pkgs to a hash hasktorch/hasktorch#143

Closed

Nix support for hasktorch-0.2.0.0 hasktorch/hasktorch#152

Closed

stites changed the title ~~pytorch: 1.0.0 -> 1.1.0 (WIP, depends on #61347)~~ pytorch: 1.0.0 -> 1.2.0 (WIP, depends on #61347) Aug 8, 2019

tbenst mentioned this pull request Aug 21, 2019

Pytorch build fails with MKL #61728

Closed

jethrokuan mentioned this pull request Aug 26, 2019

pytorchWithCuda fails to run #67468

Closed

stites changed the title ~~pytorch: 1.0.0 -> 1.2.0 (WIP, depends on #61347)~~ pytorch: 1.0.0 -> 1.2.0 (depends on #61347) Aug 29, 2019

ofborg bot added the 8.has: clean-up label Aug 29, 2019

ofborg bot requested a review from wavewave August 29, 2019 20:16

ofborg bot added the 10.rebuild-linux: 1-10 label Aug 29, 2019

ofborg bot requested a review from teh October 17, 2019 21:25

jonringer reviewed Oct 20, 2019

View reviewed changes

stites force-pushed the pytorch_11 branch from beaef89 to 901bcab Compare October 20, 2019 18:32

jonringer approved these changes Oct 20, 2019

View reviewed changes

jonringer suggested changes Oct 20, 2019

View reviewed changes

stites added 4 commits October 20, 2019 17:04

maintainers: add tscholak

b00050e

openmpi: enable cuda support

35afec6

magma: use cudatoolkit

2947c23

python3Packages.pytorch: 1.0.0 -> 1.2.0

ef8736e

stites force-pushed the pytorch_11 branch from 901bcab to ef8736e Compare October 20, 2019 21:09

jonringer approved these changes Oct 23, 2019

View reviewed changes

jonringer merged commit dfa8e8b into NixOS:master Oct 23, 2019

stites deleted the pytorch_11 branch October 23, 2019 19:09

stites mentioned this pull request Oct 24, 2019

[19.03] tensorflow-tensorboard requires tensorflow python module to be importable #61173

Closed

stites added a commit to stites/pytorch-world that referenced this pull request Dec 17, 2019

bump with merge NixOS/nixpkgs#65041

edcb01a

		@@ -36,14 +39,6 @@ in buildPythonPackage rec {
		sha256 = "076cpbig4sywn9vv674c0xdg832sdrd5pk1d0725pjkm436kpvlm";

pytorch: 1.0.0 -> 1.2.0 #65041

pytorch: 1.0.0 -> 1.2.0 #65041

Conversation

stites commented Jul 18, 2019 • edited Loading

Motivation for this change

Things done

tbenst left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stites Jul 24, 2019 • edited Loading

Choose a reason for hiding this comment

stites Jul 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stites commented Jul 25, 2019 • edited Loading

stites commented Jul 26, 2019 • edited Loading

stites commented Jul 30, 2019

stites commented Aug 8, 2019 • edited Loading

stites commented Aug 14, 2019 • edited Loading

stites commented Aug 14, 2019

stites commented Aug 20, 2019 • edited Loading

stites commented Aug 29, 2019 • edited Loading

stites commented Oct 17, 2019

teh commented Oct 20, 2019

jonringer commented Oct 20, 2019

jonringer left a comment

Choose a reason for hiding this comment

jonringer commented Oct 20, 2019

stites commented Oct 20, 2019 • edited Loading

jonringer commented Oct 20, 2019

jonringer commented Oct 20, 2019

jonringer commented Oct 20, 2019

jonringer left a comment

Choose a reason for hiding this comment

jonringer left a comment

Choose a reason for hiding this comment

stites commented Oct 23, 2019

jonringer commented Oct 23, 2019

jonringer commented Oct 23, 2019

stites commented Oct 24, 2019 • edited Loading

jonringer commented Oct 24, 2019

stites commented Oct 24, 2019 • edited Loading

danieldk commented Dec 6, 2019

stites commented Jul 18, 2019 •

edited

Loading

stites Jul 24, 2019 •

edited

Loading

stites Jul 24, 2019 •

edited

Loading

stites commented Jul 25, 2019 •

edited

Loading

stites commented Jul 26, 2019 •

edited

Loading

stites commented Aug 8, 2019 •

edited

Loading

stites commented Aug 14, 2019 •

edited

Loading

stites commented Aug 20, 2019 •

edited

Loading

stites commented Aug 29, 2019 •

edited

Loading

stites commented Oct 20, 2019 •

edited

Loading

stites commented Oct 24, 2019 •

edited

Loading

stites commented Oct 24, 2019 •

edited

Loading