External Model Definition Functionality Potentially Broken in PyTorch 1.9

[jamesmcclain]

🐛 Bug

It seems to currently not be possible to load external model definitions from inside of training pipelines. When running in local and inprocess mode locally, I get a HTTP Error 403: rate limit exceeded error when trying to use external model definition functionality (in particular a resnet18-fpn from 'AdeelH/pytorch-fpn:0.1') in a training pipeline.

This is apparently an known issue, and can be read-about here:
pytorch/vision#4156 (comment)

To Reproduce

Steps to reproduce the behavior:

1. Attempt to use an external model definition in a training pipeline. I can confirm this locally, I do not know if it exists on batch.

(I can provide the exact context and command in a different context.)

Expected behavior

I was not expecting this to happen.

Environment

Running in a docker container with the most recent

• How you installed and are running Raster Vision (pip install on local vs. inside Docker image): docker
• Raster Vision version or commit: current master of 09/10/2021
• OS (e.g., Linux): Linux
• Python version: From docker image
• CUDA/cuDNN version if running on GPU: 11.2
• Any other relevant information:

Additional context

[AdeelH]

This was also reported in #1221.

Looks like a problem with PyTorch 1.9 that won't be fixed until the next release. Maybe we should revert the dependency to 1.8?

As a workaround, you can throw a zip of the repo on S3 and link to that in the external module config.

[jamesmcclain]

This was also reported in #1221.

Looks like a problem with PyTorch 1.9 that won't be fixed until the next release. Maybe we should revert the dependency to 1.8?

As a workaround, you can throw a zip of the repo on S3 and link to that in the external module config.

Ah, I see, I missed that earlier issue. I was able to work around the issue by patching PyTorch within the image: pytorch/vision#4156 (comment) .

[AdeelH]

This is a workaround that doesn't require changing PyTorch source code: pytorch/pytorch#61755 (comment) .

Might be worth adding to RV until next PyTorch release.