Fallback to archive.org URLs for failed downloads of FOSS packages #2284
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HebaruSan
added
Pull request
Infrastructure
techman83
approved these changes
Feb 14, 2018
| /// Return an archive.org URL for this download, or null if it's not there. | ||
| /// The filenames look a lot like the filenames in Net.Cache, but don't be fooled! | ||
| /// Here it's the first 8 characters of the SHA1 of the DOWNLOADED FILE, not the URL! | ||
| /// </summary> |
There was a problem hiding this comment.
Yeah, the hash of that file will remain consistent. The url may not.
Produces a filename based of the first 8 digits in sha1 hash,
the 'identifier' and the 'version' in the metadata if the
download_hash exists. Returns '0' if there is no download hash
or has an content type other than zip/gz/tar/tar.gz.
There are some tests that ensure the correct filenames are generated.
There was a problem hiding this comment.
Yup, either way makes enough sense to me. I just wanted to note it explicitly since we have two different 8-digit hexadecimal filename prefixes floating around, and it's not easy to tell that they're (supposed to be) different at a glance.
There was a problem hiding this comment.
Yeah, actually makes sense for URL in the cache. If the url changes, you probably do want to re-download.
| : null; | ||
| } | ||
| } | ||
|
|
||
| /// <summary> |
There was a problem hiding this comment.
Some tests of the filename generation could be useful, but I wouldn't treat this as a blocker.
politas
added a commit
that referenced
this pull request
Feb 16, 2018
This was referenced Feb 20, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
SpaceDock broke today, which is fun.
VITAS reports that it looks like a denial of service attack.The NetKAN bot has been uploading all permissively-licensed mods to https://archive.org/details/kspckanmods for quite a while now. Many or most of the SpaceDock downloads that are now failing, could in principle fall back to archive.org URLs. Such a feature could also help us deal with GitHub's download throttling or SpaceDock's certificate expirations.
Changes
Now if the primary download fails for a permissively licensed mod, we try to find it on archive.org as a fallback.
Internally this involves defining a
fallbackUrlproperty inDownloadTargetandNetAsyncDownloaderDownloadPart, both populated from a newCkanModule.InternetArchiveDownloadproperty, which itself checks a newLicense.Redistributableproperty based on a new copy of the list from NetKAN-bot. Then if a download fails, we check whether it has a fallback URL, and if so and if we haven't already tried it, we try it. If the fallback fails as well, then we continue with the normal steps for a failed download.Fixes #1682.