Accept header and infrastructure for auth tokens

[HebaruSan]

Problems

CKAN's GitHub downloads fail with an HTTP 403 status fairly frequently. We do not know the exact details that trigger it, but it mostly affects large downloads.

It also turns out that the URLs we're using are supposed to be for users and browsers only, not applications. We should migrate to using the GitHub API.

See #2210 for details.

Cause

GitHub's normal download links are throttled to prevent abuse, and CKAN's use of them is sometimes affected by this.

GitHub prefers applications to use the GitHub API for programmatic access, which requires support for two new HTTP headers, one that specifies the MIME type of the download (without this, the API returns a JSON description of the file instead of the actual download):

Accept: application/octet-stream

And one that contains the user's GitHub API authentication token (this one is technically optional, but downloads will still fail frequently without it, because the limit is 60 per hour):

Authorization: token <OAuth token here>

Changes

MIME types

The Accept header is now set for all module downloads. The default is application/octet-stream, but we will use a multi-value format as well when download_content_type is set for a module:

Accept: {CkanModule.download_content_type};q=1.0,application/octet-stream;q=0.9

CkanModule has a new property download_content_type that maps to the same value in CKAN-meta (this was absent from the registry previously but stored to the metadata repo for all modules by netkan). This allows us to get the MIME type of a download.

NetAsyncDownloader now uses Net.DownloadTarget objects instead of KeyValuePair<Uri, int> to describe requested downloads, as does all of its calling code.

Net.DownloadTarget now has a mimeType property storing the MIME type of the download.

The value from CkanModule.download_content_type is passed to Net.DownloadTarget.mimeType when it's available.

The value from Net.DownloadTarget.mimeType is used to set the Accept header when it's available.

Auth tokens

The Windows/Mono registry now has a new AuthTokens key (secret value in screenshot deleted in post-processing):

When a download is initiated, its Uri.Host property is looked up as a key of this key, and if it's found, then the corresponding value is plugged in to the Authentication header.

Still To Be Done In Future Pull Requests

Partial fix for #2210. Will be complete after:

• Cmdline commands for editing the auth token table
• ConsoleUI screen for editing the auth token table
• GUI window for editing the auth token table
• netkan update to use asset.url instead of asset.browser_download_url
• Conversion of existing URLs to use the API

Note that you can set up auth tokens without UI components by using the registry editor in Windows or an equivalent tool for Mono. When this is done, this version of the client will be able to access GitHub API download URLs with that auth token.

Many thanks to @dbent for wise feedback and ideas that guided these changes in the comments of #2210.

[dbent]

Sorry for the confusion, when I said user settings are stored in the registry, I meant the Windows registry, which is emulated by Mono on nix systems. This is where global configuration data, like the location of KSP installs, are stored. That's probably the best place for any user tokens.

Everything else looks good to me.

[HebaruSan]

Ahh, that makes sense. As a bonus, that eliminates the need to pass that Dictionary around. Please see latest commit.

[dbent]

LGTM 👍