Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support modifying the resource download request #44

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

support modifying the resource download request #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions ckanext/xloader/interfaces.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,19 @@ class IXloader(Interface):
The before_submit function, when implemented
"""

def modify_download_request(self, url, resource, api_key, headers):
""" Can be used to modify the http download request.
The headers parameter is a dict which should be modified directly.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should say "can be modified" rather than "should be modified".


Return value should be the modified (or unmodified) url.

:param url: The download url
:param resource: The dict representation of the resource to be downloaded
:param api_key: The CKAN api key, in case authorization header needs to be modified
:param headers: The http request headers dict, to be modified directly
"""
return url

def can_upload(self, resource_id):
""" This call when implemented can be used to stop the processing of
the xloader submit function. This method will not be called if
Expand Down
9 changes: 7 additions & 2 deletions ckanext/xloader/jobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import sqlalchemy as sa

from ckan.plugins.toolkit import get_action
import ckan.plugins as p
try:
from ckan.plugins.toolkit import config
except ImportError:
Expand All @@ -22,6 +23,7 @@

import loader
import db
import interfaces as xloader_interfaces
from job_exceptions import JobError, HTTPError

if config.get('ckanext.xloader.ssl_verify') in ['False', 'FALSE', '0', False, 0]:
Expand All @@ -35,7 +37,6 @@
CHUNK_SIZE = 16 * 1024 # 16kb
DOWNLOAD_TIMEOUT = 30


# 'api_key': user['apikey'],
# 'job_type': 'push_to_datastore',
# 'result_url': callback_url,
Expand Down Expand Up @@ -167,8 +168,12 @@ def xloader_data_into_datastore_(input, job_dict):
headers['Authorization'] = api_key

def get_url():
_url = url
for plugin in p.PluginImplementations(xloader_interfaces.IXloader):
if hasattr(plugin, 'modify_download_request'):
_url = plugin.modify_download_request(_url, resource, api_key, headers)
return requests.get(
url,
_url,
headers=headers,
timeout=DOWNLOAD_TIMEOUT,
verify=SSL_VERIFY,
Expand Down