Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add optional custom headers field #149

Merged
merged 1 commit into from
Apr 6, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -30,6 +30,7 @@ In the `URL(s)` field, give a list of page URLs to be imported (e.g. {https://ww
- if the remote page is an SPA (React, Angular) or require Javascript to load some pieces of the content, Javascript is then required. Enabling Javascript may help here.
- more generally, disabling Javascript speeds up the import process and reduces the memory consumed.
- `Scroll to bottom`: forces a scroll to the bottom of the page. This might allow images set with earger to be loaded or any element loaded with Javascript below the fold. Increasing the `Page load timeout` might give more time to those element to be loaded.
- `Custom headers`: connection to the site you want to import content from might require some custom request headers, like a Bear, an API key (especially when hitting JSON API), a Coookie... Those headers are sent together with the fetch request (headers config property of the standard browser `fetch` API).

## Crawler

3 changes: 3 additions & 0 deletions import-bulk.html
Original file line number Diff line number Diff line change
@@ -61,6 +61,9 @@ <h2>Import - Bulk</h2>
<sp-checkbox class="option-field" id="import-scroll-to-bottom" checked>
Scroll to bottom
</sp-checkbox>

<sp-field-label for="import-custom-headers">Custom headers</sp-field-label>
<sp-textfield class="option-field" id="import-custom-headers" multiline placeholder="Define your custom headers as a JSON object with key/value (header name/header value)"></sp-textfield>
</div>
</sp-accordion-item>
</sp-accordion>
3 changes: 3 additions & 0 deletions import.html
Original file line number Diff line number Diff line change
@@ -64,6 +64,9 @@ <h2>Import - Workbench</h2>
<sp-checkbox class="option-field" id="import-scroll-to-bottom" checked>
Scroll to bottom
</sp-checkbox>

<sp-field-label for="import-custom-headers">Custom headers</sp-field-label>
<sp-textfield class="option-field" id="import-custom-headers" multiline placeholder="Define your custom headers as a JSON object with key/value (header name/header value)"></sp-textfield>
</div>
</sp-accordion-item>
</sp-accordion>
2 changes: 1 addition & 1 deletion js/dist/helix-importer.js

Large diffs are not rendered by default.

10 changes: 7 additions & 3 deletions js/import/import.ui.js
Original file line number Diff line number Diff line change
@@ -419,10 +419,13 @@ const attachListeners = () => {

let res;
try {
res = await fetch(src);
const headers = JSON.parse(config.fields['import-custom-headers'] || '{}');
res = await fetch(src, {
headers,
});
} catch (e) {
// eslint-disable-next-line no-console
console.error(`Unexpected error when trying to fetch ${src} - CORS issue ?`, e);
console.error(`Unexpected error when trying to fetch ${src} - CORS issue or invalid headers ?`, e);
}
if (res && res.ok) {
if (res.redirected) {
@@ -494,7 +497,8 @@ const attachListeners = () => {

frame.dataset.originalURL = url;
frame.dataset.replacedURL = src;
frame.src = src;

frame.src = URL.createObjectURL(await res.blob());

const current = getContentFrame();
current.removeEventListener('load', onLoad);