gatsby-source-contentful - mirroring images locally?

[tylermenezes]

I've looked through the documentation and some of the code for a while, but I can't figure this out: is there a way to download images from Contentful during the build process?

It looks like there was a cache-image.js file added as part of #5659 but as far as I can tell it only uses that cache to generate SVGs.

For context, because ctfassets.net resizes/crops/compresses images on cache-miss, requests are sometimes two orders of magnitude slower than images we host ourselves.

[KyleAMathews]

Not at the moment. That'd definitely be an interesting option to add. And yeah, we do download small images for generating svgs or generating 20x20 base64 images for the blur-up effect for gatsby-image.

[tylermenezes]

@KyleAMathews If I add an option to the plugin, would you consider merging it?

[KyleAMathews]

Yeah definitely. /cc @Khaledgarbaya

[macandcheese]

+1 ... It would be great to mirror image assets from Contentful in a local folder during build process, and use relative paths to those vs. the Contentful CDN paths. Additionally having these images locally would mean being able to commit them to a Git repo vs. relying upon the Contentful CDN.

[Khaledgarbaya]

For context, because ctfassets.net resizes/crops/compresses images on cache-miss, requests are sometimes two orders of magnitude slower than images we host ourselves.

Can you share some insights on this?

If I add an option to the plugin, would you consider merging it?

I think this can be a good idea to support offline builds but it should be configurable, but at the same time, it may introduce more complexity to the plugin like how to make sure that you are always getting the right images.

[gatsbot]

Old issues will be closed after 30 days of inactivity. This issue has been quiet for 20 days and is being marked as stale. Reply here or add the label "not stale" to keep this issue open!

[kennedyrose]

+1 We've built offline kiosks that are identical to our websites in the past. Which wouldn't be possible if using assets from Contentful.

[gatsbot]

This issue is being closed due to inactivity. Is this a mistake? Please re-open this issue or create a new issue.

[polarathene]

@Khaledgarbaya I have made a PR for this. I think I had the same concern about getting the right images(as in if the asset on Contentful has been updated, how to compare with the locally stored/cached version).

I should probably also add some logic to delete locally cached assets if their node does not exist anymore when running gatsby-source-contentful again, but I'm not sure how to go about that.

My implementation is rather simple. It's not a feature that lets the developer switch URLs from the Contentful CDN to locally stored assets seamlessly though, again, no idea how I'd go about that(maybe modifying the nodes file.url to point to the local asset instead, but I think it'd also get complicated due to the Contentful fragments for gatsby-image). Mine just iterates through the ContentfulAsset type nodes and downloads them all to cache, a separate field is added to the node localFile which can be used instead.

Can you share some insights on this?

I don't know how Contentful works under the hood, I imagine that such requests are cached on their end for some period of time along with sending cache headers to the users browser to cache on the user end, when users request from Contentful CDN, if the processed query for the image is not available, there is some additional latency to produce the processed image again?

Additionally there might be other optimizations from hosting the assets at the same server, http/2 maybe?(that feature that can improve transfer of multiple file requests)

---

Sidenote: Why did @gatsbybot close the issue after someone commented within the remaining 10 day window?

[jonniebigodes]

@polarathene concerning the side note, probably gatsbybot sneezed and accidentally clicked the issue close button (pardon the bad pun). I'm reopening as i'm typing this. Hope to see your pull request merged and see the issue fixed. And then proceed to close it. Sounds good?

[polarathene]

@jonniebigodes Yup. I'm not too familiar with the PR process here. Do I need to ping someone to review it?(I kind of tried), or just leave it and hope someone finds the time to do so. Understandably it's not the best time of the year to expect a quick review and merge.

[jonniebigodes]

@polarathene for instance either pieh or DSchau recieved the notification and are reviewing the changes you proposed, it takes a little time because some pull requests need a little more "local" testing, as oposed to others that only need the unit testing implemented by travis/appveyor and other tools all set up. Also the time of the year is a factor in all of this, because one way or another people will be probably making some plans for new years eve. But don't worry. I ask that you are a little more patient and soon and if all goes well you'll get a notification that your pull request with the changes was accepted and this issue is resolved and closed. Sounds good?

[polarathene]

@jonniebigodes Yeah no problem I fully understand that :) How do you know they are reviewing the changes? Is that shown somewhere?

Have a great new years!

[jonniebigodes]

@polarathene not shown actually, but they are notified of the pull requests made. And i want to believe that with your recent activity you left an impression and with that they might be taking a closer look at your contributions! And thank you very much. Have a great new year aswell.

[Khaledgarbaya]

Hey @polarathene,

I don't know how Contentful works under the hood, I imagine that such requests are cached on their end for some period of time along with sending cache headers to the users browser to cache on the user end, when users request from Contentful CDN, if the processed query for the image is not available, there is some additional latency to produce the processed image again?

The asset gets processed at the first request and than will be cached and won't get pocessed again if you keep using the same url.
Contentful assets are more powerful to be used just as plain files.
I still understand the use case for people to keep the images locally but I am seeing a lot of problem syncing state between local and remote assets. Also, when downloading the images this will increase signafically the build time because usually users they upload the highest res they to Contentful and let the platform handle the processing.

[polarathene]

@Khaledgarbaya ah I see. I'm not too fussed, I just wanted to have the assets local to the server hosting everything out of curiousity. For deployment all I need is to clear the cache and that should avoid any sync issues pulling everything in(which for the small projects I've been doing lately aren't that much). On larger projects I can definitely see less appeal to deploy without Contentful CDN.

When updates are done with say Contentful webhooks, the builds can also avoid the cache issues if the cache is cleared I think? The offline support is good for development where you don't plan to have new/changed content often for a while perhaps and when you want to deploy without the Contentful CDN.

It'd be better perhaps if the offline support could be a bit more seamless with the understanding that the cache may not be reliable for updates.

[gatsbot]

This issue is being closed due to inactivity. Is this a mistake? Please re-open this issue or create a new issue.

[ghost]

@Khaledgarbaya any updates on this?

[ghost]

@polarathene did you have any success downloading images locally when inserted in the post?

[polarathene]

"Inserted in the post"? Is that Rich Text or whatever it was called? That was unrelated to this, there was a separate plugin for that.

I haven't used Contentful since early 2019, I don't know what the status is with it and Gatsby currently.