Errors when processing large(ish) numbers of images with gatsby-plugin-sharp

[jp887]

Description

I'm processing around 1500 images with gatsby-plugin-sharp/gatsby-transformer-sharp and GraphQL, to display with gatsby-image as per this page in the docs.

For smaller numbers of images I have had no issues, but now that I'm processing 1500 images the build randomly fails with the following error:

(sharp:104): GLib-GObject-WARNING **: 09:35:11.293: gtype.c:4265: type id '0' is invalid

(sharp:104): GLib-GObject-WARNING **: 09:35:11.293: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:104): GLib-GObject-WARNING **: 09:35:11.293: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation

(sharp:104): GLib-GObject-CRITICAL **: 09:35:11.293: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed
Received 'segmentation fault' signal

This happens during the "Generating image thumbnails" step in the build process and it occurs apparently randomly (sometimes 10% of the way through the images, sometimes at 80%, sometimes not at all). Therefore I do not believe it is caused by a "faulty image".

[m-allanson]

Hmm that's not a lot of info to go on.. can you post an example repo that creates the error you're seeing?

[rns70]

Had the same issue. Doing a manual npm install git+https://github.com/lovell/sharp.git to install the latest unreleased Sharp package seems to have fixed it for me. This might not be a good idea, however I'm not sure what else to do until a fixed version of Sharp is released.

[HriBB]

I can confirm this bug. In my case it started happening, when I added some large images, 5mb or more.

[btk]

I had the same problem;

Using an older version for gatsby-transformer-sharp (2.1.1) and gatsby-plugin-sharp (2.0.5) fixed it for me. Maybe if there is any debuggers, this might be an additional information.

Note that I also have a lot of images on my build.

[jp887]

@m-allanson I've started working on a repro here: https://github.com/jp887/gatsby-issue6291-repro
Basically generate a large number of random images with a script and then query for them in a basic gatsby 2 site.

I'm building the project on CircleCI so build outputs can be inspected without having to run the project locally.

The most recent build reproduced the issue: https://circleci.com/gh/jp887/gatsby-issue6291-repro/3

[btk]

@jp887 would you mind trying to take a build on your repro with these gatsby-transformer-sharp (2.1.1) gatsby-plugin-sharp (2.0.5) exact versions, on a new branch maybe? I have downgraded to these versions, and I have taken more than 10 successful builds since.

[jp887]

@btk OK done. Builds on the 'downgraded' branch can be found here: https://circleci.com/gh/jp887/workflows/gatsby-issue6291-repro/tree/downgraded

[jp887]

Seems it's still reproducing the issue on the new downgraded branch: https://circleci.com/gh/jp887/gatsby-issue6291-repro/4

[niklasravnsborg]

I also have the problem in a recent project.

I also get a lot of vips warning: exif: unknown EXIF resolution unit errors.

[TylerBarnes]

I'm running into this as well but with a lot less images

For me it happens when I'm changing maxWidth and quality on a fluid graphql query

                        fluid(maxWidth: 400, quality: 100) {
                          ...GatsbyImageSharpFluid_withWebp_tracedSVG
                        }

[jonniebigodes]

@TylerBarnes ohhh the dreaded segmentation fault error, it brings me back to when i was using c language as a development tool. That looks like that somewhere deep down in the execution of that plugin, it's trying to allocate memory where's not allowed and it was not deallocated and trying to retrieve something that is not allowed. That is not a gatsby plugin issue per se, more like the underlying vips/sharp, the one written in c.

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[kamilgrzegorczyk]

I am having the same issue at the same build step.

My plugin versions are:

    "bourbon-neat": "3.0.1",
    "dotenv": "^6.2.0",
    "gatsby": "^2.0.112",
    "gatsby-image": "^2.0.29",
    "gatsby-plugin-manifest": "^2.0.17",
    "gatsby-plugin-offline": "^2.0.22",
    "gatsby-plugin-react-helmet": "^3.0.6",
    "gatsby-plugin-sass": "^2.0.10",
    "gatsby-plugin-sharp": "^2.0.20",
    "gatsby-source-wordpress": "^3.0.32",
    "gatsby-transformer-sharp": "^2.1.13",
    "node-sass": "^4.11.0",
    "normalize-scss": "^7.0.1",
    "react": "^16.7.0",
    "react-dom": "^16.7.0",
    "react-helmet": "^5.2.0"

[briandamaged]

I seem to have encountered this issue as well. I'll add more to this thread if I discover any additional clues.

[briandamaged]

Also... I haven't been able to figure out how to apply the "not stale" label to this issue. I'm guessing that I need to be granted a permission?

In either case: the issue does appears to be somewhat random. I just reran the same build without encountering the problem this time. (In both cases, I cleared all cached files before starting the build)

[btk]

@briandamaged added it for you.

[briandamaged]

@btk : Thx! So far, the issue has not reoccurred.

[mdornseif]

I still see this when build by nelify:

10:08:33 AM: success extract queries from components — 0.324 s
10:08:51 AM: (sharp:1324): GLib-GObject-WARNING **: 09:08:51.843: gtype.c:4265: type id '0' is invalid
10:08:51 AM: (sharp:1324): GLib-GObject-WARNING **: 09:08:51.843: can't peek value table for type '<invalid>' which is not currently referenced
10:08:51 AM: (sharp:1324): GLib-GObject-WARNING **: 09:08:51.843: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation
10:08:51 AM: (sharp:1324): GLib-GObject-CRITICAL **: 09:08:51.843: g_value_transform: assertion 'G_IS_VALUE (dest_value)' failed
10:08:54 AM: /usr/local/bin/build: line 32:  1324 Segmentation fault      (core dumped) gatsby build
10:08:54 AM: Caching artifacts
…
10:08:55 AM: failed during stage 'building site': Build script returned non-zero exit code: 139

It randomly happens using exactly the same git revision:

See http://filez.foxel.org/4dec3e172673 for version Information.

[mdornseif]

This seems to be unrelated and dos not lead to segfaults / failed builds (same repository):

12:04:08 AM: success Building production JavaScript and CSS bundles — 41.675 s
12:04:10 AM: success Building static HTML for pages — 2.570 s — 109/109 87.88 pages/second
12:04:16 AM: info Done building in 661.124 sec
12:04:16 AM: (sharp:1506): GLib-CRITICAL **: 23:04:16.327: g_hash_table_lookup: assertion 'hash_table != NULL' failed
12:04:16 AM: (sharp:1506): GLib-CRITICAL **: 23:04:16.750: g_hash_table_lookup: assertion 'hash_table != NULL' failed
12:04:17 AM: (sharp:1506): GLib-CRITICAL **: 23:04:17.106: g_hash_table_lookup: assertion 'hash_table != NULL' failed
12:04:17 AM: Caching artifacts
12:04:17 AM: Started saving node modules
12:04:17 AM: Finished saving node modules

[mdornseif]

Regarding the segfaults I have found this in the gatsby-plugin-sharp:

// Try to enable the use of SIMD instructions. Seems to provide a smallish
// speedup on resizing heavy loads (~10%). Sharp disables this feature by
// default as there's been problems with segfaulting in the past but we'll be
// adventurous and see what happens with it on.
sharp.simd(true);

So far locally I haven't seen any segfaults after commenting out sharp.simd(true);.

[TylerBarnes]

That would make sense. I've only seen this on sites with a lot of images and then it's still rare on those sites. Maybe once every 20 - 30 builds.

[niklasravnsborg]

@mdornseif @TylerBarnes Thanks for the discovery! I went ahead and opened a PR, removing the lines: #11925 😌

[tinder-kyleboss]

Thank you so much for the awesome finds @mdornseif & the PR @niklasravnsborg. 😄

I am also running into these gatsby-plugin-sharp-inspired segmentation faults. Unfortunately, deactivating the SIMD functionality did not seem to improve the frequency of these segmentation faults. 😢

[KyleAMathews]

Something else we could do to improve stability is to move Sharp to a child process as if it crashes, we could just restart it.

[tinder-kyleboss]

That would be a cool approach, I like it! 👍

Despite this probably slowing down a build or two, I think that this would definitely satisfy a majority of the folks who run into this issue. 😄

[pvdz]

Fwiw, I can repro the warning outputs in the new version but not the segfault. So perhaps that's an improvement. and also the core dump. Unfortunately this release is tagged node 10+ and we'll need about a month of transition time for people to migrate to node 10. Perhaps a special release can be made for us which says node 8+ but dunno. So it sounds like you'll have to wait a bit longer :/

[ehannes]

Yes, I saw your comments in the above mentioned sharp issue. Good work, sad it didn't pay off.

I don't quite understand your comment about the node version. Is that version related to this issue somehow? I'm running node v10.16.1...

[pvdz]

@ehannes no, I was able to repro in v10 as well. The comment is about the nodejs EOL cycle. In 2020 nodejs v8 reached end of life ("EOL"), meaning it won't get any more patches and fixes and people should move to node 10, which is the next long term support ("LTS") release version. So this month many packages are bumping their minimum required nodejs version to >=10, sharp being no exception. Gatsby will do so as well but we're taking a little longer to allow people to make the transition outside of the busy December month that it would otherwise have been. Hence, currently we couldn't bump the sharp version because then people on node8 (which we claim is the minimum version we support) would get node version errors when installing. (This can be worked around, I don't think we need to).

That said, the new sharp version should be more stable as a few bugs related to this issue were found and fixed. Just, apparently, not all of them.

[wardpeet]

I'm wondering if people still see this issue on gatsby-plugin-sharp@2.4.3. We have reduced memory usage of the sharp plugin itself.

[pvdz]

Keep in mind that the segfault was a real race condition problem that was only exposed by high pressure. It was not about memory.
It is possible symptoms get reduced as we rely less on sharp for queuing. But they are also prepping a next release fixing some more bugs.
I guess, basically as long as I can repro it and they can confirm it, there's at least a problem left.

[wokejacqueline]

Still experiencing this issue as of "gatsby-plugin-sharp": "2.4.4" (just upgraded from 2.3.3) & "gatsby-transformer-sharp": "2.3.13"

300-ish images processed into 1800 thumbnails of different sizes will usually fail once or twice (sometimes more) before succeeding.

It always succeeds eventually. So I just
"npm run build || npm run build || npm run build --verbose" for now lol. Thanks for looking into this, I have been experiencing this issue on my project since at least Nov 2019. :(

[ehannes]

@pvdz any updates on this issue? Still have massive problems with this...

[pvdz]

@ehannes A new version of Sharp was released (with new binary) that is supposed to fix some-if-not-all of these problems. Unfortunately it is released with a minimal nodejs version of 10 (due to EOL of node 8) and Gatsby can not bump the minimal to node 10 right now (but soon...). Meaning we can't bump Sharp at the moment to make use of these improvements.

Of course, you can try to install it locally. I haven't had a chance yet to verify the latest release, which is on my todo-list for this week.

[ehannes]

@pvdz I thought this bug was still present in the new Sharp version. You wrote this a month ago:

I was able to repro in v10 as well

[pvdz]

@ehannes yeah, and that was reported upstream, and I think there was a new release about a week ago (or two weeks?). I haven't had time to investigate. We (Gatsby) had a meetup in the US last week. No worky worky in the US for us EU people. Just haven't gotten around to it beyond that, will try to report back this week.

As for the node10 concern; the problem I mentioned is not so much the repro but rather that if we bump to the new Sharp version (0.24+) in Gatsby then about 5% of our users will have installation problems because they are still on Node 8 and npm/yarn will block installation due to Sharp requiring 10+.

Fwiw, in the end the problem (of this issue) a race condition in the binary used by the Sharp library. It is made more likely by high pressure in the rest of the system but it can happen at any time for anyone. If you want to help out feel free to look at lovell/sharp#1986 and help out. Gatsby should be able to cut over to Node 10 "soon", but I have no exact timeframe on what "soon" is.

[pvdz]

It looks like the next release of Sharp (0.25.0, not yet published) will fix these issues.
It will require node 10 so Gatsby won't ship with it until Gatsby can bump the node minver to 10 LTS, but that shouldn't stop you from being able to use it.

So it looks like that will mitigate this problem. I was able to generate 30k thumbs without signs of the issue (where before the problem would generally repro between 3k and 6k images processed).

Will keep this open until Gatsby can ship the new, fixed, version. But please go ahead and test the "sharp": "lovell/sharp#yield" version of Sharp to confirm this fixes the problem for you.

[wokejacqueline]

Can confirm after forcing my project to use Sharp on the yield branch, I am no longer experiencing segfaults with image processing. Previously, I would experience them almost every build, multiple times.

THANK YOU SO MUCH! 🥳

[ehannes]

I can also confirm that the latest version of Sharp is finally solving my build problems! By the way, 0.25.0 is now published, as well as 0.25.1.

So grateful for this fix! Thanks for your hard work @pvdz!

[TylerBarnes]

Same here! So excited to see this fixed. Amazing job @pvdz

[pvdz]

Sharp has been bumped by #22432 and that means this issue should now be closed. Please report back if you have Sharp related segfaults or see warning messages that have been reported in this thread.

This change has been published in the following packages:

• gatsby-plugin-manifest@2.3.1
• gatsby-plugin-sharp@2.5.1
• gatsby-remark-images-contentful@2.2.1
• gatsby-source-contentful@2.2.2
• gatsby-theme-blog-core@1.3.2
• gatsby-theme-blog@1.4.2
• gatsby-theme-notes@1.2.2
• gatsby-transformer-sharp@2.4.1
• gatsby-transformer-sqip@2.2.1
• gatsby@2.20.2

For any other Sharp related problems or questions, please open a new issue. Thanks!

[esetnik]

I am still having occasional segfaults using gatsby-transformer-sharp@2.4.1

$ gatsby build
success open and validate gatsby-configs - 0.039s
/bin/sh: lscpu: not found
/bin/sh: lscpu: not found
success load plugins - 0.505s
success onPreInit - 0.007s
success delete html and css files from previous builds - 0.009s
success initialize cache - 0.006s
success copy gatsby files - 0.059s
success onPreBootstrap - 0.007s
/bin/sh: lscpu: not found
success createSchemaCustomization - 0.005s
success source and transform nodes - 0.103s
success building schema - 0.271s
success createPages - 0.000s
success createPagesStatefully - 0.091s
success updating schema - 0.028s
success onPreExtractQueries - 0.001s
success extract queries from components - 0.388s
success write out redirect data - 0.001s
success Build manifest and related icons - 0.320s
success onPostBootstrap - 0.322s
info bootstrap finished - 4.358s
success run static queries - 2.702s - 6/6 2.22/s
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with signal "SIGSEGV".
error Command failed.
Exit code: 1
Command: /usr/local/bin/node
Arguments: /opt/yarn-v1.22.4/lib/cli.js build
Directory: /builds/xxx
Output:
info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.

[lucashfreitas]

The issue is still happening with the latest version of gatsby sharp: "2.10.0".

My website has a lot of pages/images and it's seems impossible to get a build done using netlify. If I trigger the build using containers inside AWS or my local computer it's working fine, but not inside Netlify.

Anyone facing the same problem when trying to deploy/build large websites/ lots of images on netlify?

[lucashfreitas]

Quick update:

I've tried to deploy my websites using Gatsby Cloud, Vercel, and Netlify and all of them complain about the same error. ('segmentation fault' signal). Apparently it's related to sharp plugin memory consumption.

I have updated all plugins and gatsby to the latest version and it's really hard to get a build running at the moment. The only place that the build would run fine it's on my laptop which has 32GB of RAM.

I was wondering if gatsby it's not suitable for huge projects with a lot of pages and images?

I have already contacted the Gatsby Cloud support and will post updates here!

[twhitson]

Just wanted to add that I'm seeing the same thing on GitHub Actions as well. @lucashfreitas did you happen to get any answers from Gatsby Cloud support?

[lucashfreitas]

Hey @twhitson ,

I reached out to gatsby cloud support and they have found the problem. Some of the plugins that I had on my project were using a outdated version of sharp library, so the quickest solution is to change the following:

upgrade gatsby-plugin-sharp

upgrade gatsby-transformer-sharp

set resolution for sharp to "0.26.3"

To set the resolutions, you can add this entry to your package.json:

  "resolutions": {
    "sharp": "^0.26.3"
  }

Hope it helps!

[twhitson]

Thank you so much @lucashfreitas that was it!

I had to set my resolution directly to 0.27.1 but it worked like a charm. Thanks for your help!

[freshcutdevelopment]

Thanks @lucashfreitas I was seeing exactly the same issue on Azure Static Web apps. Adding the following lines to my package.json file resolved the issue.

  "resolutions": {
    "sharp": "^0.27.1"
  },