If you're having memory leaks/bloating

[trev-dev]

Try running your sharp image processing in a child_process spawn. I've seen lots of posts here that generally conclude it's something node-related with RSS memory. I can't say one way or another if this is the case, and I've generally seen multiple suggestions to fix it in my search for a solution, some seem to work, others don't - this worked for me.

In my case running sharp in a server/express is where I saw the bloat. I'm using multer to upload image files and then attempting to compress/reduce them on the fly. Since multer stores the images and spits out an array of file locations, I just passed the array into my child process as a JSON string param.

// express route
var multer = require('multer');
var upload = multer({dest:'uploads/tmp'});
app.post(['/upload/:store/:name', '/form/upload/:store/:name'], upload.array('images', 6), function(req, res){
    var manifest = {shop: req.params.store, name: req.params.name, images: req.files};

    // Spawn node child process to handle image manipulation

    var spawn = require('child_process').spawn;

    var compressor = spawn('node', ['./modules/compression', JSON.stringify(manifest)]);
     
    // When the child process handles the images successfully it will emit the array of image locations for us to send to the front end.
    compressor.stdout.on('data', (data)=>{

      data = JSON.parse(data.toString());
      res.json(data);

    });
    // If there's an error, we'll see it.
    compressor.stderr.on('data', (err)=>{

      console.log(err.toString());
      res.status(500).send();

    });

 });

...My image processing module spawned in the child process...

var fs = require('fs');
var mkdirp = require('mkdirp');
var image = require('./images');

var manifest = JSON.parse(process.argv[2]);
var promises = [];

// Generate promise array loaded with image handling promises
manifest.images.forEach(function(cur){

    promises.push(image.adjustImage(cur));

});

// Iterate through all the promises for the uploaded images, then execute code when done.
Promise.all(promises).then((paths)=>{
    var dir = `./uploads/${manifest.shop}/${manifest.name}`;
    
    // If destination path does not exist, make it so.
    if (!fs.existsSync(dir)) {

        mkdirp.sync(dir, {mode:0777});

    }
    // Delete old image files in destination folder
    oldfiles = fs.readdirSync(dir);

    oldfiles.forEach(function(cur){

        fs.unlinkSync(`${dir}/${cur}`);
        
    });

    // For each of the newly uploaded image files, move them to the destination folder.
    paths.forEach(function(cur){
        fileName = cur.split('/').pop();
        fs.renameSync(`./${cur}`, `${dir}/${fileName}`);

    });

    // Send array of image files to front end.
    var newPaths = fs.readdir(dir, function(err, files){

        files = files.map(function(cur){

            return `${dir}/${cur}`.slice(1);

        });
        process.stdout.write(JSON.stringify(files));
        process.exit();

    });

}).catch(function(err){
    
    console.log(err);
    process.exit(1);
    

});

...The image.js module where the sharp code and promise chain is constructed. Pardon the many files, this all needs refactoring.

var sharp = require('sharp');
var imageSize = require('image-size');
var fs = require('fs'); 

function renameImage(img) {

  function pruneExtension (name) {

    var ext = name.split('.');
    ext = ext[ext.length - 1];
    return `.${ext}`;

  }
  img.extension = pruneExtension(img.originalname);
  img.oldPath = img.path;
  img.newPath = `${img.oldPath}${img.extension}`;
  fs.renameSync(img.oldPath, img.newPath);

}

module.exports = {

  adjustImage: function(img){
    renameImage(img);
    
    return new Promise(function(resolve, reject){

      var size = imageSize(img.newPath);
      if (size.width < 1000) {
    
        resolve(img.newPath);
    
      } else {
        
        sharp(img.newPath).resize(1000, null).jpeg({
          quality: 40
        }).toFile(`${img.oldPath}-sharp.jpg`, (err, output) => {

          if (err) {

            reject(err);

          } else {

            fs.unlinkSync(`${img.oldPath}${img.extension}`);
            img.newPath = `${img.oldPath}-sharp.jpg`;
            img.extension ='.jpg';
            resolve(img.newPath);

          }

        });

      }


    });
  }

};

Hope this concept helps people who are stuck on memory problems. Sharp is amazing, lets keep using it!

[webcarrot]

Other way is use jemalloc by setting LD_PRELOAD environment variable.

Debian example:
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 node ...

After this change, total memory usage has dropped from "out off memory" to less than 200M per server instance.

[lovell]

Please also see #955 for more evidence that the choice of memory allocator is important.

[shwao]

If someone has problems with using libjemalloc1 on debian 10 (buster), its now version 2. Eg. /usr/lib/x86_64-linux-gnu/libjemalloc.so.2

[TheAndroidGuy]

I can confirm that using jemalloc fixes the memory issue. I set jemmaloc as system wide memory allocator, and the memory usage has dropped immensely and my app is no longer crashing because of memory not releasing.

Thank you for all the suggested ideas and solutions.

If anyone is wondering, this stackoverflow thread helped me with the installation and usage of jemalloc.

[mohammed-almujil]

If someone has problems with using libjemalloc1 on debian 10 (buster), its now version 2. Eg. /usr/lib/x86_64-linux-gnu/libjemalloc.so.2

@shwao Thank you, updating the environment variable fixed it for me.

Context: I updated to version 2 since version one is not found, but didn't update the env variable, updating it fixed it for me. For anyone using docker here is the memory allocator change code:

RUN apt-get update && apt-get install --force-yes -yy \
  libjemalloc2 \
  && rm -rf /var/lib/apt/lists/*

ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2

[oliver-daniel]

Necro'ing this thread to say that the jemalloc2 fix is still applicable in February of 2023, in the context of an unexplained memory leak of a NextJS app on an Ubuntu production server. Subject to unexpected memory spikes over the next few days, our memory usage has gone from ~500 MB -- which would climb rapidly with reloading of images and never come back down -- to a steady baseline of ~300 MB with only minor blips on repeated reloads.

This is a particularly wicked leak to track for the inexperienced, as the production-only nature of Next's use of sharp means that debugging the local Node server will (definitionally) not catch it. Consider this comment something like bug-hunting SEO!

[lovell]

On an SEO note, for those visiting this page who haven't yet found the docs, please see https://sharp.pixelplumbing.com/install#linux-memory-allocator

When using Linux, sharp v0.28.0+ will attempt to detect the memory allocator and adjust its concurrency accordingly to help reduce fragmentation, so please ensure you're using a recent version.