Supporting multiple entries/inputs in the Javascript API

[chopfitzroy]

Documentation Is:

• Missing
• Needed
• Confusing
• Not Sure?

Please Explain in Detail...

The documentation surrounding supporting multiple entries in the Javascript API is confusing.

The docs cover multiple entries/inputs briefly and suggest using a rollup.config.js that exports an array as opposed to an object, i.e:

export default [{
  input: 'main-a.js',
  output: {
    file: 'dist/bundle-a.js',
    format: 'cjs'
  }
}, {
  input: 'main-b.js',
  output: [
    {
      file: 'dist/bundle-b1.js',
      format: 'cjs'
    },
    {
      file: 'dist/bundle-b2.js',
      format: 'esm'
    }
  ]
}];

Alternatively see #2935 for a more in depth discussion surrounding this.

What is unclear is whether or not you can use this in the Javascript API, essentially I thought I would be able to call rollup.rollup and pass it an array similar to the one exported in the above example, however this results in the following error:

Unknown input option: 0. Allowed options: acorn, acornInjectPlugins, cache, chunkGroupingSize, context, experimentalCacheExpiry, experimentalOptimizeChunks, experimentalTopLevelAwait, external, inlineDynamicImports, input, manualChunks, moduleContext, onwarn, perf, plugins, preserveModules, preserveSymlinks, shimMissingExports, strictDeprecations, treeshake, watch
Error: You must supply options.input to rollup

I have since realized that rollup.rollup is only capable of accepting one config object and must be called multiple times with each config.

Based on this comment in #863 this is not possible (although from what I can see rollup.watch may accept multiple arguments and as mentioned in the referenced comment this may warrant it's own issue).

As a side note it is also unclear that rollup.watch does accept an array when rollup.rollup does not.

Your Proposal for Changes

If this is the intended behaviour and there is not a planned changed on the immediate horizon I think it would be good to explicitly highlight that the Javascript API does not support multiple entries.

I think it would also be beneficial to perhaps give a rudimentary example of calling rollup.rollup multiple times in the event that a user needs to bundle multiple entries/inputs.

something like:

const { rollup } = require('rollup');

const buildEntries = entries => {
    const pending = entries.map(async item => {
        const [inputOptions, outputOptions] = item;

        const bundle = await rollup(inputOptions);
        await bundle.write(outputOptions);
        return;
    });

    // returns promise that resolves when all rollups complete
    return Promise.all(pending);
}

export default buildEntries;

Where entries would look something like:

[[{
    input: 'component.js',
    plugins: []
},
{
    dir: '/dist',
    name: 'Component',
    format: 'umd'
}]]

[tonypig93]

i've got the same question, so calling rollup.rollup multiple times will be the final solution? How about common chunks?

[lukastaegert]

i've got the same question, so calling rollup.rollup multiple times will be the final solution?

Yes

How about common chunks?

If you have common chunks, instead you should ideally use Rollup's code-splitting feature. I.e. use a single options object with multiple inputs: https://rollupjs.org/guide/en/#input
Rollup will automatically create chunks for shared code. If you need more control over the chunk creation, additionally use manual chunks: https://rollupjs.org/guide/en/#manualchunks

[fregante]

If the API accepted an object like rollup.config.js, this would be possible. Suggested in #761

[lukastaegert]

Yes, but this is a big architectural refactoring and not desired by us at the moment. The assumption is that users of the JavaScript API want more control over the process anyway and thus would be able to just call the API multiple times. This would also give them control over parallel vs. sequential execution.

[lukastaegert]

Also note that for JavaScript API users, the process is split anyway between bundling and file generation.

[tonypig93]

Yes, but this is a big architectural refactoring and not desired by us at the moment. The assumption is that users of the JavaScript API want more control over the process anyway and thus would be able to just call the API multiple times. This would also give them control over parallel vs. sequential execution.

Is there any difference between parallel and sequential execution except for time costs?

[lukastaegert]

Probably not much except if there are some plugins that do a lot of async file operations.

[dwighthouse]

@lukastaegert I'm currently unable to locally build my production version of my code thanks to an EMFILE: too many open files error that I believe is related to the fact that I'm building upwards of at least 80 bundles simultaneously. And it's worse than that, because I'm doing everything twice, once to build a iife version (compiled down to ES5) and once to build a module-compatible version for more recent browsers.

It looks like every single one is opening their own instance of each file that is imported by each base bundle file (and their dependencies), even if they are shared. It would be nice if Rollup could internally open and manage these files a single time rather than thousands of times. I'm not building a single-page app. I'm building a website with hundreds of complex pages, with different privilege levels for different types of functionality.

[lukastaegert]

Yes, that would be nice. Or you just write a really simple plugin that you share between your builds that just implements a load hook that loads files from the disk and caches its results. Could be done in at most 15 lines of code. If you depend on this, then this is your fastest option. And others might profit from your work as well. Ranting here for a big architectural change in Rollup is definitely less promising.

[lukastaegert]

Ok, it is less than 15 lines. Here it is, it would also make your build faster I guess. Mind you, it will not work with watch mode very well 😜

import { promisify } from "util";
import { readFile } from "fs";
const readFileAsync = promisify(readFile);

function cachedLoaderPlugin() {
  const loadPromises = Object.create(null);
  return {
    load(id) {
      return loadPromises[id] || (loadPromises[id] = readFileAsync(id, "utf8"));
    }
  };
}

const cachedLoader = cachedLoaderPlugin();

And then just inject the same instance of the plugin into all builds:

plugins: [..., cachedLoader]

As at the moment it is missing any validation to ignore virtual modules injected by other plugins, it should probably be the last plugin.

[dwighthouse]

My intention wasn't to rant. Only to show a practical difference between parallel and sequential execution. In my case, it's the difference between running and crashing.

I tried your plugin. Thank you. It does work, but oddly worse than manually limiting the concurrency. Caching the file imports and doing all Rollup operations simultaneously takes about 25% longer and uses more CPU than limiting Rollup to perform no more than 50 bundling operations simultaneously.

[lukastaegert]

This is not surprising, I heard similar things before. It seems that when many file operations are done concurrently, they will start to slow down each other. Could be related to how disk drivers work (I mean in the old days when HDD read heads were moving around, it HAD to be slow!). And it could be that the slow part here is not necessarily just the loading of the files but already the correct file name resolution, which already does a lot of fs operations. Unfortunately improving here is very hard for Rollup if you have an array of several unrelated configurations where each one could have different plugins or even depend on a previous one.