Limit concurrency of remote file requests

[jonesbp]

Use bluebird’s Promise.map() with concurrency option to limit HTTP requests when creating remote nodes for file attachments in order to avoid being rate-limited by Airtable’s HTTP servers.

[jbolda]

More recent versions of Gatsby removed the global polyfill for Bluebird so that is likely why your tests are failing. There probably isn't really a better / native way to handle this so it might be worth just adding Bluebird as a dependency and explicitly using the imported function here (not the global polyfill). (Unless you have a better idea.)

At minimum, I would ask that you add a comment in the code on "why this specific limit", but I'm wondering if maybe we make this configurable (with a reasonable default) since we are kind of shooting in the dark here it seems.

[jbolda]

Looking great, and we have a green build. Would you mind adding some brief documentation? Then I think we are all set to merge.

[jbolda]

@jamessimone interested in reviewing since you looked before? Otherwise I'm happy with where this is at.

Thanks again for taking this on @jonesbp! 🎉

[jamessimone]

@jonesbp I do have some feedback about the wording in the readme. I'm also happy to push a commit for this if you'd like.

[jonesbp]

I have applied the suggested documentation edits. Thanks @jamessimone .

[jbolda]

Cheers! Thanks @jonesbp, published to v2.0.8.

[jamessimone]

@jonesbp nice job!

[rgesulfo]

Hi folks,
I'm still struggling with building my site from Airtable with a lot of attachements (images) per row.
If I understand correctly, I'm suppose to do concurrency: 0 in the gatsby-config.js. I'm just not sure where to put it.
@jonesbp, it seems that you were able to solve your problem. What is the solution for you?
Thanks for your help :)

[jbolda]

@rgesulfo oh, we didn't document it. It goes adjacent to the apiKey.

[rgesulfo]

Thanks @jbolda!
So something like this should work right?

...
{
      resolve: `gatsby-source-airtable`,
      options: {
        apiKey: `***`,
        concurrency: 0,
        tables: [
          {
            baseId: `***`,
            tableName: `***`,
            mapping: {
              photos: 'fileNode',
              summary: 'text/markdown',
            },
          },
        ],
      },
    },
...

Because it still fails for me.
Thanks again.