How to execute long-running background functions (tasks) in production?

[stardvst]

I have an app where after user signs in with 3rd party application, user's followings are retrieved from that service.
For each of the followings, I get more data from another API, and store this overall information in my database.

In my test account with 70 followings, this process takes ~20 minutes (includes timeouts due to API rate limits).

const addFollowings = async followings => {
	const followingsData = []
	for (const following of followings) {
		const { id, name } = following
		const personInfo = new PersonInfo(name)
		await personInfo.init()
		const number = ...
                ...
		followingsData.push({ name, id, number, ... })

                console.log('new following:', name)
	}
	await insertFollowingsIntoDB(followingsData)
}

This works fine in development. But I have just deployed the app on Vercel, and when it comes to executing this function, firstly only a few followings are console-logged, then the function gets stuck: nothing is logged. Only when I refresh the page (which triggers /api/auth/session serverless function to get called), a few more are logged.

I guess this is because Vercel 10s limit on execution for functions.
Please advice how I should handle this. Thanks in advance.

[GabenGar]

You need a separate server for persistent tasks. Consider NextJS as a "frontend server" which only exists for request/response cycle and template rendering, not a replacement for existing servers.

[stardvst]

Thanks, I understand now. And how should I add that "backend server" into my Next.js application codebase?

[GabenGar]

Create a separate repo for it and host on heroku.

[AlessandroVol23]

I would also say you need to decouple these two a bit. Even with Next.js API routes, you'll face some limitations.

I am not sure if they function per sé should take 20 minutes to get followers but in general it is good practice that users only wait for synchronous processing on things they really need. For you function addFollowing it could for example only be some counter if it returns fast or only a message "Following initiated". All other tasks should run asynchronously in the background.

I'd use a message queue for things like that. You could pass the followings to a message queue and let it run in an async way.

For example:

1. User clicks on Button add following
2. Function in Vercel sends the message to a queue
3. Async flow starts
4. Function returns "process started"

The user now sees a response and the function addFollowings can be processed in the background.

This only makes sense in case you don't need this data right away but if it takes 20 minutes anyway I don't think you want your user to wait for that.

You could also split up your data into more messages to not reach the timeout. E.g. just 10 followings per message.

You need to define a separate consumer (either in Vercel or in another computing environment like AWS / Heroku).

Your message queue could be something like RabbitMq, SQS, or similar.

I've built https://serverlessq.com which is a hosted message queue solution that lets you decouple your applications exactly like I described above only with Vercel functions for example. It won't help you with the timeout but you could send a lot of messages and stay within your Vercel environment 🙁

I hope that helps!

[a-toms]

Site looks interesting! Would recommend showing some docs for devs before requiring sign up.

[pencilcheck]

This looks like just a pubsub, the hosting of server functions is still on vercel which has the limits that people want to get away from.

[anirudhbg]

I have been using https://www.calen-ai.com/ - they are basically the vercel for long running tasks! super easy to setup

[codinsonn]

Would Vercel's new waitUntil help with this?

https://vercel.com/changelog/waituntil-is-now-available-for-vercel-functions

[GabenGar]

No, because you still need a real server with a real CPU to process the task in the end.
The main use for this function is running maintenance logic such as logging without blocking request/response cycle.

[ataylorme]

Long running tasks should be possible in Next with containerized self hosting