Best practices for dealing with large data sets on server vs client

[Ehesp]

We're building an app using Firebase, however one of the issues we have is dealing with data from on the server vs client.

On the server, we're able to use the admin SDK, so the data loads via websockets and is generally fast to load. On the client we do this using a HTTP request to get the same data. The HTTP request is slower. For small data sets this isn't really an issue since it all loads quick.

However when the data is large, there is quite a big delay if the client loads the data. When clicking on a Link, the app "hangs" while the request is being fetched, and then loads once it has the data.

One way to get around this is to do a "if server get the data, else load it once the client componentDidMount has been called". Is this the best way to handle such a scenario? Quite new to isomorphic so would like to get opinions of other ways around this.

Cheers

[arunoda]

So, here the delay is the time spent on to fetch data on the server. Basically, you should not spend much time on that.
And you don't need to render everything in the server. But the only stuff you really really want. (That shouldn't come with cost of a big delay)

Then for other data (specially firebase) use componentDidMount to initialize them.
For that, you could simply use utilities like this: https://github.com/kadirahq/react-no-ssr

[davibe]

Is that huge data important for SEO ? if not maybe you could just load it always on componentDidMount.

I don't understand why loading from client should be any slower than loading from the server since the data has to travel to the browser (http) anyway (plus the rendered page which is normally larger than the page with no data). If your server is faster you can create a proxy api and use it on componentDidMount.

If you use some lib to manage state like redux

• by custom-handling the link you could also put up a loading spinner before routing to the new page
• you could even handle the link click so that it starts downloading the firebase data while also moving to the new page in (parallel). Then the page can wait for the promised data in componentDidMount (again).
• you could also have a look at the next.js prefetch api.

Also consider that, as a workaround, by using an <a href="page"> element you can force a full-page refresh therefore always requiring the server-rendered version of the page afaik

[arunoda]

you could even handle the link click so that it starts downloading the firebase data while also moving to the new page in (parallel).

@davibe This need a bit of a change. We don't download the data. We only prefetch JS code only.
See: #740

[Ehesp]

@davibe in this case no, so you guys are correct moving it to componentDidMount works better.

The server always has an open connect to Firebase, so there's no delay in opening a connection (initial connection time on WS is ~1000ms). The main problem we have is that with Firebase authentication, the server is unable to get data per user because firebase authentication is global. Therefore on the client we have to do it via HTTP calls.

We're building a CMS so we don't care about SEO in the case... Although we have plans for building a blog where SEO is really important, but we don't have to worry about authentication therefore server and client will be loading the data via websockets.

[sedubois]

I wonder how Apollo GraphQL subscriptions would compare to the Firebase approach in terms of these client-side performances 🤔 There are headless GraphQL CMSes which are starting to appear, so I guess it must be at least ok? (Can't yet say in my app because doesn't have much data yet... but after all it's based on Facebook technology which has huge data)

Also see the Firebase vs Graphcool feature comparison on https://graph.cool

[cdock1029]

When you say that on the client you're using http, is this because you're hitting your own backend to get the data? Because Firebase client javascript SDK certainly uses websockets.

Especially if SEO not that important, why not use client SDK to talk directly to Firebase from client?

Or, you can load specific user data on server on first load then subscribe to updates on client.
The server can get data per user, by the way:
https://firebase.google.com/docs/auth/admin/verify-id-tokens

You just have to structure your data correctly, isolated to each user, then after verifying the id-token, operate on that users data only
https://firebase.google.com/docs/database/web/structure-data

One thing not highlighted in the docs that I recently found out about is ability for SDK to cache data.
Maybe you're already doing something like this but if not and for others new to Firebase...

Firebase will cache reads in memory as long as there is an active listener for an event.

.once listeners automatically unsubscribe by definition. So on server you need to add a no-op listener to keep the cache alive.

const refSet = new Set()
function noOp() {}
function cache(ref, event) {
	if (!refSet.has(ref.toString())) {
		refSet.add(ref.toString())
		ref.on(event, noOp)
	}
}

export function getListElements({ refPath, orderBy }) {
	init()
	let ref = firebase.database().ref(refPath)
	if (orderBy) ref = ref.orderByChild(orderBy)

	// this just adds the no-op listener the first time we get data
        cache(ref, 'value')
        // use this in your own code to see the speedup. It's a lot.
	console.time('fb getListElements time')

	const promise = ref.once('value').then((listSnapshot) => {
		console.timeEnd('fb getListElements time')
		const list = []
		listSnapshot.forEach((childSnapshot) => {
			const childVal = childSnapshot.val()
			console.log('childVal:', childVal)
			list.push(childVal)
		})
		return list
	})
	return promise
}

Now on the client, even when using .on('value') or .on('child_added') with React components,
we need to add the no-op because usually we unsubscribe the listener on componentWillUnmount()

With both of these, when the event you've subscribed to fires ...by whomever, wherever... the caches get updated.