Local first tooling

[erlend-sh]

Finally someone’s doing a Rusty take on PocketBase, awesome! 🙌

I’ve been thinking a bit about how such a thing could differentiate itself.

People who are using supabase/pocketbase aren’t using them first and foremost for performance, which is trailblase’s top selling point right now as far as I can tell.

It’s gonna take a very long time to compete with these incumbents directly; they cover a lot of ground. I’m pondering what other niche specializations TB could use to wedge itself into the market, and the one that gets me the most excited is local-first development:

https://localfirstweb.dev/
https://www.localfirst.fm/17

SQLite is already a default coupling for many of these emergent local-first (and by extension dweb) stacks, since SQLite is itself innately local-friendly.

https://turso.tech/local-first
https://docs.expo.dev/guides/local-first/
https://expo.dev/blog/local-first-application-development-with-livestore
https://tinybase.org/

A bunch of us are working on this social-web stuff together here: https://blog.muni.town/muni-town/

Would love to have you around as well! ☺️

[ignatz]

Thanks for for your input and coming forward as #1 🥳

I’ve been thinking a bit about how such a thing could differentiate itself.

We should keep this conversation open, would love to hear many perspectives. Certainly something I was thinking about. I've used both in the past and liked them both but for different reasons. TrailBase is my love letter trying to marry their strengths.
I tried to share some of my thoughts on https://trailbase.io/comparison/pocketbase/. That said, being so new and just about an MVP, it's also an opportunity to do things differently.

People who are using supabase/pocketbase aren’t using them first and foremost for performance, which is trailblase’s top selling point right now as far as I can tell.

Performance is certainly highlighted very prominently right now. I think I agree that for most SupaBase users, especially hosted customers, performance might not the highest priority. I do wonder if the average PocketBase users, especially with self-hosting being the default, is a bit more intrigued by performance/footprint 🤷‍♀️

the one that gets me the most excited is local-first development:

Could you get into a bit more detail about what you have in mind? When I hear local-first, i'm not thinking about SQLite running on a server but rather on the user's device. You also pointed at TinyBase, which makes me think that you might be thinking of some TrailBase <-> Client-Sqlite sync?

[erlend-sh]

I do wonder if the average PocketBase users, especially with self-hosting being the default, is a bit more intrigued by performance/footprint

the thing is, afaik PocketBase already runs light enough to work with the very lowest ~$5 tiers of cloud hosting, so there’s no monetary upside for added performance.

There’s an ecological upside, which I for one greatly care about and also use Rust by default in my own projects for that reason, among many others 🦀 ..but ‘eco-friendly’ is usually not a strong enough sell on its own.

You also pointed at TinyBase, which makes me think that you might be thinking of some TrailBase <-> Client-Sqlite sync?

Yeah, I’m pointing in that general direction. I’m a product designer, not an engineer, so I’m thinking about this from a use-case perspective. And from my POV there’s a much greater need and breakout-potential for greenfield developments in the local-first space than there is in the traditional cloud-first space.

Therefore, I’m thinking maybe the *Base you wanna be measuring yourself against isn’t so much PocketBase but rather TinyBase.

I’m openly biased on all this though, as a team of us are working on this category of application as well. We are trying to build what we call agentic software, and local-first capabilities play a central role in that.

Our version of a vaguely TinyBase-like app is the app-engine behind weird.one — https://github.com/muni-town/weird

Here’s one way to paint the paradigm shift:
https://www.powersync.com/blog/api-calling-is-to-sync-engines-as-jquery-is-to-react

With the right sync engine, the developer can declaratively define what data should be present on the client. The sync engine then automatically takes care of the complexities of partial replication of the data, including all the complexities of dealing with the network (streaming real-time updates, error handling, retries, online vs. offline state, etc.), reconciling local updates with remote changes, conflict resolution (in many cases), optimizing syncing performance, and so on.

A good sync engine will expose a local database like SQLite which enables the developer to deal with a real local database in the front-end — significantly reducing the complexity of working with data. The sync engine ensures the local database stays in sync with the backend database as needed, just as React ensures the UI/DOM stays in sync with the application state.

Alternatively then we could say Weird should be the TinyBase-like one, while supporting Database Persistence to sqlite via TrailBase.

I’m just playing with different framings of this sort now to explore our alignments, since we already have a lot of overlap in our tech stacks.

[erlend-sh]

Another pitch for local-first here:
https://bugs.rocicorp.dev/issue/0
https://zero.rocicorp.dev/docs/introduction

People didn’t want a sync engine at all. Or rather they didn’t want to think about sync, or what data was synced to what device.

What they really wanted was a new kind of distributed database.

A database where you can just write queries, directly on the client. Just like you’re on the server. A database that figures out what to sync on its own, so that most queries resolve using local data, instantly, but where all data is still available.

Zero has two parts: zero-client is a TypeScript library you embed into your application, and zero-cache is a dockerized service you run in AWS (or wherever). We will also offer zero-cache as a service in the future.

For typical applications, the result is that almost all queries are answered locally, instantly. It feels like you have access to the entire backend database directly from the client in memory. Occasionally, when you do a more specific query, Zero falls back to the server. But this happens automatically without any extra work required.

Zero is made possible by a custom streaming query engine we built called ZQL, which uses Incremental View Maintenance on both client and server to efficiently keep large, complex queries up to date.

This provides some more insights on the benefits of the local-first paradigm. It can be used to make altogether snappier apps.

[ignatz]

Thanks! I'm all for snappy apps and using local where it makes sense (privacy, latency, offline, ... ).

A few random thoughts:

• I've since looked a little closer at https://github.com/vlcn-io/cr-sqlite, and it seems pretty cool but it's not clear to me in what state it is in, e.g. maintained. And I'm carefully skeptical (but happy to be wrong) that it wouldn't work with some of the HA solutions we've discussed on the other thread (hiqlite, rqlite, dqlite). This is not a show stopper but merely means that we might have to start out from scratch for sync and read-replicas to work together.
• Skimming Zero, it sounds pretty cool. It's important to high light that they rely on "thick client" doing a lot of the heavy lifting, e.g. streaming joins, .... At the moment TrailBase has a very thin client and tries to cater to different languages and runtimes. Limiting the runtimes to only JS would certainly make things easier there. For TrailBase the only crossplatform client option would be WASM but even that isn't trivial because the different integrations, bundling, versions across client-side runtimes (e.g. browser, nodejs, deno, dart, flutter, c#, .... python would arguably be less relevant on clients)

People didn’t want a sync engine at all. Or rather they didn’t want to think about sync, or what data was synced to what device.

Yes and no. That's simply an over-generalization or marketing speak. It's certainly true for certain data (e.g. settings that can be merged on a last-modified strategy) and/or single-client-server sync. If you have client <-> client sync and/or richer data you'll inevitably end up with the need for non-trivial conflict-resolution, i.e. not all problems can be modeled as CRDTs.

That said, I echo your desire and see the appeal of offering an easy solution where this is possible. An ideal solution for me right now would look like

1. There was a thin-client or generic client-side solution that we could satisfy by providing a TrailBase adapter. Maybe Zero's zero-cache is that adapter. I will take a look.
2. It would work for most or all client-side runtimes. I'm torn on only supporting browser-JS. I feel naively that mobile developers are the ones that benefit the most from *Base solutions.
3. And it should work with read replicas or HA solutions unless we want to closer the book on this topic

You've clearly thought about this a lot. Does the above sound about right? Do you have a particular solution or path that you fancy?

[erlend-sh]

Does the above sound about right?

Certainly! I greatly appreciate your deeply reasoned replies to my casual ruminations. Honestly I’m not expecting an outcome here other than this open dialogue about what problems need solving by which tools.

I think you have a good idea of what TrailBase should be and it’s not solving the same problems as Zero et.al., so we need both!

[erlend-sh]

Do you have a particular solution or path that you fancy?

I don’t, not in present context anyway, I just have a view towards some general approaches that seem useful to consider, the latest of which being..

https://devstarterpacks.com/blog/what-every-developer-should-know-about-offline-first-apps

It highlights one stack strategy that aligns neatly with what TrailBase is already doing:

WebAssembly + SQLite: One of the most exciting recent developments in offline storage is the ability to run SQLite directly in the browser through WebAssembly. Imagine having a complete SQL database engine running locally in your browser – it’s like carrying a miniature data center in your pocket. This approach offers several compelling advantages over traditional browser storage solutions.

For developers already familiar with SQL, this option feels like coming home. Instead of learning new APIs or adapting to the peculiarities of IndexedDB, you can write standard SQL queries just as you would in a server environment. Complex operations that might require multiple IndexedDB transactions can often be expressed in a single, SQL query.

The combination of WebAssembly and SQLite provides near-native performance, making it particularly powerful for applications that need to handle large datasets or complex queries. The familiar SQL interface also makes it easier to migrate existing applications that already use SQLite on the backend.

However, this power comes with important trade-offs to consider. The initial load time of the WebAssembly module can be noticeable, and the overall bundle size is larger than using native browser storage options. This solution works best for applications where users typically engage in longer sessions and require sophisticated data querying capabilities.

Running sqlite in the browser / webassembly is only gonna get easier over the years with projects like Limbo quickly maturing.

[ignatz]

It highlights one stack strategy that aligns neatly with what TrailBase is already doing:

Could you elaborate a bit more on where you see the alignment of Browser-Sqlite and what TrailBase is already doing?

Running sqlite in the browser / webassembly is only gonna get easier over the years with projects like Limbo quickly maturing.

FWIW, in another project of mine I've been running WASM+SQlite for years, I don't necessarily think that's the hard problem. The problem is the syncing/consistency when you have data that is concurrently being manipulated. As far as I can tell that's not in scope for Limbo, which itself is probably still years out. Would love to be wrong

[ignatz]

the thing is, afaik PocketBase already runs light enough to work with the very lowest ~$5 tiers of cloud hosting, so there’s no monetary upside for added performance.

That's fair. If you're not maxing out the cheapest offering there's not escape to the bottom. There might be a slight upside in terms of latency but PocketBase is plenty fast.

Yeah, I’m pointing in that general direction. I’m a product designer, not an engineer, so I’m thinking about this from a use-case perspective.

Hearing that you're a passionate crustacean, you might be selling yourself short.

And from my POV there’s a much greater need and breakout-potential for greenfield developments in the local-first space than there is in the traditional cloud-first space.

FWIW, I'm a big fan of local-first, there's a lot of nice properties. I'm also old enough to remember when everything was local first before SaaS became the new "traditional" :)

Therefore, I’m thinking maybe the *Base you wanna be measuring yourself against isn’t so much PocketBase but rather TinyBase.

Maybe. They don't seem mutually exclusive, i.e. TinyBase only handles the web-client side syncing with a separate storage layer. I haven't looked into the details but it should be possible to hook the both together, at least once TrailBase supports change subscriptions to enable 2-way syncing (I'm guessing here).

I also wanted to look into https://github.com/vlcn-io/cr-sqlite, which seems very related.

I’m just playing with different framings of this sort now to explore our alignments, since we already have a lot of overlap in our tech stacks.

I'm very open.

Alternatively then we could say Weird should be the TinyBase-like one, while supporting Database Persistence to sqlite via TrailBase.

From a layering point of view, it sounds very reasonable to split the sync layer from the underlying storage. If you wanted to integrate with TrailBase would you need anything beyond change subscriptions? (no strings attached)