Firebase Checkpoint Saver

[trollboss2572]

As recommended by @benjamincburns , the goal is to implement a Firebase Checkpoint Saver as a child of the abstract BaseCheckpointSaver class.

Currently, there are 3 such implementations in the langgraph code base for SQLite, PostGres, and MongoDB. Adding another checkpoint saver type for Firebase would add to the redundancy needed for an essential task within agent workflow, in case the other databases become offline. Also, this implementation would make it easier for developers who are more familiar with Firebase to ascertain the inner workings of CheckpointSaver in case they come across bugs within their own code or langgraph source code.

The reason for implementing this checkpoint saver in Firebase in particular is that Firebase is a popular database choice for developers, on par with SQLite and Mongo.

Should this issue be accepted, me and a team of developers at the University of Toronto plan to do the following:
-Implement FirebaseDBSaver as a class within ./libs, following the example set by SqliteSaver and MongoDBSaver
-Implement and use a rigorous test structure following the example of the same classes as above to ensure that the code passes all expected use cases.

If there are any questions about our implementation, let me know. I'm happy for the opportunity to be working on such an interesting and expansive team.

[benjamincburns]

I'd recommend using the GCloud test container for testing this. Note that you'll need to skip tests that use docker in CI on Windows, as well as arm64 mac builds. This is because GH Actions don't yet (and may never) support docker on arm64 mac, nor do they support Linux containers on Windows. Here's an example (related utility function is here) of how I've done this in the checkpoint-validation test suite.

I'd also recommend using the checkpoint-validation package to validate the functionality of your new checkpointer. You can follow the test setup pattern shown here for the existing validators, and use that to validate consistency with the other checkpointers. You'll still want to add tests to your new package however if there are code paths that are specific to your particular implementation that you need to exercise (e.g. things like setting up the firebase store), as these things won't be covered by the checkpoint-validation test suite.

Should this issue be accepted...

I think I said as much elsewhere, but if you check any other issue/PR in this repository, the LangGraph team don't appear to accept/reject things ahead of time like you seem to be assuming in this comment. The process seems to be much less formal than that. For example, see #541, where I introduced the idea of the acceptance suite that ultimately became the checkpoint-validation package as of #600.

If you're waiting for them to respond before you begin work on this, you may find yourself waiting for a while. My recommendation would be to just get to work on it, and if for some reason the team doesn't wish to accept the PR into this repo, you can always make a new repo that houses just your checkpointer implementation, and publish it to NPM yourselves under your own package name. It won't live under the @langchain/* namespace, but you'll still be able to publish it for others to consume.

Hope this helps!

[trollboss2572]

Thank you for the suggestions! @benjamincburns

We are currently working on the test suite for our implementation, but we're having some problems and some confusion on the current structure of the test suite.

I've gone through the documentation for langgraph, and haven't seen any developer notes for how to test a Memory Saver implementation. All of what we've seen have been examples from other test implementations.

For example, here is the MongoSaver initializer, which is the main problem we're having.

We are trying to implement a container for Firebase, but we are confused on how to reference that from our testing code.

Within the MongoDB tester, a connection is opened to the initialized container using an environment variable, but within the initializer, a container is opened via a port on line 25, however the port is unknown, so how does the tester code know the exact URL to open for the container?

If this doesn't perform what we believe it does, that being the initializer starting all the container/environments needed to test, and the testing code running those tests given that environment, then what is the point of that initializer?

Apologies if these questions are trivial or are answered in the documentation. I am not experienced with testing via containers. Point me towards any documentation that answers any of the questions I have if they exist

Thank you for your time,
Casper

[benjamincburns]

Apologies if these questions are trivial or are answered in the documentation. I am not experienced with testing via containers.

No worries, and no need to apologize! The TestContainer project makes testing with container-based infrastructure relatively easy. I'll explain more below.

If this doesn't perform what we believe it does, that being the initializer starting all the container/environments needed to test, and the testing code running those tests given that environment, then what is the point of that initializer?

Yes, that's the intended purpose of CheckpointTestInitializer.

If you haven't seen it yet, I'd recommend starting off by looking to the README to learn about the design intent and usage of @langchain/langgraph-checkpoint-validation. Here's a direct quote from the first paragraph under the Writing a CheckpointTestInitializer heading:

"The CheckpointerTestInitializer interface (example) is used by the test harness to create instances of your custom checkpointer, and any infrastructure that it requires for testing purposes."

Within the MongoDB tester, a connection is opened to the initialized container using an environment variable, but within the initializer, a container is opened via a port on line 25, however the port is unknown, so how does the tester code know the exact URL to open for the container?

I'm not sure that I follow. Specifically, I'm not sure where that const client = new MongoClient(getEnvironmentVariable("MONGODB_URL")!); line came from, but it's not related to the initialization of MongoDBSaver as used by the @langchain/langgraph-checkpoint-validation validation suite.

Perhaps that came from the tests in the libs/checkpoint-mongo directory? If so, those tests are unrelated to the validation suite. They're implementation-specific tests that run as part of the LangGraph integration test suite, which uses another mechanism for managing test infrastructure. That is, when you check out a local copy of the repo and do yarn test, those tests don't run by default - they're meant to run in GitHub actions.

Instead, the MongoDBSaver test initializer uses the MongoDB TestContainer implementation to spin up and connect to a new instance of MongoDB.

In the initializer code, the connection string for the MongoDB client is built from details retrieved from the started test container on these lines. That connection string is passed to a new instance of the mongo client.

That mongo client instance is passed to new instances of MongoDBSaver that are used for testing here, and that client instance is also used for destroying and recreating databases between tests here.

To map this over to firebase, you likely want to be making use of the GCloud TestContainer implementation in a fashion similar to how the MongoDB TestContainer is used. There are examples of how to set it up to emulate firestore, however I'm not familiar enough with the firebase client or how your implementation works to say specifically how you'd wire it up in a test initializer.

Give that a try, and if you still can't figure out how to wire it up, feel free to ping me again on this issue, or via the LangChain community slack with some more details about your checkpointer implementation, the firebase client lib you're using, etc.

[trollboss2572]

Thanks for the quick response @benjamincburns ,

We are currently working on getting our initializer to pass validation tests, and are noticing that when running the validation test, we are getting an infinite loop:

import { FirebaseSaver } from "@langchain/langgraph-checkpoint-firebase";
import { initializeApp } from "firebase/app";
import { getDatabase, Database } from "firebase/database";
import type { CheckpointerTestInitializer } from "../types.js";

let database: Database;

export const initializer: CheckpointerTestInitializer<FirebaseSaver> = {
  checkpointerName: "@langchain/langgraph-checkpoint-firebase",

  async beforeAll() {

    const firebaseConfig = {
      authDomain: "localhost",
      projectId: "initialize-Firebase",
      databaseURL: "http://localhost:9000",
    };
    // process.env.FIREBASE_URL = process.env.FIREBASEURL || "http://localhost:9000"

    const app = initializeApp(firebaseConfig);

    database = getDatabase(app);

    console.log("Connected to Firebase Realtime Database Emulator");
  },

  beforeAllTimeout: 300_000, // Allow up to 5 minutes for emulator setup

  async createCheckpointer() {
    // Create a new instance of FirebaseSaver with the initialized database
    return new FirebaseSaver(database);
  },

  async afterAll() {
    console.log("Cleaning up Firebase Realtime Database Emulator");

  },
};

export default initializer;

Above is our code that we are trying. I know it's not a problem with the beforeAll function since the final console.log() is called.
I've also read the README that was helpfully included, however nothing in there discussed the destination or source files we may have to add, if that were to be the problem, and there were no applicable troubleshooting tips for such an error.

Regarding the recommended use of the GCloud test container, only the implementation of Firestore is allowed. From our experience, confusingly enough Firebase and Firestore are 2 different database types, so our implementation would not work on the test containers that GCloud would allow us to implement. Our main strategy at the moment is to open up a local database on the tester's local machine to run those tests. (At least for now, as a proof of concept to check out our understanding of the checkpoint-validation)

If this implementation wouldn't work, I am confused as to why we are entering an infinite loop, instead of outright failing tests due to environment problems.

Has anything you've encountered ever given you such a problem?

Thanks again for being so helpful and timely!

[benjamincburns]

Are you sure it's an infinite loop and not just the list tests taking a long time to run? I forget the exact number of tests, but IIRC it runs over 700 combinatorial tests. If you connect to it with a debugger you should hopefully be able to spot whether it's an infinite loop or not.

[benjamincburns]

Regarding firebase vs firestore, my apologies for that. I haven't worked with firebase much. This project has published a firebase emulator container that you might be able to try.

Since the interface is simple (just port mappings) you likely can spin it up with the core TestContainers project, as shown in this example.

[benjamincburns]

FWIW, I'd also suggest that you want to be using firestore for this, not the firebase rtdb. See this article. Firestore is orders of magnitude cheaper, and offers more robust query mechanisms for e.g. the list method.