Discussion - How to store Insomnium using the FS

[austinwilcox]

I've been a long time Insomnia user (coming on 5 years), and I've hated that it's a nightmare to import/export data and use your own syncing features. So I would love to spark a discussion on how we can get off NeDb and use the FileSystem instead.

I am wondering if it makes sense to use a file for each request, and a folder will be reflected as such in the FS. I have been using Obsidian for my note taking, and they make it so easy to write notes, and then also to edit those notes in a different tool like Neovim for example. Having the ability to edit an API call by modifying the file directly would be incredible.

Thoughts? I haven't spent any time looking at Nedb and how it's currently stored so I might be off base with this idea to start with, but once an idea is settled in how to store everything offline I am happy to contribute the code that will make this possible.

[archywillhe]

hey Austin, thanks for using Insomnium and bringing this to the table

Yeah, using a FileSystem-based storage is in Insomnium's roadmap to milestone 1.

Currently Insomnia uses NeDB which is not maintained since 2016 (and there appear to be many patches of it inside Insomnia's code).

Now that Insomnium is going to exist as a 100% local app (strictly without any cloud services syncing, etc), it makes sense that we store projects/requests/documents as folders/files. This will make import/export less clunky, and the most recent N API results can be saved into a .api_cache folder - making it easier to feed into local LLMs for inference/finetuning.

I've been looking into Nedb recently and stumbled upon seald's fork of it, which they've been maintaining since 2021 and is pretty dope. They are basically making it faster and also rewriting the API into a subset of MongoDB's.

NeDB is not intended to be a replacement of large-scale databases such as MongoDB, and as such was not designed for speed. That said, it is still pretty fast on the expected datasets, especially if you use indexing. On a typical, not-so-fast dev machine, for a collection containing 10,000 documents, with indexing:

Insert: 10,680 ops/s
Find: 43,290 ops/s
Update: 8,000 ops/s
Remove: 11,750 ops/s
You can run these simple benchmarks by executing the scripts in the benchmarks folder. Run them with the --help flag to see how they work.

My plan for Insomnium now is to gradually transition to use seald's NeDb: having a memory-based DB is still quite useful (e.g. that makes our requests/parameters queryable, which in turn makes integrating something like Flowise into Insomnium much easier, so we can create complex interactions of requests & variable references). Also it generally makes reading/writing faster vs directly to file system.

With that in mind, the file-system-based storage can come in the form of having an option for users to open an existing folder as workspace (a bit like the way bruno does it - but keep it optional).

And when users do that, Insomnium will start another thread to performance a 1sec-debounce-save everytime a change has been made in NeDB, and create/update a file collectionName-requestName.json inside an .insomnium/ folder in the directory (akin to having.vscode/).

that will open up two possiblities:

• users can git add . and commit & push this folder to their repo (to share among an org)

• there can be another (e.g. npm, pip) library where users can do something like

import { fetch } from 'insomnium'

const request = await fetch("requestName", {auth: process.env.secret, parameters: {"user: "hello"} } )

to directly perform a query that is stored inside .insomnium/. (This is a feature I have always wanted to have since I started using Insomnia 3 years ago)

[archywillhe]

(I've editied the title to be about Insomnium and not Insomnia; I think we will be departing from Insomnia quite substantially and there is no intention of opening pull-request to Insomnia)

Yesterday I found a problem with Insomnia's NeDB setup that caused a bug when I was debugging Url segment params. But the problem was gone in production. After clearing the user-data in dev the bug disappeared; super weird. I'm really looking forward to see how good @seald/NeDb is (hopefully it will not be necessary to completely revamp the in-memory db system. And having file-system support will give it much more stability - and then the db system would play the role of complementary)

[EricLanduyt]

Hey guys, really loving the idea of a pure FS based storage, especially as it would allow sync/diff using git and so on of course.
To push this idea further, what about storing collections using open standards, such as OpenAPI for HTTP/REST collections ? (additional Insomnium related metadata could also be stored under x-* prefixed key, which is also part of the standard).

[archywillhe]

@EricLanduyt aw good idea; that's definitely very useful - and something that LLM can be really good at

(Local LLM support will be out in the next release! Can start giving it the raw request Insomnium JSON + API result + spec description, and test to see if it can make an OpenAPI spec and write test-cases)

[TCB13]

Okay, my three cents. I muay be out of touch?

1. This is and is supposed to be a fork of Insomnia without the tracking, cloud account requirements (?) etc. Although I like the ideia of a FS based storage and I would likely sync with git and/or Syncthing isn't that going to fundamentally change the code of the application making it so it will be way harder / impossible to merge future changes in the Insomnia project back to this one? This would mean a TON of work every time a new feature is released in Insomnia to port it to the FS storage over here. Isn't this project supposed to be more like https://github.com/ungoogled-software/ungoogled-chromium where you take every release of Insomnia remove the spyware / garbage and leave the rest alone thus proving always the latest and greatest? I think this approach might be more sane and keep this project running for longer.

2. Insomnia seems to currently support git sync (not with their cloud):
   

Why not just use this for sync? Yes, it is primitive, no way to setup a custom git only GitHub and GitLab but the inner workings of the thing are there, the entire UI and git operations. I guess a generic Git option will come from them very soon or could be implemented quickly.

3. I've been using https://insomnia.rest/plugins/insomnia-plugin-uu-sync to sync collections and while is isn't something magical like Postman it does work.

Thank you for the great work!

[austinwilcox]

@TCB13 I am not the perfect person to answer this, but to my understanding there are no plans on ever merging in changes with Insomnia.

[TCB13]

@TCB13 I am not the perfect person to answer this, but to my understanding there are no plans on ever merging in changes with Insomnia.

Okay... but that means... A LOT of work to implement new features and/or deal with bug fixes. Merging things back would probably make things easier. However I'm not sure about this because Kong said one of the reasons for this move was to have only one storage backend... maybe they're moving everything to the cloud?

[t00]

Just a suggestion how to manage such sync: use a separate Kong/insomnia branch and use a "patched branch" which will contain all the necessary changes to make offline-only functionality possible.
GitHub actions could be even adapted to merge latest insomnia changes into patched branch for testing/checking issues.
This way all functionalities coming to Insomnia 8.0 and later will land relatively easily in this fork.
All above would likely require either:

1. Abandoning existing fork changes and resetting git to latest Insomnia version while keeping using internally Insomnia name to avoid excessive number of merge conflicts and too many
2. Doing (1) in git but adding a script to rename Insomnia to Insomnium after git checkout, as a part of a build, possibly using regex with some exclusions. This will allow minimal git changes - only removing online features - while having a unique name.
3. Merging all changes made since 2023.5.8 to this repo (likely impossible due to repo-wide rename) and continuing on 2 branches from there with occasional merge from original repo.
4. Complete cut off from Insomnia improvements after 2023.5.8 with selective and occasional bugfix merge.

Most sensible in my opinion is option (2) - minimum effort for all new features and bugfixes.

[TCB13]

• Doing (1) in git but adding a script to rename Insomnia to Insomnium after git checkout, as a part of a build, possibly using regex with some exclusions. This will allow minimal git changes - only removing online features - while having a unique name.

This is what I was thinking as well, but I'm not sure it could be done. Kong said (Kong/insomnia#6577 (comment)) one of the reasons for this move was to have only one storage backend... this would mean there will be no way to "only remove online features" as their online storage is baked deep into the thing.

[t00]

That is sad indeed, there might be a way to replace online calls service but it means more work. RIP Insomnia 🥺

[archywillhe]

@TCB13 thanks for the pointer regarding git sync; I just checked the source & noticed it was implemented with isomorphic-git underneath, so it's pretty dope. I will re-roll back this local-based git-sync in the next release (preivously I wasn't aware that this feature existed)

[archywillhe]

and regarding these two points:

isn't that going to fundamentally change the code of the application making it so it will be way harder / impossible to merge future changes in the Insomnia project back to this one? This would mean a TON of work every time a new feature is released in Insomnia to port it to the FS storage over here.

yes. And I honestly don't think many of Insomnia's new features are going to be valuable to Insomnium. I'm not a big fan of Kong's product direction, especially since that day when it automatically updated my app to 8.0, and then all my data at work just disappeared and it really freaked me out. In short, I have no trust in their post-8.0 product direction (in terms of UI, UX, database, etc). I just don't believe they care enough about their users who don't use their cloud services.

(Of course, if I come cross fixes Kong made for front-end bugs, I will try bring them over to Insomnium ::) Similiarly, for every new fix/thing I implemented in Insomnium, I don't mind Kong taking them to Insomnia. I think that's the kind of silent agreement that comes with forking.)

Just a suggestion how to manage such sync: use a separate Kong/insomnia branch and use a "patched branch" which will contain all the necessary changes to make offline-only functionality possible. etc

The routing and data-structures as implemented in Insomnia post-8.0 are substantially different from v2023.5.8, which is the Insomnia's version that Insomnium is based on. From what I've observed, Kong wanted Insomnia to be like Notion but for APIs. They are using Notion and other cloud-centric apps to justify their product direction. This implies that in the app's architecture they wouldn't mind trading off UI/UX speed for cloud-sync, real-time collab, etc. Makes perfect sense too considering at its core Kong is a cloud company (and they are likely making their cloud/enterprise users happy. Good for them I guess.)

My direction for Insomnium is simple: keep it 100% local & forever FOSS, and keep the UI/UX fast. And then add in optional local LLMs integration (e.g. with code llama, etc) so Insomnium becomes both a API and LLM client where users can use LLM to automate test generation for their API calls, or to genereate front-end code (eg React, htmx components), etc, as well as using Insomnium to fine-tune local LLMs - because I think that's the future. (I started working with GPT2 and GPT3 three years ago. And I think we're finally entering the era of LLMs.) I'm pretty sure very soon all API clients will be integrated with LLMs one way or another. And maybe Insomnium will become the number-one API+LLM client ::)

so yeah, we will have local FS storage

[TCB13]

I will re-roll back this local-based git-sync in the next release (preivously I wasn't aware that this feature existed)

The only bad thing about their git sync is that it doesn't support generic git "servers" over SSH like any other thing.

Similiarly, for every new fix/thing I implemented in Insomnium, I don't mind Kong taking them to Insomnia. I think that's the kind of silent agreement that comes with forking.)

Well I was just tying to reduce your workload. I have no experience with their 8.x interface and I'm not aware of the roadmap so you're probably better than I could ever be at this decision.

For what's worth I really like the 2023.5.8 UI and feature set, it does everything I need and more. :)

Thank you for the fork.

[archywillhe]

thanks; they were great feedbacks btw

doesn't support generic git git "servers" over SSH like any other thing.

aw i see; looks like the best isomorphic-git can do is https isomorphic-git/isomorphic-git#1267

[TCB13]

aw i see; looks like the best isomorphic-git can do is https isomorphic-git/isomorphic-git#1267

Well that's fair git solutions usually support HTTP, even Gitolite (what I use in multiple cases) seems to have a way to make it HTTP: https://gitolite.com/gitolite/contrib/ssh-and-http.html

[hdavid]

doesn't support generic git git "servers" over SSH like any other thing.

aw i see; looks like the best isomorphic-git can do is https isomorphic-git/isomorphic-git#1267

i might be a rare case, but my company does not allow access to GIT over HTTP. in the past i relyied on an insomnia git plugin to overcome this, but it would be great if it would work over SSH for the stock git integration.

[sixdouglas]

Hi all,
Thanks for looking into this issue and of course many thanks for Insomnium! It really great for you to have done this.
Now, on this subject, a wonderful improvement would be to be able to have secret values managed in a separate .env file that would substitute the ones declared in the environment.
This way commiting request and environment could be possible while secrets will remain local thanks to an entry in .gitignore file.