Cannot restart lightningd db since had to do fsck of hdd

[d4amenace]

I recently had to run an fsck and fix damaged sectors of my drive. Since I did this (and it was the only way to get my ubuntu running again), I am getting the following error message upon running lightningd:

Default-string:~$ lightningd
2020-11-12T23:09:10.885Z INFO database: Creating database
Error executing statement: wallet/db.c:1023: CREATE TABLE version (version INTEGER): database disk image is malformed

What are my options?

[ZmnSCPxj]

Unfortunately, if you do not have a continuous backup system like PostgreSQL cluster or the @cdecker backup plugin or ldadm/btrfs/ZFS RAID-1, you have no options. Sorry.

If you have an old backup of the database file, you can risk replacing the database with the old one, and if your peers are nice, they will close channels on you with the latest state (if they are not nice, they can steal your channels, but that is probably better than not having any chance at all of getting any of your funds back). Hopefully your peers support option_static_remotekeyoption_dataloss_protect -- if not, you will lose your channel funds (but the chance of getting some of your funds is better than losing all of them IMO).

Unfortunately due to the offchain nature of Lightning, there is nobody else who has a copy of your financial data (unlike the onchain case where every archival fullnode has a copy of every user financial data).

[d4amenace]

Dammit. Are LVM snapshots any good to avoid this? Or rdiff-backup?

[ZmnSCPxj]

Not sure. Can you recover the uncorrupted db file in an LVM snapshot?

Assuming "rdiff-backup" is similar to a process doing "rsync" regularly, you would strongly prefer to use something like the backup plugin or a PostgreSQL cluster. Stale backups are dangerous and could lose your channel funds, but then not having any backups is a definite complete and total loss of all your channel funds (whereas if your channeled peers have option_dataloss_protect then a stale backup will recover your funds, but if you have channels without option_dataloss_protect then a stale backup will definitely lose your in-channel funds with them), so it might be better than nothing.

In any case, if you have backups, you should strongly prefer the continuous backups (PostgreSQL cluster, backup plugin on an NFS mount) rather than snapshot backups, as snapshot backups can lose your money (but no backups definitely will).

[d4amenace]

PostgreSQL cluster, backup plugin on an NFS moun

Do you think you could point me in the right directions to find out more about these solutions. Links or web pages?

[ZmnSCPxj]

It seems we do not have single coherent document for this yet, sorry. Certainly could be something we add to doc/ directory.

[jsarenik]

Related to #4181 ?

[d4amenace]

If PostgreSQL is superior to SQLite, then why isn't it standard in the software? Curious

[ZmnSCPxj]

PostgreSQL requires a lot more setup especially if you want to replicate the data across multiple machines. And C-Lightning started out using SQLITE3 and without PostgreSQL support, so for back-compatibility, SQLITE3 support is default and PostgreSQL support is not compiled in by default (otherwise people using SQLITE3 databases from old C-Lightning versions would be forced to install the much larger PostgreSQL packages, whereas SQLITE3 is a fairly small library). Probably @cdecker can give more perspective on PostgreSQL use (possibly he is the only core developer of C-Lightning who has a PostgreSQL setup even, so...)

Also, nothing in the protocol requires any particular database, it only mandates when particular data is stored in permanent storage relative to your communications with peers, but that is just me being pedantic, you probably mean just "C-Lightning software" and not "protocol".

[rustyrussell]

If you can get an old copy of the db, it's worth a try: you might lose funds but usually not because peers are honest.

However, generally we can't recover from corrupted storage very well :(

[d4amenace]

you probably mean just "C-Lightning software" and not "protocol".

Exactly.

[d4amenace]

I am actually aware of how my db got corrupted. I can avoid it on my other servers. I am rolling out a payment system that uses C-lightning nodes and I am just looking for a backup solution for professional peace of mind.

[cdecker]

I think this issue has been addressed by the backup documentation (thanks @ZmnSCPxj ) so I think we can close this for now?