wallet/db_sqlite3.c: Support direct replication of SQLITE3 backends.

ChangeLog-Added: With the `sqlite3://` scheme for `--wallet` option, you can now specify a second file path for real-time database backup by separating it from the main file path with a `:` character.

doc/BACKUP.md

@@ -82,6 +82,92 @@ any in-channel funds.
 To recover in-channel funds, you need to use one or more of the other
 backup strategies below.
 
+## SQLITE3 `--wallet` `:` And Remote NFS Mount
+
+`/!\` WHO SHOULD DO THIS: Casual users.
+
+`/!\` **CAUTION** `/!\` This technique is only supported on 0.10.3
+or later.
+On earlier versions, the `:` character is not special and will be
+considered part of the path of the database file.
+
+When using the SQLITE3 backend (the default), you can specify a
+second database file to replicate to, by separating the second
+file with a single `:` character in the `--wallet` option, after
+the main database filename.
+
+For example, if the user running `lightningd` is named `user`, and
+you are on the Bitcoin mainnet with the default `${LIGHTNINGDIR}`, you
+can specify in your `config` file:
+
+    wallet=sqlite3:///home/user/.lightning/bitcoin/lightningd.sqlite3:/my/backup/lightningd.sqlite3
+
+Or via command line:
+
+    lightningd --wallet=sqlite3:///home/user/.lightning/bitcoin/lightningd.sqlite3:/my/backup/lightningd.sqlite3
+
+If the second database file does not exist but the directory that would
+contain it does exist, the file is created.
+If the directory of the second database file does not exist, `lightningd` will
+fail at startup.
+If the second database file already exists, on startup it will be overwritten
+with the main database.
+During operation, all database updates will be done on both databases.
+
+The main and backup files will **not** be identical at every byte, but they
+will still contain the same data.
+
+It is recommended that you use the same filename for both files, just on
+different directories.
+
+This has the advantage compared to the `backup` plugin below of requiring
+exactly the same amount of space on both the main and backup storage.
+The `backup` plugin will take more space on the backup than on the main
+storage.
+It has the disadvantage that it will only work with the SQLITE3 backend and
+is not supported by the PostgreSQL backend, and is unlikely to be supported
+on any future database backends.
+
+You can only specify *one* replica.
+
+It is recommended that you use a network-mounted filesystem for the backup
+destination.
+For example, if you have a NAS you can access remotely.
+
+At the minimum, set the backup to a different storage device.
+This is no better than just using RAID-1 (and the RAID-1 will probably be
+faster) but this is easier to set up --- just plug in a commodity USB
+flash disk (with metal casing, since a lot of writes are done and you need
+to dissipate the heat quickly) and use it as the backup location, without
+repartitioning your OS disk, for example.
+
+Do note that files are not stored encrypted, so you should really not do
+this with rented space ("cloud storage").
+
+To recover, simply get all the backup database files.
+Note that SQLITE3 will sometimes create a `-journal` or `-wal` file, which
+is necessary to ensure correct recovery of the backup; you need to copy
+those too, with corresponding renames if you use a different filename for
+the backup database, e.g. if you named the backup `backup.sqlite3` and
+when you recover you find `backup.sqlite3` and `backup.sqlite3-journal`
+files, you rename `backup.sqlite3` to `lightningd.sqlite3` and
+`backup.sqlite3-journal` to `lightningd.sqlite3-journal`.
+Note that the `-journal` or `-wal` file may or may not exist, but if they
+*do*, you *must* recover them as well.
+It is recommended that you use the same filename for both main and
+backup databases (just on different directories), and put the backup in
+its own directory, so that you can just recover all the files in that
+directory without worrying about missing any needed files or correctly
+renaming.
+
+If your backup destination is a network-mounted filesystem that is in a
+remote location, then even loss of all hardware in one location will allow
+you to still recover your Lightning funds.
+
+However, if instead you are just replicating the database on another
+storage device in a single location, you remain vulnerable to disasters
+like fire or computer confiscation.
+
 ## `backup` Plugin And Remote NFS Mount
 
 `/!\` WHO SHOULD DO THIS: Casual users.

doc/lightningd-config.5.md

@@ -193,6 +193,13 @@ The default wallet corresponds to the following DSN:
 --wallet=sqlite3://$HOME/.lightning/bitcoin/lightningd.sqlite3
 ```
 
+For the `sqlite3` scheme, you can specify a single backup database file
+by separating it with a `:` character, like so:
+
+```
+--wallet=sqlite3://$HOME/.lightning/bitcoin/lightningd.sqlite3:/backup/lightningd.sqlite3
+```
+
 The following is an example of a postgresql wallet DSN:
 
 ```

tests/test_db.py

@@ -7,6 +7,7 @@
 import base64
 import os
 import pytest
+import shutil
 import time
 import unittest
 
@@ -379,3 +380,36 @@ def test_local_basepoints_cache(bitcoind, node_factory):
     # after we verified.
     l1.restart()
     l2.restart()
+
+
+@unittest.skipIf(os.getenv('TEST_DB_PROVIDER', 'sqlite3') != 'sqlite3', "Tests a feature unique to SQLITE3 backend")
+def test_sqlite3_builtin_backup(bitcoind, node_factory):
+    l1 = node_factory.get_node(start=False)
+
+    # Figure out the path to the actual db.
+    main_db_file = l1.db.path
+    # Create a backup copy in the same location with the suffix .bak
+    backup_db_file = main_db_file + ".bak"
+
+    # Provide the --wallet option and start.
+    l1.daemon.opts['wallet'] = "sqlite3://" + main_db_file + ':' + backup_db_file
+    l1.start()
+
+    # Get an address and put some funds.
+    addr = l1.rpc.newaddr()['bech32']
+    bitcoind.rpc.sendtoaddress(addr, 1)
+    bitcoind.generate_block(1)
+    wait_for(lambda: len(l1.rpc.listfunds()['outputs']) == 1)
+
+    # Stop the node.
+    l1.stop()
+
+    # Copy the backup over the main db file.
+    shutil.copyfile(backup_db_file, main_db_file)
+
+    # Remove the --wallet option and start.
+    del l1.daemon.opts['wallet']
+    l1.start()
+
+    # Should still see the funds.
+    assert(len(l1.rpc.listfunds()['outputs']) == 1)