scripts+make+GH: Add simple backwards compatibility test to the CI

.github/workflows/main.yml

@@ -528,7 +528,23 @@ jobs:
 
       - name: release notes check
         run: scripts/check-release-notes.sh
-
+
+  ########################
+  # Backwards Compatibility Test
+  ########################
+  backwards-compatability-test:
+    name: backwards compatability test
+    runs-on: ubuntu-latest
+    steps:
+      - name: git checkout
+        uses: actions/checkout@v4
+
+      - name: 🐳 Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: 🛡️ backwards compatibility test
+        run: make backwards-compat-test
+
   # Notify about the completion of all coverage collecting jobs.
   finish:
     if: ${{ always() }}

Makefile

@@ -364,6 +364,11 @@ help: Makefile
 	@$(call print, "Listing commands:")
 	@sed -n 's/^#?//p' $< | column -t -s ':' |  sort | sed -e 's/^/ /'
 
+#? backwards-compat-test: Run basic backwards compatibility test
+backwards-compat-test:
+	@$(call print, "Running backwards compatability test")
+	./scripts/bw-compatibility-test/test.sh
+
 #? sqlc: Generate sql models and queries in Go
 sqlc:
 	@$(call print, "Generating sql models and queries in Go")

docs/release-notes/release-notes-0.19.0.md

@@ -319,6 +319,9 @@ The underlying functionality between those two options remain the same.
 
 * [Fixed](https://github.com/lightningnetwork/lnd/pull/9549) a long standing
   unit test flake found in the `chainntnfs/bitcoindnotify` package.
+
+* Add a new CI-step to do some basic [backwards compatibility 
+  testing](https://github.com/lightningnetwork/lnd/pull/9540) for each PR. 
 
 ## Database
 

scripts/bw-compatibility-test/.env

@@ -0,0 +1,3 @@
+BITCOIND_VERSION=26
+LND_LATEST_VERSION=v0.18.5-beta
+TIMEOUT=15

scripts/bw-compatibility-test/README.md

@@ -0,0 +1,125 @@
+# Basic Backwards Compatibility Test
+
+This directory houses some docker compose files and various bash helpers all 
+used by the main `test.sh` file which runs the test. The idea is to be able to 
+test that a node can be upgraded from a stable release to a checked out branch
+of LND and still function as expected with nodes running the stable release.
+
+## Test Flow
+
+The test sets up the following network:
+
+```
+Alice <---> Bob <---> Charlie <---> Dave
+```
+
+Initially, all the nodes are running a tagged LND release. This is all 
+configured via the main `docker-compose.yaml` file. 
+
+1. The Bob node is the node we will focus on. While Bob is still on a stable 
+   version of LND, we ensure that he can: send and receive multi-hop payments as
+   well as route payments.
+
+2. Bob is then shutdown. 
+
+3. The `docker-compose.override.yaml` file is then loaded and used to spin up
+   Bob again but this time using the checked out branch of LND. This is done by 
+   using the `dev.Dockerfile` in the LND repo. 
+
+4. The test now waits for this new version of Bob (which uses the same data 
+   directory as the previous version) to sync up with the network, reactivate 
+   its channels.
+
+5. Finally, basic send, receive and routing tests are run to ensure that Bob
+   is still functional after the upgrade. 
+
+## How to use this directory
+
+1. If you would just like to run the full test from start to finish, then all 
+   you need to do is run: 
+
+   ```bash
+    ./test.sh
+   ```
+
+2. If you would like to run the test in parts, then you can use the `execute.sh`
+   script to call the various functions in the directory. Here is an example:
+   ```bash
+   # Spin up the docker containers.
+   ./execute.sh compose_up
+
+   # Wait for the nodes to start.
+   ./execute.sh wait_for_nodes alice bob charlie dave
+
+   # Query various nodes.
+   ./execute.sh alice getinfo
+
+   # Set-up a basic channel network.
+   ./execute.sh setup-network
+
+   # Wait for bob to see all the channels in the network.
+    ./execute.sh wait_graph_sync bob 3
+
+   # Open a channel between Bob and Charlie.
+   ./execute.sh open_channel bob charlie
+
+   # Send a payment from Alice to Dave.
+   ./execute.sh send_payment alice dave
+
+   # Take down a single node. 
+   ./execute.sh compose_stop dave
+
+   # Start a single node.
+   ./execute.sh compose_start dave
+
+   ```
+
+## File Descriptions:
+
+##### `test.sh`
+
+This script runs the full backwards compatibility test. 
+
+ ```bash
+   ./test.sh
+ ```
+
+##### `execute.sh`
+
+A helper script that allows you to call the various functions in the directory.
+
+This is useful if you are debugging something and want to step through the test
+steps manually while keeping the main network running.
+
+For example: 
+```bash
+ # Spin up the docker containers.
+ ./execute.sh compose_up
+
+ # Query various nodes.
+ ./execute.sh alice getinfo
+```
+
+#### `compose.sh`
+
+This script contains all the docker-compose variables and helper functions that 
+are used in the test.
+
+#### `network.sh`
+
+This script contains all the various helper functions that can be used to 
+interact with the Lightning nodes in the network.
+
+### `vars.sh`
+
+Any global variables used across the other scripts are defined here.
+
+#### `docker-compose.yaml`
+
+This docker compose file contains the network configuration for all the nodes
+running a stable LND tag. 
+
+#### `docker-compose.override.yaml`
+
+This compose file adds defines the `bob-pr` service which will pull and build
+the LND branch that is currently checked out.

scripts/bw-compatibility-test/compose.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+
+# DIR is set to the directory of this script.
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# The way we call docker-compose depends on the installation.
+if which docker-compose > /dev/null; then
+  COMPOSE_CMD="docker-compose"
+else
+  COMPOSE_CMD="docker compose"
+fi
+
+# Common arguments that we want to pass to docker-compose.
+# By default, this only includes the main docker-compose file
+# and not the override file. Use the `compose_upgrade` method
+# to load both docker compose files.
+COMPOSE_ARGS="-f $DIR/docker-compose.yaml -p regtest"
+COMPOSE="$COMPOSE_CMD $COMPOSE_ARGS"
+
+# compose_upgrade sets COMPOSE_ARGS and COMPOSE such that
+# both the main docker-compose file and the override file
+# are loaded.
+function compose_upgrade() {
+  export COMPOSE_ARGS="-p regtest"
+  export COMPOSE="$COMPOSE_CMD $COMPOSE_ARGS"
+}
+
+# compose_up starts the docker-compose cluster.
+function compose_up() {
+  echo "🐳 Starting the cluster"
+  $COMPOSE up -d --quiet-pull
+}
+
+# compose_down tears down the docker-compose cluster
+# and removes all volumes and orphans.
+function compose_down() {
+  echo "🐳 Tearing down the cluster"
+  $COMPOSE down --volumes --remove-orphans
+}
+
+# compose_stop stops a specific service in the cluster.
+function compose_stop() {
+  local service="$1"
+  echo "🐳 Stopping $service"
+  $COMPOSE stop "$service"
+}
+
+# compose_start starts a specific service in the cluster.
+function compose_start() {
+  local service="$1"
+  echo "🐳 Starting $service"
+  $COMPOSE up -d $service
+}
+
+# compose_rebuild forces the rebuild of the image for a
+# specific service in the cluster.
+function compose_rebuild() {
+  local service="$1"
+  echo "🐳 Rebuilding $service"
+  $COMPOSE build --no-cache $service
+}

scripts/bw-compatibility-test/docker-compose.override.yaml

@@ -0,0 +1,43 @@
+services:
+  bob-pr:
+    build:
+      context: ../../
+      dockerfile: dev.Dockerfile
+    container_name: bob-pr
+    restart: unless-stopped
+    ports:
+      - 10012:10009
+      - 9742:9735
+      - 8092:8080
+    networks:
+      regtest:
+        aliases:
+          - bob
+    volumes:
+      - "bob:/root/.lnd"
+    depends_on:
+      - bitcoind
+    command: >
+      lnd
+      --logdir=/root/.lnd
+      --alias=bob
+      --rpclisten=0.0.0.0:10009
+      --restlisten=0.0.0.0:8080
+      --color=#cccccc
+      --noseedbackup
+      --bitcoin.active
+      --bitcoin.regtest
+      --bitcoin.node=bitcoind
+      --bitcoind.rpchost=bitcoind
+      --bitcoind.rpcuser=lightning
+      --bitcoind.rpcpass=lightning
+      --bitcoind.zmqpubrawblock=tcp://bitcoind:28332
+      --bitcoind.zmqpubrawtx=tcp://bitcoind:28333
+      --debuglevel=debug
+      --externalip=bob
+      --tlsextradomain=bob
+      --accept-keysend
+      --protocol.option-scid-alias
+      --protocol.zero-conf
+      --protocol.simple-taproot-chans
+      --trickledelay=50