-
Notifications
You must be signed in to change notification settings - Fork 268
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add cluster doc, and packaging improvements (#1646)
We now use a regular release zip, and use it in the awseb bundle. The default logback configuration now logs to a file, like `eclair-node`, and we use a dedicated logback configuration for AWS beanstalk. By default the front reads the node secret key from the node seed in the standard `.eclair` directory.
- Loading branch information
Showing
13 changed files
with
691 additions
and
64 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,177 @@ | ||
| ## How to _clusterize_ your Eclair node | ||
|
|
||
| Eclair allows you to scale up one _logical_ lightning node across multiple servers. | ||
|
|
||
| Front servers take care of routing table related gossip and syncing requests from peers, which is cpu/bandwidth intensive. The backend server can focus on core channel management. So, BOLT 1&7 messages are handled in the frontend, while BOLT 2 messages go through and are processed in the backend. | ||
|
|
||
| Front servers are stateless, they can be stopped/killed at will. The node will remain operational and reachable as long as there is at least one `frontend` available. | ||
|
|
||
| ``` | ||
| +---+ +-----------+ | ||
| | | | +-------+ | | ||
| | |-----|-| | | | ||
| P -----| L |-----|-| FRONT |-|---, | ||
| U | O |-----|-| | | \ | ||
| B ---| A | | +-------+ | \ | ||
| L | D | | | \ | ||
| I | | | +-------+ | \ +------+ | ||
| C -----| B |-----|-| | | `| | | ||
| | A |-----|-| FRONT |-|---------| BACK |<-- channels management | ||
| N | L |-----|-| | | ,| | relay logic | ||
| E ----| A | | +-------+ | / +------+ | ||
| T | N | | | / | ||
| W | C | | +-------+ | / | ||
| O | E |-----|-| | | / | ||
| R -----| R |-----|-| FRONT |-|---' | ||
| K | |-----|-| |<------- connection management | ||
| | | | +-------+ | routing table sync | ||
| +---+ +-----------+ | ||
| ``` | ||
|
|
||
| The goal is to offload your node from connection and routing table management: | ||
| - incoming connections | ||
| - outgoing connections | ||
| - gossip queries + pings | ||
| - incoming gossip aggregation | ||
| - outgoing gossip dispatch (rebroadcast) | ||
|
|
||
| ### Prerequisite | ||
|
|
||
| You already have a node up and running in a standalone setup (with Bitcoin Core properly configured, etc.). | ||
|
|
||
| You know what your `node id` is. | ||
|
|
||
| In the following, what we previously called `eclair-node` will be called `backend`. It is to be launched, configured and backed-up exactly like in a standalone setup. | ||
|
|
||
| ### Minimal/Demo setup | ||
|
|
||
| Use this if you want to experiment with the cluster mode on a single local server. | ||
|
|
||
| Set the following values in `.eclair/eclair.conf`: | ||
| ``` | ||
| akka.actor.provider = cluster | ||
| akka.extensions = ["akka.cluster.pubsub.DistributedPubSub"] | ||
| // replace this with your node id | ||
| // if you don't know what your node id is, you should probably stop right here | ||
| eclair.front.pub = 03............... | ||
| ``` | ||
|
|
||
| Start the `backend`: | ||
| ```shell | ||
| $ ./eclair-node.sh | ||
| ``` | ||
|
|
||
| Then run an instance of `frontend`: | ||
| ```shell | ||
| $ ./eclair-front.sh -Dakka.remote.artery.canonical.port=25521 -Declair.server.port=9736 | ||
| ``` | ||
|
|
||
| NB: we override the ports, otherwise they would conflict since in this example everything runs on the same server. You can run multiple `frontend`s on the same server, just make sure to change the ports. | ||
|
|
||
| ### Production setup | ||
|
|
||
| In production you should: | ||
| - run multiple `frontend`s | ||
| - run one app per server | ||
| - enable `tcp-tls` to encrypt communications between members of the cluster with your own generated certificate (see below) | ||
| - use a load balancer to hide all your `frontend` servers under the same ip address | ||
| - set firewall rules to disable lightning connections (port 9735) on your `backend` server, so all connections go through the `frontend` | ||
| - enable [monitoring](Monitoring.md) | ||
| - on AWS, use AWS Secrets Manager (see [AWS deployment](#aws-deployment)) | ||
|
|
||
| #### Enable encrypted communication for the cluster | ||
|
|
||
| We use a self-signed certificate, which offers a good compromise. More advanced options are available, see [akka doc](https://doc.akka.io/docs/akka/current/remoting-artery.html#remote-security). | ||
| > Have a single set of keys and a single certificate for all nodes and disable hostname checking | ||
| > - The single set of keys and the single certificate is distributed to all nodes. The certificate can be self-signed as it is distributed both as a certificate for authentication but also as the trusted certificate. | ||
| > - If the keys/certificate are lost, someone else can connect to your cluster. | ||
| > - Adding nodes to the cluster is simple as the key material can be deployed / distributed to the new node. | ||
| Generate a self-signed certificate (set a strong password): | ||
| ```shell | ||
| $ keytool -genkeypair -v \ | ||
| -keystore akka-cluster-tls.jks \ | ||
| -dname "O=ACME, C=FR" \ | ||
| -keypass:env <password> \ | ||
| -storepass:env <password> \ | ||
| -keyalg RSA \ | ||
| -keysize 4096 \ | ||
| -validity 9999 | ||
| ``` | ||
|
|
||
| Copy the resulting certificate to your `.eclair` directory: | ||
| ```shell | ||
| $ cp akka-cluster-tls.jks ~/.eclair | ||
| ``` | ||
|
|
||
| Add this to your `eclair.conf`: | ||
| ``` | ||
| AKKA_TLS_PASSWORD=<password> | ||
| akka.remote.artery.transport = "tls-tcp" | ||
| ``` | ||
|
|
||
| ### AWS Deployment | ||
|
|
||
| For convenience, we provide a prebuilt AWS Beanstalk bundle for the `frontend` (choose a WebServer environment type, and Java platform). | ||
|
|
||
| You can run it as-is for testing. | ||
|
|
||
| #### TLS encryption | ||
|
|
||
| If you intend to use it in production, you need to enable encryption with your own certificate: | ||
| 1. Follow the procedure above to generate your `akka-tls.jks` | ||
|
|
||
| 2. We recommend forking the project and building your own bundle: | ||
| ```shell | ||
| $ git clone git@github.com:ACINQ/eclair.git | ||
| $ vi eclair-core/src/main/reference.conf # set akka.remote.artery.transport = "tls-tcp" | ||
| $ cp akka-cluster-tls.jks eclair-front/modules/awseb/ # copy the file you generated | ||
| $ vi eclair-front/modules/awseb.xml # uncomment the relevant parts | ||
| $ mvn package -DskipTests | ||
| ``` | ||
| Alternatively, you can also edit the existing bundle and manually add the `akka-cluster-tls.jks` file to the root of the zip archive. You will also need to set `akka.remote.artery.transport=tls-tcp` at runtime. | ||
|
|
||
| #### Private key | ||
|
|
||
| In production, we highly recommend using AWS Secrets manager to provide the node private key. This is done by setting `eclair.front.priv-key-provider=aws-sm`. Default secret name is "node-priv-key", but it is configurable with `eclair.front.aws-sm.priv-key-name` | ||
|
|
||
| #### Configuration | ||
|
|
||
| We recommend using Beanstalk environment variables for `AKKA_TLS_PASSWORD`, `BACKEND_IP`, and `NODE_PUB_KEY`. Other configuration keys should be set in the `AKKA_CONF` environment variable, semicolon separated. Example: | ||
| - `AKKA_CONF`: `eclair.enable-kamon=true; akka.remote.artery.transport=tls-tcp; eclair.front.priv-key-provider=aws-sm...` | ||
| - `AKKA_TLS_PASSWORD`: `xxxxxxxx` | ||
| - `BACKEND_IP`: `1.2.3.4` | ||
| - `NODE_PUB_KEY`: `03933884aaf1d6b108397e5efe5c86bcf2d8ca8d2f700eda99db9214fc2712b134` | ||
|
|
||
| Port `25520` needs to be open, within the Beanstalk security group, and between the Beanstalk security group and your `backend` node. | ||
|
|
||
| ### Tor | ||
|
|
||
| We recommend running your Tor hidden service on a separate server, and use `eclair.tor.targets` to redirect clearnet(*) connections to your `frontend` servers. | ||
|
|
||
| (*) Clearnet for Tor, but BOLT 8 encrypted. | ||
|
|
||
| Here is the resulting architecture: | ||
|
|
||
| ``` | ||
| +---+ +-----------+ | ||
| | | | +-------+ | | ||
| | |-----|-| | | | ||
| P -----| L |-----|-| FRONT |-|---, | ||
| U | O |-----|-| | | \ | ||
| B ---| A | | +-------+ | \ | ||
| L | D | | | \ | ||
| I | | | +-------+ | \ +------+ | ||
| C -------| B |-----|-| | | `| | | ||
| | A |-----|-| FRONT |-|---------| BACK |<-- channels management | ||
| N | L |-----|-| | | ,| | relay logic | ||
| E ----| A | | +-------+ | / +------+ | ||
| T | N | | | / | ||
| W +-------+ | C | | +-------+ | / | ||
| O | | | E |-----|-| | | / | ||
| R ---| Tor |------| R |-----|-| FRONT |-|---' | ||
| K | | | |-----|-| |<------- connection management | ||
| +-------+ | | | +-------+ | routing table sync | ||
| +---+ +-----------+ | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| <assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0" | ||
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd"> | ||
| <id>${git.commit.id.abbrev}-awseb_bundle</id> | ||
| <formats> | ||
| <format>zip</format> | ||
| </formats> | ||
| <includeBaseDirectory>false</includeBaseDirectory> | ||
| <fileSets> | ||
| <fileSet> <!-- release zip --> | ||
| <directory>${project.build.directory}</directory> | ||
| <outputDirectory>.</outputDirectory> | ||
| <includes> | ||
| <include>eclair-front-*-bin.zip</include> | ||
| </includes> | ||
| </fileSet> | ||
| <fileSet> | ||
| <directory>${project.basedir}/modules/awseb</directory> | ||
| <outputDirectory>.</outputDirectory> | ||
| <includes> | ||
| <include>Procfile</include> | ||
| <include>logback_eb.xml</include> | ||
| </includes> | ||
| <lineEnding>unix</lineEnding> | ||
| </fileSet> | ||
| <!-- uncomment if you want to include a keystore in the awseb package for TLS --> | ||
| <!--fileSet> | ||
| <directory>${project.basedir}/modules/awseb</directory> | ||
| <outputDirectory>.</outputDirectory> | ||
| <includes> | ||
| <include>*.jks</include> | ||
| </includes> | ||
| <fileMode>0400</fileMode> | ||
| <lineEnding>keep</lineEnding> | ||
| </fileSet--> | ||
| <fileSet> | ||
| <directory>${project.basedir}/modules/awseb</directory> | ||
| <outputDirectory>.</outputDirectory> | ||
| <includes> | ||
| <include>run.sh</include> | ||
| </includes> | ||
| <fileMode>0755</fileMode> | ||
| <lineEnding>unix</lineEnding> | ||
| </fileSet> | ||
| </fileSets> | ||
| </assembly> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <!-- | ||
| ~ Copyright 2019 ACINQ SAS | ||
| ~ | ||
| ~ Licensed under the Apache License, Version 2.0 (the "License"); | ||
| ~ you may not use this file except in compliance with the License. | ||
| ~ You may obtain a copy of the License at | ||
| ~ | ||
| ~ http://www.apache.org/licenses/LICENSE-2.0 | ||
| ~ | ||
| ~ Unless required by applicable law or agreed to in writing, software | ||
| ~ distributed under the License is distributed on an "AS IS" BASIS, | ||
| ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| ~ See the License for the specific language governing permissions and | ||
| ~ limitations under the License. | ||
| --> | ||
|
|
||
| <configuration> | ||
|
|
||
| <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> | ||
| <filter class="ch.qos.logback.core.filter.EvaluatorFilter"> | ||
| <!-- this filters out AWS ELB probes --> | ||
| <evaluator> | ||
| <expression> | ||
| return formattedMessage.contains("connected to /10.") || | ||
| (formattedMessage.contains("connection closed") && !mdc.containsKey("nodeId")) || | ||
| (formattedMessage.contains("transport died") && !mdc.containsKey("nodeId")) || | ||
| (formattedMessage.contains("stopping") && !mdc.containsKey("nodeId")); | ||
| </expression> | ||
| </evaluator> | ||
| <OnMismatch>NEUTRAL</OnMismatch> | ||
| <OnMatch>DENY</OnMatch> | ||
| </filter> | ||
| <target>System.out</target> | ||
| <withJansi>false</withJansi> | ||
| <encoder> | ||
| <pattern>${HOSTNAME} %d %-5level %logger{24}%X{category}%X{nodeId}%X{channelId}%X{paymentHash}%.-11X{parentPaymentId}%.-11X{paymentId} - %msg%ex{12}%n</pattern> | ||
| </encoder> | ||
| </appender> | ||
|
|
||
| <root level="INFO"> | ||
| <appender-ref ref="CONSOLE"/> | ||
| </root> | ||
|
|
||
| </configuration> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,11 @@ | ||
| export LOCAL_IP=$(curl -s 169.254.169.254/latest/meta-data/local-ipv4) | ||
| export HOSTNAME=$(hostname) | ||
| exec java -javaagent:lib/kanela-agent-1.0.5.jar -jar application.jar | ||
|
|
||
| # make the eclair home directory | ||
| mkdir -p /home/webapp/.eclair | ||
|
|
||
| # if provided, copy the certificate to the proper directory | ||
| [[ -e akka-cluster-tls.jks ]] && cp -v akka-cluster-tls.jks /home/webapp/.eclair/ | ||
|
|
||
| unzip -o eclair-front-*-bin.zip | ||
| exec ./eclair-front-*/bin/eclair-front.sh -with-kanela -Dlogback.configurationFile=logback_eb.xml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.