Telegraf as bonus for collecting system metrics

[frennkie]

Telegraf is a nice solution to gather system metrics that can be collected locally or remotely in a InfluxDB and this can then be displayed in Grafana dashboard. https://medium.com/@dorian599/iot-raspberry-pi-container-and-system-monitoring-with-influxdb-telegraf-and-grafana-a1767c38c109

sudo -i 

curl -sL https://repos.influxdata.com/influxdb.key | apt-key add -
DISTRIB_ID=$(lsb_release -c -s)
echo "deb https://repos.influxdata.com/debian ${DISTRIB_ID} stable" | tee /etc/apt/sources.list.d/influxdb.list
apt-get update
apt-get install -y telegraf
sudo usermod -aG video telegraf
setcap 'cap_net_admin,cap_net_raw+ep' $(which ping)

Get a RaspiBlitz specific telegraf.conf from somewhere and put in /etc/telegraf/telegraf.conf

systemctl restart telegraf

[PatrickScheich]

Hi, I would like to contribute to this topic. How can I do so?

I had created a telegraf.conf for my own raspiblitz and a matching grafana dashboard

I'm quite keen with programming, etc... but haven't yet worked on a github project
so I might need some pointers there

kindest regards, patrick

[rootzoll]

@PatrickScheich In the next week we will produce some introduction tutorials on how to start contributing. But basically you can start small with the following:

• get it working on your own raspiblitz
• edit the /mnt/hdd/app-data/custom-installs.sh in a way that it contains all needed setup shell commands so that it will reinstall the service sucessfully if you start your RaspiBlitz with a fresh sd card (update/recovery) - check /home/admin/raspiblitz.recover.log for debug output after recovery.
• then you can can post that shell script code in this issue here .. we can point you to the next steps from there

[PatrickScheich]

Great, I will take a look into this.

Unfortunately telegraf needs a bunch of individual decisions on config settings like

• ip and port of the backend influxdb
• name of the target database
• credentials for the database

I will assume that proper entries will exists in raspiblitz.conf

[rootzoll]

Check if you can set those parameters on the install commands you can add to custom-installs.sh so that they dont need interaction ... in the next step when turning custom-installs.sh into an install script those parameters then get written into the raspiblitz.conf and the install script gets called using those config values during the recover process.

[PatrickScheich]

here we go, it took a bit of time...

attached you find the files that should go to /mnt/hdd/app-data.
plus a snippet for raspiblitz.conf
beside the telegraf part I added some code that autocopies my ssh-key into /root/.ssh/authorized_keys

app-data-for-custom-install.zip

The Grafana dashboards are a bit tricky
In my installation they are working fine, but they are not yet generic
(with proper variable setup for datasource name and hostname of the raspiblitz)
I surely can provide the JSON model for reference if helpful. Let me know.

[PatrickScheich]

Screenshots of Grafana dashboards

Quick Summary:

System and services uptime and connections/peers:

Blockchain and Mempool:

Basic Computer Metrics:

[frennkie]

Coming along nicely... @PatrickScheich your plan is to run the influxdb and Grafana locally on the RaspiBlitz, correct?

I already have a online VPS running both and I also have more than one active Raspiblitz. So it would be great it you take this into consideration for the Grafana dashboard.. 😊

I surely can provide the JSON model for reference if helpful. Let me know.

Having a nice would definitely be great.

[PatrickScheich]

your plan is to run the influxdb and Grafana locally on the RaspiBlitz, correct?

@frennkie no, only telegraf will run on the raspiblitz for metric collection.
influxDB and Grafana will be located elsewhere

[frennkie]

Can you share the telegraf config file and the Grafana JSON?

[PatrickScheich]

sure,

• the telegraf.conf is in the zip file
• the grafana JSON is attached here grafana.json.txt

[frennkie]

1. You hardcoded your node name - this should be a variable (very easy to do - see below)

2. You used a panel (in the Quick Summary) that's marked a deprecated in my Grafana installation - would make sense to use the current version.

[frennkie]

Can you check this updated version on your Grafana: grafana.json.txt

I also added the telegraf user to the lndadmin group using:

sudo usermod telegraf -a -G lndadmin

and changed the lnddir here:

[[inputs.exec]]
  interval                    = "60s"
  commands                    = ["/usr/local/bin/lncli --lnddir=/mnt/hdd/app-data/lnd getinfo" ]
  name_override               = "ln-info"
  data_format                 = "json"

[frennkie]

The installation of telegraf that you put into custom-installs.sh would be best go into a bonus script in the ~/config.scripts directory. There are many examples (and now there is even a Youtube video explaining this) how to do it. Could you work on a PR?

[PatrickScheich]

Could you work on a PR?

Hmm, I had not worked with this feature of github yet.
I used to work locally with git and VSC but have no idea I how to collaborate

(and now there is even a Youtube video explaining this)

Maybe you could point me to the video, I will check it first

There are many examples

can you name some, I will look into them and try to adopt

thanks for the hint with Grafana variables

[frennkie]

https://youtu.be/ZVtZepV3OfM

This was recorded this week.

Regarding git and GitHub ... There are many resources... Just try one or two tutorials and see what works best for you. Definitely a must have skill - IMHO.. 😉

[PatrickScheich]

@frennkie I opened the PR

[rootzoll]

OK I merged the PR into dev for upcomming v1.7 release :) @PatrickScheich & @frennkie I am a complete noob to telegraf .. so if one of you can test this on RC1 release (targeting this weekend) or even demo as part of the lightning hacksprint this weekend please feel invited - all info on the event: https://wiki.fulmo.org/wiki/Lightning_HackSprint_March_2021

The entrypoint to test this with the coming v1.7RC1 would be by config script:
/home/admin/config.scripts/bonus.telegraf.sh

[rootzoll]

OK .. so the install looks good. Closing issue.

[PatrickScheich]

Hi, I had been a bit busy lately so I had tested the v1.7.0RC2 yesterday. It the "telegraf" part looks fine for me. After updating in made sure that /mnt/hdd/raspiblitz.conf contains the required telegraf entries. Then the installation went smoothly with sudo /home/admin/config.scripts/ bonus.telegraf.sh on

…

On Sun, Apr 18, 2021 at 8:11 PM rootzoll ***@***.***> wrote: Closed #1369 <#1369>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL54MKPSFZOLWIK3JD6SPYDTJMOGJANCNFSM4PCNAHAA> .