- Overview
- Installing Gomon
- Employing Prometheus, Loki, and Grafana
- Putting it all together
- Visualize
- Notices
The gomon command starts itself as a server that monitors its system. Gomon periodically measures the state of the system, including its CPU, memory, filesystems, network interfaces, and processes. Gomon also observes system logs and the system's management of files and processes. Hence, gomon's processing consists of two fundamental operations: measurement and observation.
What gomon measures of the system is performed through kernel interfaces that report the state of resources of the system: the CPU, the memory, the filesystems, the network interfaces, and the processes.
What gomon observes of the system are reports of events captured by the logging, file management, and process management subsystems. While gomon cannot observe the events directly, it assumes that the subsystems' reporting is timely. A potential enhancement to gomon would be for it to initiate periodically its own log, file, and process events to sample the lag between initiation and reporting of events. This could dovetail nicely with gomon's measurement operation.
Gomon records its measurements and observations and reports them in a message stream. By default, gomon streams these messages as JSON objects to standard out.
Analysis of gomon's message stream may yield insights into the system's operations through discovery of event patterns and interrelationships and what these signify about the condition and operation of the system. An excellent application for stream analysis and visualization is Grafana. Together with Prometheus to record the measurements, and Loki to record the observations, Grafana can chart the measurements and report the observations. Look in the gomon/assets folder for sample configuration files for Prometheus and Grafana to enable this scenario.
The inspection of process relationships is also available if Graphviz is installed. Gomon can use Graphviz to render a web view of the inter-process and remote host connections of the system.
To set up Gomon, first download and install Go. Then proceed with the following Terminal commands to download and install Gomon. (On macOS, you may be prompted to install the Xcode Command Line Tools.)
go install github.com/zosmac/gomon/cmd/gomon@latestAs performing several of the measurements and observations requires root authority, use sudo to invoke gomon:
sudo gomon -prettyAlternatively, assign the executable to root and grant it setuid permission:
sudo chown 0:0 $(which gomon)
sudo chmod u+s $(which gomon)
gomon -prettyGomon periodically (default every 15s) makes system measurements and gathers observations, consolidating these into a stream that it writes to standard out as JSON objects.
To view all the flags that the gomon command accepts for configuration, enter gomon -help. To see all the metrics that Gomon captures, enter gomon -document.
Follow these steps for deploying the three servers that record measurements and observations to facilitate visualization. Building these servers depends on two Javascript applications, the Node runtime and the Yarn classic package manager. Download the current Node installer for your system and install. Then use the Node Package Manager (npm) to install Yarn:
sudo npm install -global yarnPrometheus is an open-source systems monitoring server for scraping and storing time-series data. At startup, Gomon opens a /metrics endpoint for Prometheus to collect Gomon’s system resource measurement stream.
To install Prometheus, select an appropriate binary from the Prometheus download page for your Operating System and Architecture, download, and install.
To enable collection, open the prometheus.yml configuration file for the Prometheus deployment and add the following to scrape_configs (a sample configuration file is in assets/prometheus.yml):
- job_name: "gomon"
scheme: https
tls_config:
ca_file: /etc/ssh/cert.pem
static_configs:
- targets: ["localhost:1234"]Loki is an open-source log aggregation server. Via Gomon’s HTTP POSTs to the /loki/api/v1/push endpoint, Loki can receive the observations. To install Loki, create a loki folder, select appropriate loki and promtail binaries from the list of Assets on the Loki releases page for your platform, and download. Each binary also requires a configuration file; follow the instructions on the Loki installation page to copy these to the loki folder.
cd ${LOKI_DIR}
unzip =(curl -L "https://github.com/grafana/loki/releases/latest/download/loki-$(go env GOOS)-$(go env GOARCH).zip")
unzip =(curl -L "https://github.com/grafana/loki/releases/latest/download/promtail-$(go env GOOS)-$(go env GOARCH).zip")
chmod a+x loki-$(go env GOOS)-$(go env GOARCH) promtail-$(go env GOOS)-$(go env GOARCH)
curl -O -L "https://raw.githubusercontent.com/grafana/loki/main/cmd/loki/loki-local-config.yaml"
curl -O -L "https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml"Grafana is an open-source server that can query and display Prometheus time-series data and Loki aggregated logs. Together these three servers provide an excellent front-end for Gomon.
To install Grafana, select an appropriate binary from the Grafana download page for your platform, download, and install.
Add the data sources for Prometheus and Loki to Grafana
Note: while the Data Source's Settings Panel shows the default data source URL, you must still type it into the URL field. Then, scroll down and select Save & test.
Note: while the Data Source's Settings Panel shows the default data source URL, you must still type it into the URL field. Then, scroll down and select Save & test.
Install the Gomon Dashboard to Grafana
curl ${GRAFANA_CRED} -X POST -i -w "\n" -H "Content-Type: application/json" -T ${GOMON_DIR}/assets/dashboard.json "http://localhost:3000/api/dashboards/db"cd ${PROMETHEUS_DIR}
./prometheus >prometheus.log 2>&1 &cd ${LOKI_DIR}
./loki-$(go env GOOS)-$(go env GOARCH) -config.file loki-local-config.yaml >loki.log 2>&1 &cd ${GRAFANA_DIR}
bin/$(go env GOOS)-$(go env GOARCH)/grafana-server web >grafana.log 2>&1 &sudo gomon -pretty -port 1234
If Graphviz is installed, Gomon can render a node graph of the inter-process and remote host connections via the /gomon endpoint:
To download and install Graphviz, select a stable release, download its tar file, build, and install.
Note: gomon specifies -Tsvgz to the dot command. Ensure that the zlib development library is installed on your system, e.g. on Ubuntu sudo apt install zlib1g-dev, on Fedora sudo yum install zlib devel.
tar xzvf =(curl -L "https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/7.1.0/graphviz-7.1.0.tar.gz")
cd graphviz-7.1.0
./configure
make
sudo make installNote: installing from the git repository (i.e. git clone https://gitlab.com/graphviz/graphviz/) requires pre-configuration with autogen.sh, which in turn requires GNU autoconf, automake, and libtool. Find details at https://graphviz.org/download/source/#git-repos
Copyright © 2021-2023 The Gomon Project.