Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ZC docs and tests for manual installation/configuration #3700

Merged
merged 8 commits into from
Oct 4, 2023
Merged

Update ZC docs and tests for manual installation/configuration #3700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a3a3e99
Update docs
jeffreyc-splunk Sep 25, 2023
991d743
Update splunk-otel-js to v2.4.2
jeffreyc-splunk Sep 26, 2023
583dc76
Combine sample systemd config files
jeffreyc-splunk Sep 25, 2023
e1cbb0a
Install splunk-otel-js globally for tests
jeffreyc-splunk Sep 26, 2023
a510947
Update deps installation for node tests
jeffreyc-splunk Sep 27, 2023
dc8fd14
Update splunk-otel-js to v2.4.4
jeffreyc-splunk Oct 2, 2023
50da96b
Update linux-manual.md
aurbiztondo-splunk Oct 3, 2023
32d1fdd
Update test for centos/oraclelinux 7 arm64
jeffreyc-splunk Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 49 additions & 53 deletions docs/getting-started/linux-manual.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ installed on x86_64/amd64 platforms.
apt-get update
apt-get install -y splunk-otel-collector

# Optional: install Splunk OpenTelemetry Auto Instrumentation for Java
# Optional: install Splunk OpenTelemetry Auto Instrumentation
apt-get install -y splunk-otel-auto-instrumentation
```
- RPM with `yum`:
Expand All @@ -87,7 +87,7 @@ installed on x86_64/amd64 platforms.

yum install -y splunk-otel-collector

# Optional: install Splunk OpenTelemetry Auto Instrumentation for Java
# Optional: install Splunk OpenTelemetry Auto Instrumentation
yum install -y splunk-otel-auto-instrumentation
```
- RPM with `dnf`:
Expand All @@ -105,7 +105,7 @@ installed on x86_64/amd64 platforms.

dnf install -y splunk-otel-collector

# Optional: install Splunk OpenTelemetry Auto Instrumentation for Java
# Optional: install Splunk OpenTelemetry Auto Instrumentation
dnf install -y splunk-otel-auto-instrumentation
```
- RPM with `zypper`:
Expand All @@ -123,14 +123,14 @@ installed on x86_64/amd64 platforms.

zypper install -y splunk-otel-collector

# Optional: install Splunk OpenTelemetry Auto Instrumentation for Java
# Optional: install Splunk OpenTelemetry Auto Instrumentation
zypper install -y splunk-otel-auto-instrumentation
```
1. See the [Collector Debian/RPM Post-Install
Configuration](#collector-debianrpm-post-install-configuration) section.
1. If the optional Splunk OpenTelemetry Auto Instrumentation for Java package
was installed, see the [Auto Instrumentation Post-Install
Configuration](#auto-instrumentation-post-install-configuration) section.
1. If the optional Splunk OpenTelemetry Auto Instrumentation package was
installed, see the [Auto Instrumentation Post-Install Configuration](
#auto-instrumentation-post-install-configuration) section.
1. If log collection is required, see the [Fluentd](#fluentd) section.
1. To upgrade the Collector, run the following commands:
- Debian:
Expand Down Expand Up @@ -165,10 +165,9 @@ installed on x86_64/amd64 platforms.
sudo apt-get update
sudo apt-get install --only-upgrade splunk-otel-auto-instrumentation
```
**Note:** You may be prompted to keep or overwrite the configuration file
at `/usr/lib/splunk-instrumentation/instrumentation.conf`. Choosing to
overwrite will revert this file to the default file provided by the new
package.
**Note:** You may be prompted to keep or overwrite the configuration files
in the `/etc/splunk/zeroconfig` directory. Choosing to overwrite will
revert this file to the default file provided by the new package.
- RPM:
- `yum`
```sh
Expand All @@ -181,8 +180,12 @@ installed on x86_64/amd64 platforms.
- `zypper`
```sh
sudo zypper refresh
sudo zypper update splunk-otel-instrumentation
sudo zypper update splunk-otel-auto-instrumentation
```
**Note:** If the default configuration files in `/etc/splunk/zeroconfig`
have been modified after initial installation, the existing files will be
preserved and the files from the new package may be installed
with a `.rpmnew` extension.

#### Collector Debian/RPM Packages

Expand Down Expand Up @@ -317,54 +320,47 @@ and install it with the following commands (requires `root` privileges).
```sh
sudo dpkg -i <path to splunk-otel-auto-instrumentation deb>
```
**Note:** You may be prompted to keep or overwrite the configuration file
at `/usr/lib/splunk-instrumentation/instrumentation.conf`. Choosing to
overwrite will revert this file to the default file provided by the new
package.
**Note:** You may be prompted to keep or overwrite the configuration files
in the `/etc/splunk/zeroconfig` directory. Choosing to overwrite will
revert this file to the default file provided by the new package.
- RPM
```sh
sudo rpm -Uvh <path to splunk-otel-auto-instrumentation rpm>
```
**Note:** If the default configuration files in `/etc/splunk/zeroconfig`
have been modified after initial installation, the existing files will be
preserved and the files from the new package may be installed
with a `.rpmnew` extension.

#### Auto Instrumentation Post-Install Configuration

Choose one of the following methods to activate and configure Splunk
OpenTelemetry Auto Instrumentation ***globally*** with either the provided
`libsplunk.so` shared object library or sample `systemd` drop-in files. To
activate and configure auto instrumentation for individual services or
applications, see
[Instrument back-end applications to send spans to Splunk APM](
https://docs.splunk.com/Observability/gdi/get-data-in/application/application.html).

1. Preload method
- The `/usr/lib/splunk-instrumentation/libsplunk.so` shared object library
can be added to the [`/etc/ld.so.preload`](
https://man7.org/linux/man-pages/man8/ld.so.8.html#FILES) file to activate
auto instrumentation for ***all*** supported processes.
- The `/usr/lib/splunk-instrumentation/instrumentation.conf` configuration
file can be configured for resource attributes and other supported
parameters. By default, this file will contain the `java_agent_jar`
parameter set to the path of the installed [Java Instrumentation Agent](
https://github.com/signalfx/splunk-otel-java)
(`/usr/lib/splunk-instrumentation/splunk-otel-javaagent.jar`).
- See [Linux Java Auto Instrumentation](../../instrumentation/libsplunk.md)
for more details.

2. `Systemd` method
- The sample `systemd` drop-in files in the
`/usr/lib/splunk-instrumentation/examples/systemd/` directory can be
configured and copied to the host's [`systemd` configuration
directory](
https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html),
for example `/usr/lib/systemd/system.conf.d/`, to activate and
configure auto instrumentation for ***all*** supported applications
running as `systemd` services.
- See [Splunk OpenTelemetry Zero Configuration Auto Instrumentation for
Systemd](../../instrumentation/systemd.md) for more details.

**Note:** After installation/upgrade or any configuration changes, reboot the
system or restart the application(s) on the host for automatic instrumentation
to take effect and/or to source the updated values.
The `splunk-otel-auto-instrumentation` deb/rpm package installs and supports
configuration for the following Auto Instrumentation agents:

- [Java](https://docs.splunk.com/Observability/gdi/get-data-in/application/java/get-started.html)
- [Node.js](https://docs.splunk.com/Observability/en/gdi/get-data-in/application/nodejs/get-started.html)

To manually activate and configure the Auto Instrumentation agents:
1. Check agent compatibility and requirements:
- [Java](https://docs.splunk.com/Observability/gdi/get-data-in/application/java/java-otel-requirements.html)
- [Node.js](https://docs.splunk.com/Observability/en/gdi/get-data-in/application/nodejs/nodejs-otel-requirements.html)
2. If Auto Instrumentation for Node.js is required, install the provided
`/usr/lib/splunk-instrumentation/splunk-otel-js.tgz` Node.js package with
`npm`. For example:
```sh
sudo npm install --global /usr/lib/splunk-instrumentation/splunk-otel-js.tgz
```
> **Notes:**
> - Ensure that all Node.js applications/services to be instrumented have
> access to the installation path of the Node.js package.
> - On `arm64` architectures, it may be necessary to build/compile the
> Node.js package when installing with `npm`. Ensure that any required
> tools/libraries are installed on these systems, for example,
> `sudo apt-get install build-essential` on Debian-based systems or
> `sudo yum groupinstall 'Development Tools'` on RPM-based systems.
3. See [Activation and Configuration](
../../instrumentation/README.md#activation-and-configuration) for supported
methods and options.

#### Fluentd

Expand Down
147 changes: 113 additions & 34 deletions instrumentation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,50 +1,129 @@
# Splunk OpenTelemetry Zero Configuration Auto Instrumentation for Linux

**Splunk OpenTelemetry Zero Configuration Auto Instrumentation for Linux** (`splunk-otel-auto-instrumentation`)
Debian/RPM package installs Splunk OpenTelemetry Auto Instrumentation agent(s), the `libsplunk.so` shared object
library, and sample configuration files to automatically instrument applications and services to capture and report
distributed traces and metrics to the [Splunk OpenTelemetry Collector](
The **Splunk OpenTelemetry Zero Configuration Auto Instrumentation for Linux** Debian/RPM package
(`splunk-otel-auto-instrumentation`) installs Splunk OpenTelemetry Auto Instrumentation agents, the `libsplunk.so`
shared object library, and default/sample configuration files to automatically instrument applications and services to
capture and report distributed traces and metrics to the [Splunk OpenTelemetry Collector](
https://docs.splunk.com/Observability/gdi/opentelemetry/opentelemetry.html), and then on to [Splunk APM](
https://docs.splunk.com/Observability/apm/intro-to-apm.html).

Currently, `splunk-otel-auto-instrumentation` installs and supports configuration of the following Auto Instrumentation
agent(s):
The `splunk-otel-auto-instrumentation` deb/rpm package installs and supports configuration of the following Auto
Instrumentation agents:

- [Java](https://docs.splunk.com/Observability/gdi/get-data-in/application/java/get-started.html)
- [Node.js](https://docs.splunk.com/Observability/en/gdi/get-data-in/application/nodejs/get-started.html)

For other languages or if the `splunk-otel-auto-instrumentation` deb/rpm package is not applicable for the target host
or applications/services, see [Instrument back-end applications to send spans to Splunk APM](
https://docs.splunk.com/Observability/en/gdi/get-data-in/application/application.html).

## Prerequisites/Requirements

- Check agent compatibility and requirements:
- [Java](https://docs.splunk.com/Observability/gdi/get-data-in/application/java/java-otel-requirements.html)
- [Node.js](https://docs.splunk.com/Observability/en/gdi/get-data-in/application/nodejs/nodejs-otel-requirements.html)
- [Install and configure](https://docs.splunk.com/Observability/gdi/opentelemetry/install-linux.html) the Splunk
OpenTelemetry Collector.
- Debian or RPM based Linux distribution (amd64/x86_64 or arm64/aarch64).

## Zero Configuration Options

The following options are supported to enable the installed Auto Instrumentation agent(s):

- **[`libsplunk.so`](./libsplunk.md)**: The provided `/usr/lib/splunk-instrumentation/libsplunk.so` shared object
library can be added to the `/etc/ld.so.preload` file to enable and configure Auto Instrumentation for ***all***
supported processes on the system, or it can be assigned to the `LD_PRELOAD` environment variable for specific
applications/services. Configuration of the installed agent(s) is supported by the
[`/usr/lib/splunk-instrumentation/instrumentation.conf`](./libsplunk.md#configuration-file) file.

- **[`systemd`](./systemd.md)**: The provided `systemd` drop-in file(s) in the
`/usr/lib/splunk-instrumentation/examples/systemd/` directory can be copied to the host's `systemd` configuration
directory, e.g. `/usr/lib/systemd/system.conf.d/`, to enable and configure Auto Instrumentation for ***all***
supported applications running as `systemd` services. Configuration of the installed agent(s) is supported by
modifying these files or adding custom drop-in files with the desired environment variables.

Alternatively, [manually install and configure](
https://docs.splunk.com/Observability/gdi/get-data-in/application/application.html)
Auto Instrumentation if neither option is appropriate for the target host or application.

> ### Notes
>
> 1. To prevent conflicts and duplicate traces/metrics, only one option should be enabled on the target system.
> 2. The configuration files and the options defined within are only applicable for the respective option that is
> configured. For example, `/usr/lib/splunk-instrumentation/instrumentation.conf` is only applicable with
> `libsplunk.so`, and the systemd drop-in file is not applicable for `libsplunk.so`.
> 3. The [`splunk.linux-autoinstr.executions`](./libsplunk.md#disable_telemetry-optional) telemetry
> metric is currently only provided with `libsplunk.so`.
## Installation

### Manual

1. [Install and configure](https://docs.splunk.com/Observability/gdi/opentelemetry/install-linux.html) the Splunk
OpenTelemetry Collector
2. [Install](../docs/getting-started/linux-manual.md#auto-instrumentation-debianrpm-packages) the
`splunk-otel-auto-instrumentation` deb/rpm package
3. If Auto Instrumentation for Node.js is required, [install](
../docs/getting-started/linux-manual.md#auto-instrumentation-post-install-configuration) the provided
`/usr/lib/splunk-instrumentatgion/splunk-otel-js.tgz` Node.js package with `npm`.
4. [Activate and configure](#activation-and-configuration) Auto Instrumentation with the supported methods and options.

## Activation and Configuration

The following methods are supported to manually activate and configure Auto Instrumentation after installation of the
`splunk-otel-auto-instrumentation` deb/rpm package (requires `root` privileges):

- [System-wide](#system-wide)
- [`Systemd` services only](#systemd-services-only)

> **Note**: To prevent conflicts and duplicate traces/metrics, only one method should be activated on the target system.

### System-wide

1. Add the path of the provided `/usr/lib/splunk-instrumentation/libsplunk.so` shared object library to the
[`/etc/ld.so.preload`](https://man7.org/linux/man-pages/man8/ld.so.8.html#FILES) file to activate Auto
Instrumentation for ***all*** supported processes on the system. For example:
```
echo /usr/lib/splunk-instrumentation/libsplunk.so >> /etc/ld.so.preload
```
2. The default configuration files in the `/etc/splunk/zeroconfig` directory includes the required environment variables
to activate the respective agents with the default options:
- `/etc/splunk/zeroconfig/java.conf`:
```
JAVA_TOOL_OPTIONS=-javaagent:/usr/lib/splunk-instrumentation/splunk-otel-javaagent.jar
```
- `/etc/splunk/zeroconfig/node.conf`:
```
NODE_OPTIONS=-r @splunk/otel/instrument
```
Configuration of the respective agents is supported by the adding/updating the following environment variables in
each of these files (***any environment variable not in this list will be ignored***):
- `OTEL_EXPORTER_OTLP_ENDPOINT`
- `OTEL_RESOURCE_ATTRIBUTES`
- `OTEL_SERVICE_NAME`
- `SPLUNK_METRICS_ENABLED`
- `SPLUNK_PROFILER_ENABLED`
- `SPLUNK_PROFILER_MEMORY_ENABLED`

Check the following for details about these environment variables and default values:
- [Java](https://docs.splunk.com/Observability/en/gdi/get-data-in/application/java/configuration/advanced-java-otel-configuration.html)
- [Node.js](https://docs.splunk.com/Observability/en/gdi/get-data-in/application/nodejs/configuration/advanced-nodejs-otel-configuration.html)
3. Reboot the system or restart the applications/services for any changes to take effect. The `libsplunk.so` shared
object library will then be preloaded for all subsequent processes and inject the environment variables from the
`/etc/splunk/zeroconfig` configuration files for Java and Node.js processes.

### `Systemd` services only

> **Note**: The following steps utilize a sample `systemd` drop-in file to activate/configure the provided agents for
> all `systemd` services via default environment variables. `Systemd` supports many options, methods, and paths for
> configuring environment variables at the system level or for individual services, and are not limited to the steps
> below. Before making any changes, it is recommended to consult the documentation specific to your Linux distribution
> or service, and check the existing configurations of the system and individual services for potential conflicts or to
> override an environment variable for a particular service. For general details about `systemd`, see the
> [`systemd` man page](https://www.freedesktop.org/software/systemd/man/index.html).

1. Copy the provided sample `systemd` drop-in file
`/usr/lib/splunk-instrumentation/examples/systemd/00-splunk-otel-auto-instrumentation.conf` to the host's `systemd`
[drop-in configuration directory](https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html) to
activate Auto Instrumentation for ***all*** supported applications running as `systemd` services. For example:
```
mkdir -p /usr/lib/systemd/system.conf.d/ && cp /usr/lib/splunk-instrumentation/examples/systemd/00-splunk-otel-auto-instrumentation.conf /usr/lib/systemd/system.conf.d/
```
This file includes the required environment variables to activate the respective agents with the default options:
- Java:
```
DefaultEnvironment="JAVA_TOOL_OPTIONS=-javaagent:/usr/lib/splunk-instrumentation/splunk-otel-javaagent.jar"
```
- Node.js:
```
DefaultEnvironment="NODE_OPTIONS=-r @splunk/otel/instrument"
```
2. To configure the activated agents, add/update [`DefaultEnvironment`](
https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html#DefaultEnvironment=) within the target file
from the previous step for the desired environment variables. For example:
```
cat <<EOH >> /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf
DefaultEnvironment="OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317"
DefaultEnvironment="OTEL_RESOURCE_ATTRIBUTES=deployment.environment=my_deployment_environment"
DefaultEnvironment="OTEL_SERVICE_NAME=my_service_name"
DefaultEnvironment="SPLUNK_METRICS_ENABLED=true"
DefaultEnvironment="SPLUNK_PROFILER_ENABLED=true"
DefaultEnvironment="SPLUNK_PROFILER_MEMORY_ENABLED=true"
EOH
```
Check the following for all supported environment variables and default values:
- [Java](https://docs.splunk.com/Observability/en/gdi/get-data-in/application/java/configuration/advanced-java-otel-configuration.html)
- [Node.js](https://docs.splunk.com/Observability/en/gdi/get-data-in/application/nodejs/configuration/advanced-nodejs-otel-configuration.html)
3. Reboot the system, or run `systemctl daemon-reload` and then restart the applicable `systemd` services for any
changes to take effect.
Loading