Skip to content

Commit

Permalink
Deprecate gRPC-Plugin support (#5388)
Browse files Browse the repository at this point in the history 
## Which problem is this PR solving?
- Part of [#4647 ](#4647)

## Description of the changes
- Remove sidecar plugin support based on hashicorp go-plugin library.



## Checklist
- [x] I have read
https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
- [x] I have signed all commits
- [ ] I have added unit tests for the new functionality
- [x] I have run lint and test steps successfully
  - for `jaeger`: `make lint test`
  - for `jaeger-ui`: `yarn lint` and `yarn test`

---------

Signed-off-by: Ashutosh Srivastava <ashutosh3002@gmail.com>
Signed-off-by: Yuri Shkuro <github@ysh.us>
Signed-off-by: Yuri Shkuro <yurishkuro@users.noreply.github.com>
Co-authored-by: Yuri Shkuro <yurishkuro@users.noreply.github.com>
Co-authored-by: Yuri Shkuro <github@ysh.us>
  • Loading branch information
@h4shk4t @yurishkuro
3 people authored May 11, 2024
1 parent 299d942 commit e9c819d
Show file tree
Hide file tree
Showing 20 changed files with 59 additions and 777 deletions.
1 change: 0 additions & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ cover.html
vendor/
examples/hotrod/hotrod
examples/hotrod/hotrod-*
examples/memstore-plugin/memstore-plugin
cmd/all-in-one/all-in-one-*
cmd/agent/agent
cmd/agent/agent-*
Expand Down
1 change: 0 additions & 1 deletion Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,6 @@ badger-storage-integration-test:

.PHONY: grpc-storage-integration-test
grpc-storage-integration-test:
(cd examples/memstore-plugin/ && go build .)
STORAGE=grpc $(MAKE) storage-integration-test

# this test assumes STORAGE environment variable is set to elasticsearch|opensearch
Expand Down
6 changes: 0 additions & 6 deletions examples/memstore-plugin/README.md
View file

This file was deleted.

78 changes: 0 additions & 78 deletions examples/memstore-plugin/main.go
View file

This file was deleted.

160 changes: 23 additions & 137 deletions plugin/storage/grpc/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,33 +1,32 @@
gRPC Storage Plugins
====================
gRPC Remote Storage Plugins
===========================

Update (Jan 2022): as of Jaeger v1.30, the gRPC storage extension can be implemented as a remote gRPC server, in addition to the gRPC plugin architecture described below. The remote server needs to implement the same `storage_v1` gRPC interfaces defined in `plugin/storage/grpc/proto/`.
Update (May 2024): as of v1.58, Jaeger will no longer support grpc-sidecar plugin model. Only gRPC Remote Storage API is suppored.

gRPC Storage Plugins currently use the [Hashicorp go-plugin](https://github.com/hashicorp/go-plugin). This requires the
implementer of a plugin to develop the "server" side of the go-plugin system. At a high level this looks like:
In order to support an ecosystem of different backend storage implementations, Jaeger supports a gRPC-based Remote Strorage API. The custom backend storage solutions need to implement `storage_v1` gRPC interfaces defined in `plugin/storage/grpc/proto/`.

```
+----------------------------------+ +-----------------------------+
| | | |
| +-------------+ | unix-socket | +-------------+ |
| +-------------+ | gRPC | +-------------+ |
| | | | | | | |
| jaeger-component | grpc-client +----------------------> grpc-server | plugin-impl |
| jaeger-component | grpc-client +----------------------> grpc-server | custom impl |
| | | | | | | |
| +-------------+ | | +-------------+ |
| | | |
+----------------------------------+ +-----------------------------+
parent process child sub-process
Jaeger official components Custom backend implementation
```

Implementing a plugin
----------------------

Although the instructions below are limited to Go, plugins can be implemented any language. Languages other than
Go would implement a gRPC server using the `storage_v1` proto interfaces. The `proto` file can be found in `plugin/storage/grpc/proto/`.
To generate the bindings for your language you would use `protoc` with the appropriate `xx_out=` flag. This is detailed
To generate the bindings for your language you would use `protoc` with the appropriate `xx_out=` flag. This is detailed
in the [protobuf documentation](https://developers.google.com/protocol-buffers/docs/tutorials) and you can see an example of
how it is done for Go in the top level Jaeger `Makefile`.
how it is done for Go in the top level Jaeger `Makefile`.

The easiest way to generate the gRPC storage plugin bindings is to use [Docker Protobuf](https://github.com/jaegertracing/docker-protobuf/) which is a lightweight `protoc` Docker image containing the dependencies needed to generate code for multiple languages. For example, one can generate bindings for C# on Windows with Docker for Windows using the following steps:
1. First clone the Jaeger github repo to a folder (e.g. `c:\source\repos\jaeger`):
Expand All @@ -40,79 +39,23 @@ $ git clone https://github.com/jaegertracing/jaeger.git c:\source\repos\jaeger
```
$ git submodule update --init --recursive
```
3. Then execute the following Docker command which mounts the local directory `c:\source\repos\jaeger` to the directory `/jaeger` in the Docker container and then executes the `jaegertracing/protobuf:0.2.0` command. This will create a file called `Storage.cs` in your local Windows folder `c:\source\repos\jaeger\code` containing the gRPC Storage Plugin bindings.
3. Then execute the following Docker command which mounts the local directory `c:\source\repos\jaeger` to the directory `/jaeger` in the Docker container and then executes the `jaegertracing/protobuf:0.2.0` command. This will create a file called `Storage.cs` in your local Windows folder `c:\source\repos\jaeger\code` containing the gRPC Storage API bindings.
```
$ docker run --rm -u 1000 -v/c/source/repos/jaeger:/jaeger -w/jaeger \
jaegertracing/protobuf:0.2.0 "-I/jaeger -Iidl/proto/api_v2 -I/usr/include/github.com/gogo/protobuf -Iplugin/storage/grpc/proto --csharp_out=/jaeger/code plugin/storage/grpc/proto/storage.proto"
```

There are instructions on implementing a `go-plugin` server for non-Go languages in the
[go-plugin non-go guide](https://github.com/hashicorp/go-plugin/blob/master/docs/guide-plugin-write-non-go.md).
Take note of the required [health check service](https://github.com/hashicorp/go-plugin/blob/master/docs/guide-plugin-write-non-go.md#3-add-the-grpc-health-checking-service).

A Go plugin is a standalone application which calls `grpc.Serve(&pluginServices)` in its `main` function, where the `grpc` package
is `github.com/jaegertracing/jaeger/plugin/storage/grpc`.

```go
package main

import (
"flag"
"github.com/jaegertracing/jaeger/plugin/storage/grpc"
)

func main() {
var configPath string
flag.StringVar(&configPath, "config", "", "A path to the plugin's configuration file")
flag.Parse()

plugin := myStoragePlugin{}

grpc.Serve(&shared.PluginServices{
Store: plugin,
ArchiveStore: plugin,
})
}
```

Note that `grpc.Serve` is called as the final part of the main. This should be called after you have carried out any necessary
setup for your plugin, as once running Jaeger may start calling to read/write spans straight away. You could defer
setup until the first read/write but that could make the first operation slow and also lead to racing behaviours.

A plugin must implement the StoragePlugin interface of:

```go
type StoragePlugin interface {
SpanReader() spanstore.Reader
SpanWriter() spanstore.Writer
DependencyReader() dependencystore.Reader
}
```

As your plugin will be dependent on the protobuf implementation within Jaeger you will likely need to `vendor` your
dependencies, you can also use `go.mod` to achieve the same goal of pinning your plugin to a Jaeger point in time.
An example of a Go binary that implements Remote Storage API can be found in `cmd/remote-storage`. That specific binary does not implement a custom backend, instead it supports the same backend implementations as available directly in Jaeger, but it makes them accessible via a Remote Storage API (and is being used in the integration tests).

A simple plugin which uses the memstore storage implementation can be found in the `examples` directory of the top level
of the Jaeger project.
The API consists of several gRPC services:
* `SpanReaderPlugin` - used for querying the data
* `SpanWriterPlugin` - used for writing data
* (optional) `StreamingSpanWriterPlugin` - allows more efficient transmission
* (optional) `ArchiveSpanWriterPlugin` and `ArchiveSpanReaderPlugin` - to support archiving storage
* (optional) `DependenciesReaderPlugin` - for reading service dependencies
* (optional) `PluginCapabilities` - can be interrogated to find out which services an implementation supports

To support archive storage a plugin must implement the ArchiveStoragePlugin interface of:

```go
type ArchiveStoragePlugin interface {
ArchiveSpanReader() spanstore.Reader
ArchiveSpanWriter() spanstore.Writer
}
```

If you don't plan to implement archive storage simply do not fill `ArchiveStore` property of `shared.PluginServices`:

```go
grpc.Serve(&shared.PluginServices{
Store: plugin,
})
```

The plugin framework supports writing spans via gRPC stream, instead of unary messages. Streaming writes can improve throughput and decrease CPU load (see benchmarks in Issue #3636). The plugin needs to implement `StreamingSpanWriter` interface and indicate support via the `streamingSpanWriter` flag in the `Capabilities` response.
The API supports writing spans via gRPC stream, instead of unary messages. Streaming writes can improve throughput and decrease CPU load (see benchmarks in Issue #3636). The backend needs to implement `StreamingSpanWriterPlugin` service and indicate support via the `streamingSpanWriter` flag in the `Capabilities` response.

Note that using the streaming spanWriter may make the collector's `save_by_svr` metric inaccurate, in which case users will need to pay attention to the metrics provided by the plugin.

Expand Down Expand Up @@ -141,72 +84,15 @@ func TestJaegerStorageIntegration(t *testing.T) {
```
For more details, refer to one of the following implementations.

1. [grpc-plugin](https://github.com/jaegertracing/jaeger/blob/cbceceb1e0cc308cdf0226b1fa19b9c531a3a2d3/plugin/storage/integration/grpc_test.go#L189-L203)
2. [jaeger-clickhouse](https://github.com/jaegertracing/jaeger-clickhouse/blob/798c568c1e1a345536f35692fca71196a796811e/integration/grpc_test.go#L88-L107)
3. [Timescale DB via Promscale](https://github.com/timescale/promscale/blob/ccde8accf5205450891e805e23566d9a11dbf8d3/pkg/tests/end_to_end_tests/jaeger_store_integration_test.go#L79-L97)

Running with a plugin
---------------------
A plugin can be run using the `all-in-one` application within the top level `cmd` package of the Jaeger project. To do this
an environment variable must be set to tell the `all-in-one` application to use the gRPC plugin storage:
`export SPAN_STORAGE_TYPE="grpc-plugin"`

Once this has been set then there are two command line flags that can be used to configure the plugin. The first is
`--grpc-storage-plugin.binary` which is required and is the path to the plugin **binary**. The second is
`--grpc-storage-plugin.configuration-file` which is optional and is the path to the configuration file which will be
provided to your plugin as a command line flag. This command line flag is `config`, as can be seen in the code sample
above. An example invocation would be:

```
./all-in-one --grpc-storage-plugin.binary=/path/to/my/plugin --grpc-storage-plugin.configuration-file=/path/to/my/config
```

As well as passing configuration values via the command line through the configuration file it is also possible to use
environment variables. When you invoke `all-in-one` any environment variables that have been set will also be accessible
from within your plugin, this is useful if using Docker.

Logging
-------
In order for Jaeger to include the log output from your plugin you need to use `hclog` (`"github.com/hashicorp/go-hclog"`).
The plugin framework will only include any log output created at the `WARN` or above levels. If you log output in this
way before calling `grpc.Serve` then it will still be included in the Jaeger output.

An example logger instantiation could look like:

```
logger := hclog.New(&hclog.LoggerOptions{
Level: hclog.Warn,
Name: "my-jaeger-plugin",
JSONFormat: true,
})
```

There are more logger options that can be used with `hclog` listed on [godoc](https://godoc.org/github.com/hashicorp/go-hclog#LoggerOptions).

Note: Setting the `Output` option to `os.Stdout` can confuse the `go-plugin` framework and lead it to consider the plugin
errored.
1. [jaeger-clickhouse](https://github.com/jaegertracing/jaeger-clickhouse/blob/798c568c1e1a345536f35692fca71196a796811e/integration/grpc_test.go#L88-L107)
2. [Timescale DB via Promscale](https://github.com/timescale/promscale/blob/ccde8accf5205450891e805e23566d9a11dbf8d3/pkg/tests/end_to_end_tests/jaeger_store_integration_test.go#L79-L97)

Tracing
-------

When `grpc-plugin` is used, it will be running as a separated process, thus context propagation is necessary for inter-process scenarios.

In order to get complete traces containing both `jaeger-component`(s) and the `grpc-plugin`, developers should enable tracing at server-side.
Thus, we can leverage gRPC interceptors,

```golang
grpc.ServeWithGRPCServer(&shared.PluginServices{
Store: memStorePlugin,
ArchiveStore: memStorePlugin,
}, func(options []googleGRPC.ServerOption) *googleGRPC.Server {
return plugin.DefaultGRPCServer([]googleGRPC.ServerOption{
grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor(otelgrpc.WithTracerProvider(tracerProvider))),
grpc.StreamInterceptor(otelgrpc.StreamServerInterceptor(otelgrpc.WithTracerProvider(tracerProvider))),
})
})
```
Jaeger requests to the backend implementation will include standard OTEL tracing headers. The implementation may choose to participate in those traces to allow end to end visibility of Jaeger's own operations (typically only enabled for read requests,
as tracing write requests results in traces for traces and may cause infinite loops).

Refer to `example/memstore-plugin` for more details.

Bearer token propagation from the UI
------------------------------------
Expand Down
Loading

0 comments on commit e9c819d

Please sign in to comment.