tracing: Support dynamically loading tracing libraries. (#2252)

This PR adds support for dynamically loading tracers into Envoy. It will allow authors of OpenTracing compliant tracers to support envoy without requiring envoy to bake their tracers into its build. Tracing libraries can be enabled by adding a section like this to envoy's configuration:

  tracing:
    http:
      name: envoy.dynamic.ot
      config:
        library: /path/to/tracing/library
        config: {
          ....
        }
With this change, Envoy will be able to use Jaeger's new C++ tracing library. Some features this will allow that aren't available when using Jaeger as a collector for Zipkin:

Propagation will work correctly with other services using Jaeger without requiring them to make configuration changes.
Jaeger's different sampling strategies can be used.
More efficient tranports are available.
While it might also make sense sometime in the future to offer a version of Jaeger or other tracers built into Envoy, decoupled plug-in support will allow for quicker on-boarding of tracers and more experimentation in the area without making Envoy's external dependencies become unmanageable.

Notes:

One challenge with using dynamically loaded libraries in Envoy is that the standard c++ library is statically compiled in with the option -static-libstdc++. If plugin libraries are linked to the libstdc++ in a standard way, this will create problems as two versions of the standard library will be loaded and calls to their functions from the plugin will be intermingled between them, breaking many of the standard library functions. The approach I took with Jaeger was to have it also use -static-libstdc++ and then use an export map to ensure that it's version of the standard library is used without any intermingling (see dsohowto for background on the approach). It does mean that there will be multiple copies of libstdc++ running, but from what I've heard this can be made to work. Another approach could be to add a config option to link libstdc++ dynamically.
This depends on opentracing/opentracing-cpp#45 being merged in to opentracing-cpp, but I wanted to get a PR in early to get feedback on the approach in case we need to make changes to that PR.
Risk Level: Medium

Testing:
I have a small example that demoes this functionality using docker and a branch of jaeger.

I also plan to add unit test coverage using a mock tracer library that will be added to opentracing-cpp.

Docs Changes:

envoyproxy/data-plane-api#386

Release Notes:

Included in PR.

Signed-off-by: Ryan Burn <ryan.burn@gmail.com>

RAW_RELEASE_NOTES.md

@@ -63,6 +63,7 @@ final version.
 * Added the ability to pass a URL encoded Pem encoded peer certificate in the x-forwarded-client-cert header.
 * Added support for abstract unix domain sockets on linux. The abstract
   namespace can be used by prepending '@' to a socket path.
+* Added support for dynamically loading a tracer.
 * Added `GEORADIUS_RO` and `GEORADIUSBYMEMBER_RO` to the Redis command splitter whitelist.
 * Added support for trusting additional hops in the X-Forwarded-For request header.
 * Added setting host header value for http health check request.

bazel/repository_locations.bzl

@@ -39,8 +39,8 @@ REPOSITORY_LOCATIONS = dict(
         remote = "https://github.com/grpc/grpc.git",
     ),
     io_opentracing_cpp = dict(
-        commit = "e57161e2a4bd1f9d3a8d3edf23185f033bb45f17",
-        remote = "https://github.com/opentracing/opentracing-cpp", # v1.2.0
+        commit = "f3c1f42601d13504c68e2bc81c60604f0de055dd",
+        remote = "https://github.com/opentracing/opentracing-cpp",
     ),
     com_lightstep_tracer_cpp = dict(
         commit = "6a198acd328f976984699f7272bbec7c8b220f65",

source/common/config/well_known_names.h

@@ -158,6 +158,8 @@ class HttpTracerNameValues {
   const std::string LIGHTSTEP = "envoy.lightstep";
   // Zipkin tracer
   const std::string ZIPKIN = "envoy.zipkin";
+  // Dynamic tracer
+  const std::string DYNAMIC_OT = "envoy.dynamic.ot";
 };
 
 typedef ConstSingleton<HttpTracerNameValues> HttpTracerNames;

source/common/tracing/BUILD

@@ -53,6 +53,20 @@ envoy_cc_library(
     ],
 )
 
+envoy_cc_library(
+    name = "dynamic_opentracing_driver_lib",
+    srcs = [
+        "dynamic_opentracing_driver_impl.cc",
+    ],
+    hdrs = [
+        "dynamic_opentracing_driver_impl.h",
+    ],
+    deps = [
+        ":http_tracer_lib",
+        ":opentracing_driver_lib",
+    ],
+)
+
 envoy_cc_library(
     name = "lightstep_tracer_lib",
     srcs = [

source/common/tracing/dynamic_opentracing_driver_impl.cc

@@ -0,0 +1,38 @@
+#include "common/tracing/dynamic_opentracing_driver_impl.h"
+
+#include "common/common/assert.h"
+
+namespace Envoy {
+namespace Tracing {
+
+DynamicOpenTracingDriver::DynamicOpenTracingDriver(Stats::Store& stats, const std::string& library,
+                                                   const std::string& tracer_config)
+    : OpenTracingDriver{stats} {
+  std::string error_message;
+  opentracing::expected<opentracing::DynamicTracingLibraryHandle> library_handle_maybe =
+      opentracing::DynamicallyLoadTracingLibrary(library.c_str(), error_message);
+  if (!library_handle_maybe) {
+    throw EnvoyException{formatErrorMessage(library_handle_maybe.error(), error_message)};
+  }
+  library_handle_ = std::move(*library_handle_maybe);
+
+  opentracing::expected<std::shared_ptr<opentracing::Tracer>> tracer_maybe =
+      library_handle_.tracer_factory().MakeTracer(tracer_config.c_str(), error_message);
+  if (!tracer_maybe) {
+    throw EnvoyException{formatErrorMessage(tracer_maybe.error(), error_message)};
+  }
+  tracer_ = std::move(*tracer_maybe);
+  RELEASE_ASSERT(tracer_ != nullptr);
+}
+
+std::string DynamicOpenTracingDriver::formatErrorMessage(std::error_code error_code,
+                                                         const std::string& error_message) {
+  if (error_message.empty()) {
+    return fmt::format("{}", error_code.message());
+  } else {
+    return fmt::format("{}: {}", error_code.message(), error_message);
+  }
+}
+
+} // namespace Tracing
+} // namespace Envoy

source/common/tracing/dynamic_opentracing_driver_impl.h

@@ -0,0 +1,41 @@
+#pragma once
+
+#include "envoy/runtime/runtime.h"
+#include "envoy/thread_local/thread_local.h"
+#include "envoy/tracing/http_tracer.h"
+#include "envoy/upstream/cluster_manager.h"
+
+#include "common/tracing/opentracing_driver_impl.h"
+
+#include "opentracing/dynamic_load.h"
+
+namespace Envoy {
+namespace Tracing {
+
+/**
+ * This driver provides support for dynamically loading tracing libraries into Envoy that provide an
+ * implementation of the OpenTracing API (see https://github.com/opentracing/opentracing-cpp).
+ * TODO(rnburn): Add an example showing how to use a tracer library with this driver.
+ */
+class DynamicOpenTracingDriver : public OpenTracingDriver {
+public:
+  DynamicOpenTracingDriver(Stats::Store& stats, const std::string& library,
+                           const std::string& tracer_config);
+
+  static std::string formatErrorMessage(std::error_code error_code,
+                                        const std::string& error_message);
+
+  // Tracer::OpenTracingDriver
+  opentracing::Tracer& tracer() override { return *tracer_; }
+
+  PropagationMode propagationMode() const override {
+    return OpenTracingDriver::PropagationMode::TracerNative;
+  }
+
+private:
+  opentracing::DynamicTracingLibraryHandle library_handle_;
+  std::shared_ptr<opentracing::Tracer> tracer_;
+};
+
+} // namespace Tracing
+} // namespace Envoy

source/common/tracing/lightstep_tracer_impl.cc

@@ -15,25 +15,21 @@
 namespace Envoy {
 namespace Tracing {
 
-namespace {
-class LightStepLogger : Logger::Loggable<Logger::Id::tracing> {
-public:
-  void operator()(lightstep::LogLevel level, opentracing::string_view message) const {
-    const fmt::StringRef fmt_message{message.data(), message.size()};
-    switch (level) {
-    case lightstep::LogLevel::debug:
-      ENVOY_LOG(debug, "{}", fmt_message);
-      break;
-    case lightstep::LogLevel::info:
-      ENVOY_LOG(info, "{}", fmt_message);
-      break;
-    default:
-      ENVOY_LOG(warn, "{}", fmt_message);
-      break;
-    }
+void LightStepLogger::operator()(lightstep::LogLevel level,
+                                 opentracing::string_view message) const {
+  const fmt::StringRef fmt_message{message.data(), message.size()};
+  switch (level) {
+  case lightstep::LogLevel::debug:
+    ENVOY_LOG(debug, "{}", fmt_message);
+    break;
+  case lightstep::LogLevel::info:
+    ENVOY_LOG(info, "{}", fmt_message);
+    break;
+  default:
+    ENVOY_LOG(warn, "{}", fmt_message);
+    break;
   }
-};
-} // namespace
+}
 
 LightStepDriver::LightStepTransporter::LightStepTransporter(LightStepDriver& driver)
     : driver_(driver) {}
@@ -68,20 +64,24 @@ void LightStepDriver::LightStepTransporter::onSuccess(Http::MessagePtr&& respons
     active_request_ = nullptr;
     Grpc::Common::validateResponse(*response);
 
-    Grpc::Common::chargeStat(*driver_.cluster(), lightstep::CollectorServiceFullName(),
-                             lightstep::CollectorMethodName(), true);
     // http://www.grpc.io/docs/guides/wire.html
     // First 5 bytes contain the message header.
     response->body()->drain(5);
     Buffer::ZeroCopyInputStreamImpl stream{std::move(response->body())};
     if (!active_response_->ParseFromZeroCopyStream(&stream)) {
       throw EnvoyException("Failed to parse LightStep collector response");
     }
+    Grpc::Common::chargeStat(*driver_.cluster(), lightstep::CollectorServiceFullName(),
+                             lightstep::CollectorMethodName(), true);
     active_callback_->OnSuccess();
   } catch (const Grpc::Exception& ex) {
     Grpc::Common::chargeStat(*driver_.cluster(), lightstep::CollectorServiceFullName(),
                              lightstep::CollectorMethodName(), false);
     active_callback_->OnFailure(std::make_error_code(std::errc::network_down));
+  } catch (const EnvoyException& ex) {
+    Grpc::Common::chargeStat(*driver_.cluster(), lightstep::CollectorServiceFullName(),
+                             lightstep::CollectorMethodName(), false);
+    active_callback_->OnFailure(std::make_error_code(std::errc::bad_message));
   }
 }
 
@@ -112,7 +112,7 @@ LightStepDriver::TlsLightStepTracer::TlsLightStepTracer(
   enableTimer();
 }
 
-const opentracing::Tracer& LightStepDriver::TlsLightStepTracer::tracer() const { return *tracer_; }
+opentracing::Tracer& LightStepDriver::TlsLightStepTracer::tracer() { return *tracer_; }
 
 void LightStepDriver::TlsLightStepTracer::enableTimer() {
   const uint64_t flush_interval =
@@ -160,7 +160,7 @@ LightStepDriver::LightStepDriver(const Json::Object& config,
   });
 }
 
-const opentracing::Tracer& LightStepDriver::tracer() const {
+opentracing::Tracer& LightStepDriver::tracer() {
   return tls_->getTyped<TlsLightStepTracer>().tracer();
 }
 

source/common/tracing/lightstep_tracer_impl.h

@@ -31,6 +31,14 @@ struct LightstepTracerStats {
   LIGHTSTEP_TRACER_STATS(GENERATE_COUNTER_STRUCT)
 };
 
+/**
+ * LightStepLogger is used to translate logs generated from LightStep's tracer to Envoy logs.
+ */
+class LightStepLogger : Logger::Loggable<Logger::Id::tracing> {
+public:
+  void operator()(lightstep::LogLevel level, opentracing::string_view message) const;
+};
+
 /**
  * LightStep (http://lightstep.com/) provides tracing capabilities, aggregation, visualization of
  * application trace data.
@@ -50,7 +58,7 @@ class LightStepDriver : public OpenTracingDriver {
   LightstepTracerStats& tracerStats() { return tracer_stats_; }
 
   // Tracer::OpenTracingDriver
-  const opentracing::Tracer& tracer() const override;
+  opentracing::Tracer& tracer() override;
   PropagationMode propagationMode() const override { return propagation_mode_; }
 
 private:
@@ -90,7 +98,7 @@ class LightStepDriver : public OpenTracingDriver {
     TlsLightStepTracer(const std::shared_ptr<lightstep::LightStepTracer>& tracer,
                        LightStepDriver& driver, Event::Dispatcher& dispatcher);
 
-    const opentracing::Tracer& tracer() const;
+    opentracing::Tracer& tracer();
 
   private:
     void enableTimer();

source/common/tracing/opentracing_driver_impl.h

@@ -51,7 +51,7 @@ class OpenTracingDriver : public Driver, protected Logger::Loggable<Logger::Id::
   SpanPtr startSpan(const Config& config, Http::HeaderMap& request_headers,
                     const std::string& operation_name, SystemTime start_time) override;
 
-  virtual const opentracing::Tracer& tracer() const PURE;
+  virtual opentracing::Tracer& tracer() PURE;
 
   enum class PropagationMode { SingleHeader, TracerNative };
 

source/exe/BUILD

@@ -80,6 +80,7 @@ envoy_cc_library(
     deps = [
         ":envoy_main_common_lib",
         ":extra_protocol_proxies_lib",
+        "//source/server/config/http:dynamic_opentracing_lib",
         "//source/server/config/http:lightstep_lib",
         "//source/server/config/http:zipkin_lib",
     ],

source/server/config/http/BUILD

@@ -161,6 +161,18 @@ envoy_cc_library(
     ],
 )
 
+envoy_cc_library(
+    name = "dynamic_opentracing_lib",
+    srcs = ["dynamic_opentracing_http_tracer.cc"],
+    hdrs = ["dynamic_opentracing_http_tracer.h"],
+    deps = [
+        "//source/common/config:well_known_names",
+        "//source/common/tracing:dynamic_opentracing_driver_lib",
+        "//source/common/tracing:http_tracer_lib",
+        "//source/server:configuration_lib",
+    ],
+)
+
 envoy_cc_library(
     name = "ratelimit_lib",
     srcs = ["ratelimit.cc"],

source/server/config/http/dynamic_opentracing_http_tracer.cc

@@ -0,0 +1,37 @@
+#include "server/config/http/dynamic_opentracing_http_tracer.h"
+
+#include <string>
+
+#include "envoy/registry/registry.h"
+
+#include "common/common/utility.h"
+#include "common/config/well_known_names.h"
+#include "common/tracing/dynamic_opentracing_driver_impl.h"
+#include "common/tracing/http_tracer_impl.h"
+
+namespace Envoy {
+namespace Server {
+namespace Configuration {
+
+Tracing::HttpTracerPtr DynamicOpenTracingHttpTracerFactory::createHttpTracer(
+    const Json::Object& json_config, Server::Instance& server,
+    Upstream::ClusterManager& /*cluster_manager*/) {
+  const std::string library = json_config.getString("library");
+  const std::string config = json_config.getObject("config")->asJsonString();
+  Tracing::DriverPtr dynamic_driver{
+      std::make_unique<Tracing::DynamicOpenTracingDriver>(server.stats(), library, config)};
+  return std::make_unique<Tracing::HttpTracerImpl>(std::move(dynamic_driver), server.localInfo());
+}
+
+std::string DynamicOpenTracingHttpTracerFactory::name() {
+  return Config::HttpTracerNames::get().DYNAMIC_OT;
+}
+
+/**
+ * Static registration for the dynamic opentracing http tracer. @see RegisterFactory.
+ */
+static Registry::RegisterFactory<DynamicOpenTracingHttpTracerFactory, HttpTracerFactory> register_;
+
+} // namespace Configuration
+} // namespace Server
+} // namespace Envoy

source/server/config/http/dynamic_opentracing_http_tracer.h

@@ -0,0 +1,29 @@
+#pragma once
+
+#include <string>
+
+#include "envoy/server/instance.h"
+
+#include "server/configuration_impl.h"
+
+namespace Envoy {
+namespace Server {
+namespace Configuration {
+
+/**
+ * Config registration for the dynamic opentracing tracer. @see HttpTracerFactory.
+ */
+class DynamicOpenTracingHttpTracerFactory : public HttpTracerFactory {
+public:
+  // HttpTracerFactory
+  Tracing::HttpTracerPtr createHttpTracer(const Json::Object& json_config, Server::Instance& server,
+                                          Upstream::ClusterManager& cluster_manager) override;
+
+  std::string name() override;
+
+  bool requiresClusterName() const override { return false; }
+};
+
+} // namespace Configuration
+} // namespace Server
+} // namespace Envoy

source/server/configuration_impl.cc

@@ -96,11 +96,6 @@ void MainImpl::initializeTracers(const envoy::config::trace::v2::Tracing& config
     return;
   }
 
-  if (server.localInfo().clusterName().empty()) {
-    throw EnvoyException("cluster name must be defined if tracing is enabled. See "
-                         "--service-cluster option.");
-  }
-
   // Initialize tracing driver.
   std::string type = configuration.http().name();
   ENVOY_LOG(info, "  loading tracing driver: {}", type);
@@ -111,6 +106,11 @@ void MainImpl::initializeTracers(const envoy::config::trace::v2::Tracing& config
 
   // Now see if there is a factory that will accept the config.
   auto& factory = Config::Utility::getAndCheckFactory<HttpTracerFactory>(type);
+  if (factory.requiresClusterName() && server.localInfo().clusterName().empty()) {
+    throw EnvoyException(fmt::format("cluster name must be defined for the tracing driver {}. See "
+                                     "--service-cluster option.",
+                                     type));
+  }
   http_tracer_ = factory.createHttpTracer(*driver_config, server, *cluster_manager_);
 }
 

source/server/configuration_impl.h

@@ -54,6 +54,11 @@ class HttpTracerFactory {
    * factory.
    */
   virtual std::string name() PURE;
+
+  /**
+   * Returns true if the tracing driver requires cluster name to be defined.
+   */
+  virtual bool requiresClusterName() const { return true; }
 };
 
 /**