How do I set thread_name?

[persona94]

I'd like to have a custom thread name for every log generated from a different thread. From the docs it looks like I need to set thread_name. To accomplish I did

#define LOG_TEST(...) \
    quill::detail::set_thread_name("Test");\
    LOG_INFO(*logger, __VA_ARGS__)

I'm using a user-defined JsonConsoleSink as discussed in this discussion - #601 (comment)

The first 2 calls to LOG_TEST() do show the thread name, but subsequent calls from other threads only show [#1] or [#2] which I believe is an indication that it crashed or terminated? What am I doing wrong?

[odygrd]

Hey, make sure to set your thread name once when the thread starts and before issuing any log statements from that thread. The thread name is retrieved and cached on the first log statement for that thread.

The function quill::detail::set_thread_name is used internally to set the backend worker thread’s name. While it technically works for other threads, it’s not recommended to use functions from the detail namespace directly in your application code. Ideally, you should implement your own function to set thread names. You can use the detail namespace for testing, though.

Once the thread name is set, it will be accessible in the sink, where you can include it in your log entries.

quill/include/quill/sinks/JsonFileSink.h

Line 63 in 62450f9

std::string_view thread_id, std::string_view thread_name,

if you still have issues you can submit a minimal example and i can help

[odygrd]

note: thread name on linux is max 15 chars

#include "quill/Backend.h"
#include "quill/Frontend.h"
#include "quill/LogMacros.h"
#include "quill/Logger.h"
#include "quill/sinks/JsonConsoleSink.h"

#include <string>

class CustomJsonConsoleSink : public quill::StreamSink
{
public:
  CustomJsonConsoleSink() : StreamSink("stdout", nullptr) {}
  ~CustomJsonConsoleSink() override = default;

  void write_log(quill::MacroMetadata const* log_metadata, uint64_t log_timestamp,
                 std::string_view thread_id, std::string_view thread_name,
                 std::string const& process_id, std::string_view logger_name, quill::LogLevel log_level,
                 std::string_view log_level_description, std::string_view log_level_short_code,
                 std::vector<std::pair<std::string, std::string>> const* named_args,
                 std::string_view, std::string_view) override
  {
    _json_message.clear();

    char const* message_format;

    if (strchr(log_metadata->message_format(), '\n') != nullptr)
    {
      _format = log_metadata->message_format();

      for (size_t pos = 0; (pos = _format.find('\n', pos)) != std::string::npos; pos++)
      {
        _format.replace(pos, 1, " ");
      }

      message_format = _format.data();
    }
    else
    {
      message_format = log_metadata->message_format();
    }

    // !! - Add thread_name to json
    _json_message.append(fmtquill::format(
      R"({{"timestamp":"{}","file_name":"{}","line":"{}","thread_id":"{}",thread_name:"{}","logger":"{}","log_level":"{}","message":"{}")",
      std::to_string(log_timestamp), log_metadata->file_name(), log_metadata->line(), thread_id,
      thread_name, logger_name, log_level_description, message_format));

    if (named_args)
    {
      for (auto const& [key, value] : *named_args)
      {
        _json_message.append(std::string_view{",\""});
        _json_message.append(key);
        _json_message.append(std::string_view{"\":\""});
        _json_message.append(value);
        _json_message.append(std::string_view{"\""});
      }
    }

    _json_message.append(std::string_view{"}\n"});

    StreamSink::write_log(log_metadata, log_timestamp, thread_id, thread_name, process_id, logger_name, log_level,
                          log_level_description, log_level_short_code, named_args, std::string_view{},
                          std::string_view{_json_message.data(), _json_message.size()});
  }

private:
  fmtquill::memory_buffer _json_message;
  std::string _format;
};

int main()
{
  // Start the backend thread
  quill::BackendOptions backend_options;
  quill::Backend::start(backend_options);

  // Frontend

  // Create a json sink
  auto json_sink = quill::Frontend::create_or_get_sink<CustomJsonConsoleSink>("json_sink_1");

  // Create a logger
  // When logging json, it is ideal to set the logging pattern to empty to avoid unnecessary message formatting.
  quill::Frontend::create_or_get_logger(
    "json_logger", std::move(json_sink),
    quill::PatternFormatterOptions{"", "%H:%M:%S.%Qns", quill::Timezone::GmtTime});

  auto t1 = std::thread([]() {
                          quill::Logger* logger = quill::Frontend::get_logger("json_logger");

                          // ideally use your own implementation to set the thread nae
                          quill::detail::set_thread_name("t1_thread");

                          LOG_INFO(logger, "A json message {var_1}", "test");
                          LOG_INFO(logger, "A json message {var_1}", "test2");
                          LOG_INFO(logger, "A json message {var_1}", "test3");
                        });

  auto t2 = std::thread([]() {
                          quill::Logger* logger = quill::Frontend::get_logger("json_logger");

                          // ideally use your own implementation to set the thread nae
                          quill::detail::set_thread_name("t2_thread");

                          LOG_INFO(logger, "A json message {var_1}", "bar");
                          LOG_INFO(logger, "A json message {var_1}", "bar2");
                          LOG_INFO(logger, "A json message {var_1}", "bar3");
                        });

  t1.join();
  t2.join();
}

[persona94]

Thank you. I use a Custom JSON sink and the implementation is quite similar to the sample code you posted. However the catch here is that I use a thread pool and the threads do not join until the program stops (which is never)

The reason I needed to set the "thread name" was because I wanted to sort of override the thread name to include thread specific debugging info. In my initial implementation I pushed them into thread_local but I guess since the logging thread runs in it's own thread I can't access those variables. I feel like I'm having a use case that's very similar to this MDC discussion from some time agp.

The reason I need this is to tie threads together. If thread A spawns thread B and there's a debug chunk in thr A and then thr B it would be useful to connect them if the "thread ID" to link them

[odygrd]

The t1.join() in the example is necessary in C++ when the program ends—it ensures proper thread termination. However, it does not influence the thread_name.

We currently do not have a Mapping Diagnostic Context (MDC) because supporting it on the hot path would come with a performance cost. However, you can easily simulate this by passing the relevant variables directly to each log statement each time. This method avoids relying on MDC to append the variables to your log entries automatically.

The reason I need this is to tie threads together. If thread A spawns thread B and there's a debug chunk in thread A and then in thread B, it would be useful to connect them using a "thread ID."

To achieve this, you can use a distinct Logger instance with a unique name for each root thread sharing the same json_sink. For example:

Logger* thread_a_logger = quill::Frontend::create_or_get_logger(
    "thread_A_logger", json_sink,
    quill::PatternFormatterOptions{"", "%H:%M:%S.%Qns", quill::Timezone::GmtTime});

Logger* thread_d_logger = quill::Frontend::create_or_get_logger(
    "thread_D_logger", json_sink,
    quill::PatternFormatterOptions{"", "%H:%M:%S.%Qns", quill::Timezone::GmtTime});

By associating each root thread with a unique Logger instance, you can easily track log entries.

For instance, if you start with a root thread, say thread A, you can pass Logger* thread_a_logger to any subsequent threads or functions you call, such as thread B then pass the same Logger* thread_a_logger to thread_c etc..

Subsequently, thread B will use LOG_INFO(thread_a_logger, ...) to log messages. In the log output, you will see the logger name, allowing you to determine the origin (root thread) of each log statement.

[persona94]

Right I understand why thread.join() was needed. In my use case however I use multiple thread pools. Some are managed by poco some are managed by me. I think I end up with several hundred threads altogether and I don't have control of which thread calls what. Maybe I can daisy chain them by passing but it'll also mean a big lift on how so many loggers are created and maintained esp. since I currently use 1 instance of logger* thru' a singleton interface.

Is there any way to have the logger run in-thread rather than in it's own thread? That might affect performance but at least I can leverage thread_local variables that way

[odygrd]

It is not possible to run it without the logging thread, the whole architecture of the library is designed around async logging

Not sure I understand what you would like to achieve with the thread locals… you could just pass them to each log statement, that is the same as MDC but with additional typing eg LOG_INFO(logger, “{t}”, threadLocal)

the best way to know what each thread calls is to create a Logger with a unique name per thread and pass the Logger* to the functions you are calling from that thread

Logger instance is a cheap object even if you create thousay is fine

[persona94]

Yeah makes sense. Yeah I can do the additional typing was just hoping to have it happen automatically behind the scenes

[odygrd]

This is an example of how you can track it with Logger instances

#include "quill/Backend.h"
#include "quill/Frontend.h"
#include "quill/LogMacros.h"
#include "quill/Logger.h"
#include "quill/sinks/JsonConsoleSink.h"

#include <string>

class CustomJsonConsoleSink : public quill::StreamSink
{
public:
  CustomJsonConsoleSink() : StreamSink("stdout", nullptr) {}
  ~CustomJsonConsoleSink() override = default;

  void write_log(quill::MacroMetadata const* log_metadata, uint64_t log_timestamp,
                 std::string_view thread_id, std::string_view thread_name,
                 std::string const& process_id, std::string_view logger_name, quill::LogLevel log_level,
                 std::string_view log_level_description, std::string_view log_level_short_code,
                 std::vector<std::pair<std::string, std::string>> const* named_args,
                 std::string_view, std::string_view) override
  {
    _json_message.clear();

    char const* message_format;

    if (strchr(log_metadata->message_format(), '\n') != nullptr)
    {
      _format = log_metadata->message_format();

      for (size_t pos = 0; (pos = _format.find('\n', pos)) != std::string::npos; pos++)
      {
        _format.replace(pos, 1, " ");
      }

      message_format = _format.data();
    }
    else
    {
      message_format = log_metadata->message_format();
    }

    // !! - Add thread_name to json
    _json_message.append(fmtquill::format(
      R"({{"logger":"{}","message":"{}")",logger_name, message_format));

    if (named_args)
    {
      for (auto const& [key, value] : *named_args)
      {
        _json_message.append(std::string_view{",\""});
        _json_message.append(key);
        _json_message.append(std::string_view{"\":\""});
        _json_message.append(value);
        _json_message.append(std::string_view{"\""});
      }
    }

    _json_message.append(std::string_view{"}\n"});

    StreamSink::write_log(log_metadata, log_timestamp, thread_id, thread_name, process_id, logger_name, log_level,
                          log_level_description, log_level_short_code, named_args, std::string_view{},
                          std::string_view{_json_message.data(), _json_message.size()});
  }

private:
  fmtquill::memory_buffer _json_message;
  std::string _format;
};

void mul(quill::Logger* logger, size_t a, size_t b)
{
  LOG_INFO(logger, "{a} * {b} = {c}", a, b, a * b);
}

void add(quill::Logger* logger, size_t a, size_t b)
{
  LOG_INFO(logger, "{a} + {b} = {c}", a, b, a + b);

  mul(logger, a, b);
}

int main()
{
  // Start the backend thread
  quill::BackendOptions backend_options;
  quill::Backend::start(backend_options);

  // Create a json sink
  auto json_sink = quill::Frontend::create_or_get_sink<CustomJsonConsoleSink>("json_sink_1");

  std::vector<std::thread> threads;

  for (size_t i = 0; i < 10; ++i)
  {
    threads.emplace_back([i]()
                      {
                        // setup once when the thread starts ..

                        // get existing sink
                        auto json_sink = quill::Frontend::get_sink("json_sink_1");

                        // create logger
                        quill::Logger* logger = quill::Frontend::create_or_get_logger(
                          "thread_" + std::to_string(i) , std::move(json_sink),
                          quill::PatternFormatterOptions{"", "%H:%M:%S.%Qns", quill::Timezone::GmtTime});

                        // thread main loop ..
                        for (size_t j = 0; j < 10; ++j)
                        {
                          add(logger, j, j);
                        }

                        // when thread exits ..
                        quill::Frontend::remove_logger(logger);
                      });
  }

  // simulate program run..
  std::this_thread::sleep_for(std::chrono::seconds{10});

  for (auto& thread : threads)
  {
    thread.join();
  }
}

[persona94]

I ended up doing something very similar. The only thing I did on top of this was define my own macros that called Quill's macros

#define LOG_TEST(...) \
    LOG_INFO(logger , __VA_ARGS__, "id1", getid())

This way I pass in the ID captured in the local thread context, and hands off to Quill.

Everything else include the json custom logger are very similar. Thanks!