logging: add TRACE log level and move some DEBUG logging to it

[ksedgwic]

DEBUG logging is very large, would be nice to reduce: #7279

This PR adds a new logging level TRACE between DEBUG and the IO levels and moves some examples of noisy logging to it.

Implications:

• no logging is removed, debugging users can set the logging level to TRACE and get everything as in the past
• it is possible to consider setting logging to DEBUG for longer periods of time

[vincenzopalazzo]

Concept ACK b7ea088

The CI is complaining, so the following diff should make happy

diff --git a/doc/schemas/lightning-getlog.json b/doc/schemas/lightning-getlog.json
index e179cfe75..018c14303 100644
--- a/doc/schemas/lightning-getlog.json
+++ b/doc/schemas/lightning-getlog.json
@@ -17,6 +17,7 @@
           "unusual",
           "info",
           "debug",
+          "trace",
           "io"
         ],
         "description": [
@@ -69,6 +70,7 @@
                 "UNUSUAL",
                 "INFO",
                 "DEBUG",
+                "TRACE",
                 "IO_IN",
                 "IO_OUT"
               ]
@@ -111,7 +113,8 @@
                       "BROKEN",
                       "UNUSUAL",
                       "INFO",
-                      "DEBUG"
+                      "DEBUG",
+                      "TRACE"
                     ]
                   }
                 }

[vincenzopalazzo]

ACK 29f5026

I like it because with the trace level we can remove a lot of bad information for the log, usually plugins are going to log in a verbose way, but in this way we can save some space on the disk :)

Nice catch for the drive by fix 😸 I learn a new thing with calloc

[rustyrussell]

Can't push to your tree, but you're missing a Changelog-Added line, and docs in two places. See this:

From cabfd54a97e597abc5bc7a44c355450c9f420575 Mon Sep 17 00:00:00 2001
From: Rusty Russell <rusty@rustcorp.com.au>
Date: Tue, 7 May 2024 10:39:15 +0930
Subject: [PATCH] doc: document 'trace' level in lightning-cli,
 lightningd-config

This commit also gives me a place to hang the Changelog message,
even though I did nothing to contribute!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: new log level `trace` where we moved the very noisiest `debug` logs.

diff --git a/doc/lightning-cli.1.md b/doc/lightning-cli.1.md
index e121e1f0d..84dd77f46 100644
--- a/doc/lightning-cli.1.md
+++ b/doc/lightning-cli.1.md
@@ -68,7 +68,7 @@ field without parsing JSON.
 * **--notifications**/**-N**=*LEVEL*
 
   If *LEVEL* is 'none', then never print out notifications.  Otherwise,
-print out notifications of *LEVEL* or above (one of `io`, `debug`,
+print out notifications of *LEVEL* or above (one of `io`, `trace`, `debug`,
 `info` (the default), `unusual` or `broken`: they are prefixed with `#
 `.  (Note: currently not supported with `--commando`).
 
diff --git a/doc/lightningd-config.5.md b/doc/lightningd-config.5.md
index 8f67d709a..71bcb3340 100644
--- a/doc/lightningd-config.5.md
+++ b/doc/lightningd-config.5.md
@@ -170,7 +170,7 @@ binary.
 
 * **log-level**=*LEVEL*\[:*SUBSYSTEM*\]\[:*PATH*\]
 
-  What log level to print out: options are io, debug, info, unusual,
+  What log level to print out: options are io, trace, debug, info, unusual,
 broken.  If *SUBSYSTEM* is supplied, this sets the logging level
 for any subsystem (or *nodeid*) containing that string. If *PATH* is supplied, it means this log-level filter is only applied to that `log-file`, which is useful for creating logs to capture a specific subsystem.  This option may be specified multiple times.
 Subsystems include:

[rustyrussell]

Changelog and doc updates needed