Add user-facing docs for new logging configuration

[joshdover]

Add user-facing documentation (in asciidoc) for the new logging configuration options

Related to #41956

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[mshustov]

• add docs for new logging config options
• deprecate the legacy logging config options
• deprecate elasticsearch.logQueries after Provide substitution for elasticsearch.logQueries in the new logging config #57546

I suspect we might need help from the Docs team.

[TinaHeiligers]

add docs for new logging config options

The new docs system doesn't have a launch date yet, meaning that we have to add docs to the existing docs. We need to decide what needs to be documented and then where to add it in.

@restrry. @lukeelmers where do you think we should add this? In our current docs, we only mention logging in the Kibana configuration section and then in the dev docs. The dev docs don't have much though, they link to the logging README.

For now, let's focus on the user-facing documentation.

[mshustov]

For now, let's focus on the user-facing documentation.

Agree. However, I always consider logging/README.md as user-facing documentation. Let's convert it to a format compatible with our documentation system. That's what you've done in #91958

[TinaHeiligers]

Let's convert it to a format compatible with our documentation system

Our current docs system is based on ascii doc. What I'm asking is what do we do now? Duplicate documentation is a pain to keep in sync but it's doable for a short while.

I see a few options here:

1. Do nothing until we have a launch date for the new system and move it all over there
2. Create new ascii docs from sections in the logging README that are relevant for end users
3. Keep the logging README as it is (.md) file or a Github-render-friendly version of a .mdx file (in other words, only convert the file to .mdx but leave links and sections as they are?
4. Make a copy of the README as a .mdx file and convert it completely to a new-docs format?

@restrry @lukeelmers @joshdover What approach do you think we should take?

[rudolf]

Until the new docs system is ready user-facing docs need to be in asciidoc so I will add that to the description to clarify.

[TinaHeiligers]

@rudolf where do you think these should slot in? ATM, we don't have much on logging setup in Kibana. We mention some logging config and then a but more in the Dev docs.

We'll need to update the dev docs section and change whatever we need to in the config guide but I don't think the config guide is the place to give the extra info we want to. We link to the logging README from the dev docs. I assume it's the contents of the logging README we want to add directly to the kibana docs but I'm not sure where to put that.

[lukeelmers]

We mention some logging config

I think that logging config you link to is actually specific to settings for the logs UI plugin, so we shouldn't need to worry about that one. There are also a handful of logging items in the Configuring Kibana page which would need to be updated.

I wonder if we need to split the README apart:

• the guide with logging overview added to the Core API logging service section (there's enough content that maybe it justifies breaking the "Core API" section up into individual subpages?)
• simple list of config options added to the Configuring Kibana page (removing legacy ones and perhaps also linking back to the guide pages)
• the section on config migration probably would need to be reworded & captured in the 8.0 breaking changes page

[TinaHeiligers]

I agree that we'll need to split up our documentation somehow and your suggestions are a great start!
Fortunately, docs can be updated at any time so we can iterate as we go.
I'll start working on this and we can go from there.

[TinaHeiligers]

Include a section:
"mention in the docs that switching to the new logging config is going to mean you start seeing duplicate log entries in both formats.

Most users will have not run into this yet, but if they are ingesting their Kibana logs somewhere, it would be good for them to know that using the new config will result in additional log entries which are in a different structure."

[TinaHeiligers]

simple list of config options added to the Configuring Kibana page (removing legacy ones and perhaps also linking back to the guide pages)

@lukeelmers This isn't as easy as it sounds as our KP logging config is far more involved than the legacy logging system. What level of detail should we go into here? A full list of config options is going to be awkward because a lot of the fields depend on other logging configuration.(we'd have to use placeholders).
So far, I've added info for:

• logging.root, logging.root.level, logging.root.appenders
• logging.loggers, logging.loggers.name, logging.loggers.level, logging.loggers.appenders
• logging.appenders, logging.appenders.console, logging.appenders.file, logging.appenders.rolling-file

In summary, there's a short description on the top level logging fields: loggers, appenders and root.

I'm stuggling to find a logical place to add info about layout and the pattern layout conversion fields.

Elasticsearch has a whole page dedicated to logging configuration that starts with an example. We could do something similar and add a new page covering logging configuration with an extended example and the result one would get from that (for example, the full example we have in the logging README).

Does this make sense or do you have another suggestion?

[lukeelmers]

Elasticsearch has a whole page dedicated to logging configuration that starts with an example. We could do something similar and add a new page covering logging configuration with an extended example and the result one would get from that

++ I think a dedicated page makes sense. IMHO the items you've added so far make sense for the main list of config options in "Configuring Kibana", and from each of those we could link out to corresponding sections in a dedicated logging page which goes into more detail. It's useful to have those items in the main config page because it makes them more discoverable, but I agree that getting into layouts/patterns there is probably too much detail.

In terms of where to put the dedicated page, I would vote to break apart the Kibana Core API page into dedicated subpages, one of which would be solely dedicated to the logging service.

[TinaHeiligers]

I would vote to break apart the Kibana Core API page into dedicated subpages, one of which would be solely dedicated to the logging service

@lukeelmers I was thinking the same and have already started doing that. It's going to take a bit of work to reword and reorganize the documentation but I hope to get a draft up within a week or two.

[TinaHeiligers]

Once the docs are done, change the links to the logging README in the deprecation warnings for legacy logging configuration to new links to the relevant docs sections.
Related to #82431