[Lens] Custom Breakdown Presentation Order

[MichaelMarcialis]

As part of the One Discover contextual logs effort, we would like to present a histogram of documents broken down by the log.level field. To make this information easier to interpret, it would be ideal if we could have greater control over the breakdown presentation order, allowing us to order the legend and breakdown stacking by log.level severity (see example in mockup below). As I understand it, this is currently not possible on either the Lens and Elastic Charts side.

Related issue: #86184

CCing @stratoula, @teresaalvarezsoler, @nickofthyme.

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[teresaalvarezsoler]

@stratoula is this chart only for ES|QL or also for the dataview mode?

[stratoula]

Both! right @davismcphee ?

[teresaalvarezsoler]

Scope for this issue: we need a bit more investigation to figure out how to solve this problem so we are repurposing this as a spike and time-boxing it to < 1 week

[davismcphee]

@stratoula correct! This is needed for both data view and ES|QL mode.

[nickofthyme]

This functionality will be available in kibana when #195447 is merged.

[markov00]

#195447 is merged but this can be achieved today only through Lens override functions, that are not persisted into Lens nor into a Dashboard.
I'd like to propose adding this feature in Lens by reusing the sorting order of the color mapping for rendering purposes too.
The sorting order of the data is inherently connected to the breakdown. This should be part of the breakdown configuration, where you can specify the data sorting (not the ranking): usually you can specify it with asc/desc/numerical/alphabetical but you can also specify it via in a list/index of terms/values to follow.
For now, we can probably also consider using the color mapping for that functionality, because it aligns with that concept.
This can be applied on a request basis via a switch in the color mapping like the following example:

In the long run, probably these operations should be described in the "data processing" part of Lens and not connected to the data styling.

[MichaelMarcialis]

I'd like to propose adding this feature in Lens by reusing the sorting order of the color mapping for rendering purposes too.

@markov00: I agree that we should provide configuration controls for users to apply a custom order to their visualization breakdowns. However, I do have a few concerns about relegating that control to the color mapping interface:

1. I fear it will have very low discoverability there. What if a user doesn't wish to change their breakdown colors (which I believe is a majority of users)? Would they know to find such sorting controls there?
2. Using a breakdown dimension isn't the only way for users to stack bar or area charts. For example, what if a user has multiple stacked bar layers with no breakdowns? Would we need to accommodate for adjusting the sort order in such a scenario? If so, I don't think the color mapping interface would be viable, as it is attributed to a single layer.
3. The current color mapping interface has no drag-and-drop support for the reordering of terms. We can of course add drag-and-drop to it, but I fear that it would make an already complex UI even more complex.

Also, this related issue from Wylie seemed to indicate that there were use cases beyond just breakdowns.

Personally, my initial instinct is to add any sort of sorting controls at the global visualization level (i.e. toolbar menus), if we need to account for that multi-layer example I describe above. If we don't have to account for the multi-layer example, then at the root-level of the appropriate layer dimension's configuration flyout (i.e. not within the deeper color mapping configuration) would be my preference. Thoughts?

[markov00]

@MichaelMarcialis let's discuss this offline and understand what/where we can go

[kertal]

What's the current state of this issue, as it appears to block #194549? thx

[markov00]

Unfortunately we don't have the capacity to take this fully right now. We suggested some other workarounds that involves revisiting the way UnifiedHistogram works to unblock this in the mean time.
This currently has a low priorities. Please contact @ghudgins if this priority needs to be escalated

[ghudgins]

we spoke about it this AM for an hour and likely need to sync with @ninoslavmiskovic and @davismcphee to figure out how important this is and which alternative path is the right one:

1. change query in histogram to use a sort column discover team says unified histogram isn't safe to change and will be refactored

FROM kibana_sample_data_logs
  | EVAL sortIndex = CASE(extension.keyword == "zip", 1, extension.keyword == "deb", 2, extension.keyword == "css", 3, extension.keyword == "gz", 4, extension.keyword == "rpm", 5, 100)
  | STATS COUNT(*) BY time = BUCKET(@timestamp, 100, ?_tstart, ?_tend),  extension.keyword, sortIndex
  | SORT sortIndex ASC, time ASC
  | DROP sortIndex

2. implement the sort logic in lens hard coded on log.level - not a great solution as it would be broken in the UI when you use logs data
3. a bad lens custom sort text box we have to live with forever (it's in saved object--would be some delimiter separated list of strings and would have issues) - commits us to a somewhat limited solution in the client. would require us to clean it up later or double down on it with more user experience. most of the work to do 3 is the same as 5 so we might as well do 5
4. override sort in discover but visualization changes when you save to dashboard, sort is broken not a viable solution because it breaks the promise that the visualization is something you can use in a dashboard
5. a good lens custom sort user experience that is like color by term that we live with forever (it's in the saved object) - would take 1 month to do. up against priorities and other enhancements but would provide a good user experience
6. ESQL adds custom SORT (list of strings) - added to ESQL list, not prioritized yet
7. ESQL adds dynamic SORT by query (how we do breakdown order in DSL for date histogram) - additional functionality ESQL needs beyond the custom SORT in the long run (or we double down on the client)

[davismcphee]

Thanks for the update @ghudgins 👍 I agree we should do a sync about it to work through some of the details and make sure we're all on the same page.

discover team says unified histogram isn't safe to change and will be refactored

I don't want to mislead here, we can definitely make changes to Unified Histogram to support custom breakdown orders. There are issues with it currently so we intend to refactor regardless, but it wouldn't stop us from making necessary changes.

I think the main issue with that approach is just that it's ES|QL only, which may be ok since we've been asked to focus primarily on ES|QL now, but of course a solution that works for all use cases would be preferred if feasible. Maybe this would work in combination with #7 to support both cases though:

ESQL adds dynamic SORT by query (how we do breakdown order in DSL for date histogram)

I think I'm missing some context though since I'm not familiar with "how we do breakdown order in DSL for date histogram". Definitely something to explore in our sync.

[kertal]

should we already use our Monday DataDiscovery sync for this? however it collides with VisEditors sync, but we had nice VisEditior folks at the start of our sync the last 2 weeks, it could become a tradition ... or we setup a seperate one