Kibana usage information

[arisonl]

There is demand for offering usage data within Kibana, referring to saved objects, spaces and users. Examples include:

• Most and least used dashboards, visualisations and solutions features, to help keep track and organise their saved objects, to understand which visualisations are used the most and hence they should prioritise to develop further etc.
• Most and currently active users (open sessions), active users per day or within a time window
• Number of visualisations and dashboards per Kibana space
• Relevant dashboards to visualise this activity
• A history of when an object was modified, who modified it, and what space they modified it from
• etc.

As we are delivering the new Kibana audit logging (and server side sessions), Kibana usage analytics and related features remain to be addressed separately and continue being one of the most heavily requested functionalities.

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[pgayvallet]

A few remarks

Most and least used dashboards, visualisations and solutions features

We don't have the info of number of 'hit' / display for specific pages or features. We would need an API for that.

Most and currently active users, active users per day or within a time window

Will require to collect 'real-time' user statistics to be able to expose that to a collector, we don't atm

Number of visualisations and dashboards per Kibana space

That could be collected without modifications

[mbarretta]

@arisonl does the new audit data logged by Kibana contain enough detail to be allow users to generate their own dashboards on kibana object usage?

[rayafratkina]

A related use case is described in #90215

[arisonl]

@mbarretta Depending on the specific needs, the new audit logging capability may offer a workaround, please have a look at the documentation and let me know how much it could cover based on what you have in mind. Also, let me know of any questions. Authorisation to access is logged together with a number of attributes, including usernames, objects and space IDs, of course the timestamp etc.

[pgayvallet]

Most and least used dashboards, visualisations and solutions features, to help keep track and organise their saved

Most and currently active users, active users per day or within a time window

It's unclear if those metrics should / could really be managed / provided by the 'low-level' core API. It looks more like it should be provided/handled by the associated apps / services (e.g dashboard and security respectively here)

I mean, we could add an accessCount and/or a lastAccessDate on all SOs, but that would only be it, a timeframe-independant counter and/or date.

We would not be able to, for example:

• Distinguish between calls accessing the dashboard to display it to users (Most and least used dashboards) from 'system' calls accessing the dashboard for other reasons that should not be counter in this metric
• Provide metrics on a given time range, such as active users per day or within a time window

Overall, I think that, if these metrics makes a lot of sense, they should be considered as high-level informations that needs to be computed by the owning services/apps (dashboard, security...), and not something that can be generically provided by a core service or API.

[cjcenizal]

Relates to #9202

[lukeelmers]

Also related to #125795, which focuses on usage information we could surface in the saved objects management UI.

[rudolf]

I think it's important that we don't jump to solutions before exploring the problem sufficiently. I didn't read all enhancement requests but many of them are similar to this quote:

We have hundreds of dashboards in Kibana, some of them are probably not used by Kibana users, so we would like to identify them and removed.
How can we check which dashboard is used and which not used by Kibana users?

The problem isn't that this user wants to know usage information, the real problem is that they have too many dashboards/visualizations and want to clean it up. Why is having many dashboards a problem? We know very few users truly have too many saved objects, most are far away from hitting performance problems because of the number of saved objects.

So I suspect the real problem is the fact that all users see all other user's dashboards/visualizations making it hard to find just the ones you are interested in. Spaces is an subpar solution to this problem, it just takes longer before your space gets filled up with noise from your team and having a space per person like on our telemetry cluster doesn't scale either.

We need to spend more time validation this, but my assumption is that having content management functionality like google drive where objects are private by default but can be shared to teams and organised by tags (hopefully we don't need folders too?) would be a much better solution.

[rudolf]

If we do realize the real problem is understanding usage information I agree with pgayvallet's comments that this can't really be solved by just adding more metadata and counters to saved objects. We're already collecting audit logs and I think these provide a much better source of data to make the above kinds of analysis on. We could make it easy to ingest kibana audit logs and provide built-in dashboards that aim to answer the most common questions an admin might have.

[syepes]

Hope this gets added soon, this is really important to understand the platform usage.

Example: https://github.com/izmaxxsun/kibana-access

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)