Add API to get Index Metrics

[Jiaweihu08]

Add an API to QbeastTable to retrieve OTree index metrics more easily!

This is how would you use it:

val qbeastTable = QbeastTable.forPath(spark, tmpDir)
val metrics = qbeastTable.getIndexMetrics()

The metrics included so far are:
General index metadata:

• desiredCubeSize
• number of cubes
• depth
• average fan out of the cubes
• dimension count
• number of rows

Some more specific details such as depthOverLogNumNodes = depth / log(cubeCounts), depthOnBalance = depth / log(rowCount/desiredCubeSize), both logs use base = dimensionCount.

We also take a closer look at the non leaf cube sizes. NonLeafCubeSizeDetails contains their min, max, quantiles, and how far each of the cube sizes are from the desiredCubeSize.

Map[CubeId, CubeStatus] of the index is also returned, since some of the information stored in CubeStatus are interesting to analyze, such as the distribution of cube weights for different indexes.

You can access this information through metrics.cubeStatuses.

Example output:

OTree Index Metrics:
dimensionCount: 2
elementCount: 1001
depth: 2
cubeCounts: 7
desiredCubeSize: 100
avgFanOut: 2.0
depthOverLogNumNodes: 0.7124143742160444
depthOnBalance: 0.6020599913279624
Non-lead Cube Size Stats:
(All values are 0 if there's no non-leaf cubes):
- min: 3729
- firstQuartile: 3729
- secondQuartile: 4832
- thirdQuartile: 5084
- max: 5084
- dev: 2.0133907E7

[osopardo1]

Seems good! Whenever you can, @Jiaweihu08 , please resolve the conflicts and merge it!

[Jiaweihu08]

Shouldn't we also add IndexStatuses as well? @osopardo1
The distribution of weights can also be interesting to analize. We can include it in IndexMetrics and leave toString untouched.

[osopardo1]

Shouldn't we also add IndexStatuses as well? @osopardo1 The distribution of weights can also be interesting to analize. We can include it in IndexMetrics and leave toString untouched.

Do you mean the Cube and Weight map? Because I don't know if including the files makes sense. But yeah, sure!

Also, let's include a document in docs where we explain the API (sorry, I missed doing this part when I added new methods). Let me know if you take the task or I will!

[Jiaweihu08]

Shouldn't we also add IndexStatuses as well? @osopardo1 The distribution of weights can also be interesting to analize. We can include it in IndexMetrics and leave toString untouched.

Do you mean the Cube and Weight map? Because I don't know if including the files makes sense. But yeah, sure!

Also, let's include a document in docs where we explain the API (sorry, I missed doing this part when I added new methods). Let me know if you take the task or I will!

Yes, I meant Cube Statuses. I'll do the documentation. Thanks!