Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify index metrics #118

Merged
merged 9 commits into from
Jul 21, 2022
Merged

Conversation

Jiaweihu08
Copy link
Member

Modify index metrics computation. Some others statistics are added since during many experiments they are found useful.

Changes made:

  • Add L1, L2 errors to measure how inner cube sizes deviate from the desiredCubeSize, values are between 0 and 1.
  • For inner cubes, add a method to display average normalizedWeight and cube size per level
  • Compute depthOnBalance using geometrics sum since many indexes don't have a high number of levels
  • Add test for single cube tree

How to use:

import io.qbeast.spark.QbeastTable

val path = "yourTablePath"
val metrics = QbeastTable.forPath(spark, path).getIndexMetrics()

println(metrics)
  • Spark Version: 3.1.3
  • Hadoop Version: 3.2
  • Cluster or local? Local

@Jiaweihu08 Jiaweihu08 added the type: enhancement Improvement of existing feature or code label Jul 18, 2022
@Jiaweihu08 Jiaweihu08 requested a review from osopardo1 July 18, 2022 14:44
@Jiaweihu08 Jiaweihu08 self-assigned this Jul 18, 2022
@osopardo1
Copy link
Member

Looks good! The only missing part is to document what are those metrics in the docs/QbeastTable.md file

@codecov
Copy link

codecov bot commented Jul 20, 2022

Codecov Report

Merging #118 (0757c96) into main (5abd0ad) will increase coverage by 1.58%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #118      +/-   ##
==========================================
+ Coverage   89.13%   90.71%   +1.58%     
==========================================
  Files          60       60              
  Lines        1307     1325      +18     
  Branches       98      102       +4     
==========================================
+ Hits         1165     1202      +37     
+ Misses        142      123      -19     
Impacted Files Coverage Δ
src/main/scala/io/qbeast/spark/QbeastTable.scala 88.00% <100.00%> (+37.12%) ⬆️

@Jiaweihu08 Jiaweihu08 merged commit 9a17df8 into Qbeast-io:main Jul 21, 2022
@Jiaweihu08 Jiaweihu08 deleted the modify-index-metrics branch July 21, 2022 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement Improvement of existing feature or code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants