Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#634: Scoring overhaul #163

Merged
merged 3 commits into from
Aug 8, 2023
Merged

#634: Scoring overhaul #163

merged 3 commits into from
Aug 8, 2023

Conversation

ericz1803
Copy link
Contributor

@ericz1803 ericz1803 commented Aug 2, 2023

biothings/biothings_explorer#634

Benchmarks comparison of old scoring system vs new scoring system:

Old:

Benchmark: treats
                        k=1     k=5     k=10    k=20
Precision @ k           0.0000  0.0273  0.0318  0.0295
Recall @ k              0.0000  0.0714  0.1667  0.3095
mAP @ k                 0.0000  0.0273  0.0535  0.0597
Top-k Accuracy          0.0000  0.0714  0.1667  0.3095

Mean Reciprocal Rank    0.06169455544455544

Benchmark: DrugCentral_creative
                        k=1     k=5     k=10    k=20
Precision @ k           0.2600  0.0680  0.0360  0.0200
Recall @ k              0.2600  0.3400  0.3600  0.4000
mAP @ k                 0.2600  0.2867  0.2900  0.2930
Top-k Accuracy          0.2600  0.3400  0.3600  0.4000

Mean Reciprocal Rank    0.2929946524064171

New:

Benchmark: treats
                        k=1     k=5     k=10    k=20    k=50    k=100   k=200   k=500   k=1000
Precision @ k           0.2273  0.1364  0.0864  0.0523  0.0264  0.0145  0.0075  0.0031  0.0015
Recall @ k              0.1190  0.3571  0.4524  0.5476  0.6905  0.7619  0.7857  0.8095  0.8095
mAP @ k                 0.2273  0.2449  0.2550  0.2631  0.2707  0.2722  0.2726  0.2728  0.2728
Top-k Accuracy          0.1190  0.3571  0.4524  0.5476  0.6905  0.7619  0.7857  0.8095  0.8095

Mean Reciprocal Rank    0.314370593945062

Benchmark: DrugCentral_creative
                        k=1     k=5     k=10    k=20    k=50    k=100   k=200   k=500   k=1000
Precision @ k           0.4200  0.1120  0.0640  0.0370  0.0164  0.0088  0.0045  0.0019  0.0009
Recall @ k              0.4200  0.5600  0.6400  0.7400  0.8200  0.8800  0.9000  0.9400  0.9400
mAP @ k                 0.4200  0.4767  0.4890  0.4959  0.4986  0.4995  0.4996  0.4997  0.4997
Top-k Accuracy          0.4200  0.5600  0.6400  0.7400  0.8200  0.8800  0.9000  0.9400  0.9400

Mean Reciprocal Rank    0.4997344748094153

@tokebe tokebe merged commit 32eb684 into main Aug 8, 2023
0 of 2 checks passed
@tokebe tokebe deleted the scoring-overhaul branch November 29, 2023 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants