Journal suggestion bias towards certain journals

[jdagdelen]

I was just playing around with this and it seems like the Journal of Electroceramics is suggested for a ton of abstracts in our example set. @kevinyang8 It seems like there may be some biasing effects that we should work on. I don't this this disqualifies the app, but we should put a disclaimer on it and describe how the suggestions are calculated.

[jdagdelen]

We might want to do some benchmarking and see if there are any journals that should be downsampled. Maybe run the prediction for 10,000 abstracts from various journals and see if there are any journals that turn up more than we'd expect?

[computron]

@jdagdelen are there clear examples of abstracts that don't belong to that journal, but for which that journal is predicted?

[computron]

@kevinyang8 one easy test you can do to test for journal bias is to look at your test set, and see whether certain journals have larger errors than others.

e.g., let's say that abstracts in the test set which should be in "Nature Materials" list Nature Materials as one of the top 3 predictions 80% of time. But abstracts in the test set which should be in "Physical Review B" list that journal as one of the top 3 predictions only 20% of time. This would be evidence of bias against Physical Review B.

One reason for journal bias could be imbalanced data classes. e.g., if you have 500 abstracts in the test set for Nature Materials, and 20 abstracts for Physical Review B, then the algorithm will get loewr errors on the test set by guessing Nature Materials more frequently