Improve inchikey pair selection and data generators

[niekdejonge]

Refactoring of the InchikeyPairSelection and datagenerators.

Changes made:

• Made DataGenerator and SelectedCompoundPairs less intertwined. SelectedCompoundPairs just contains the inchikey pairs and the dataGenerator just adds the spectrum picking and tensorization.
• Fixed the bug that only one spectrum per inchikey was used during training
• Sampling of inchikeys is more balanced, the algorithm is now like this:
  • First tanimoto scores are collected per tanimoto score bin a 100 pairs are selected for each inchikey (if available). The 100 pairs per inchikey is used to keep the algorithm scalable and not run into memory issues, in this way memory scales linearly instead of quadratically.
  • Pairs are converted from a matrix to just a list of pairs in the form (inchikey1, inchikey2, score). Any inverted duplicates are removed (e.g. (inchikey2, inchikey1, score)).
  • We count the available number of pairs per bin. The number of pairs in the bin that has the least number of bins is used as pairs per bin, to ensure a good balance over the different bins.
  • We start selecting pairs per bin from the bin with the lowest number of pairs to the highest number of pairs.
  • We track the frequency of each inchikey in all pairs over all bins. (both the first and the second inchikey in each pair is counted, which was not the case before).
  • When picking pairs we start with the inchikey with the lowest frequency that is available for that tanimoto bin. We select all available pairs for this inchikey in this bin. There can be mulitple pairs available. (e.g. [(inchikey5, inchikey3, 0.9), (inchikey5, inchikey8, 0.92), (inchikey5, inchikey6, 0.91)]. From these pairs we pick the pair were the second inchikey is the least frequent.
  • This results in a list of pairs, that have a perfect balance over the different bins, with an almost perfect balance of unique inchikeys.

Additional changes:

• I have improved the tests, we now test the balance of inchikeys, the balance of scores, and that we actually select all spectra (and not just one per inchikey).
• Before an epoch would be exactly nr_of_inchikeys long. Even if it did not fit with the batch size. This could result in the final batch being smaller than the batch_size. Now I have changed this to just using the batch size for the final batch size as well, resulting in an epoch that takes a few more pairs than the nr_of_inchikeys.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud