downloader.py sets method and basis non-deterministically #44

leifjacobson · 2022-09-28T21:20:13Z

I'm was trying to reproduce the energies in a few examples of your SPICE dataset as downloaded with downloader.py. I was unable to successfully reproduce the energies and after some debugging discovered that spec['method'] and spec['basis'] was set to b3lyp and tzvp respectively. I further found that repeated invocations of downloader.py can result in setting these values in a non-deterministic way. Since these values are retrieved inside a loop over subsets I suspect this could also result in a mixed dataset.

I'm downloading again after hardcoding the appropriate values and will try to reproduce the energies again. Regardless of outcome I suggest this is investigated and fixed.

The text was updated successfully, but these errors were encountered:

peastman · 2022-09-28T21:44:59Z

@jchodera @pavankum is this caused by #39? If so it needs to be fixed urgently. QCPortal should always give the high quality results by default unless someone specifically requests the low quality ones.

pavankum · 2022-09-28T21:56:31Z

I couldn't reproduce the error, I see the zeroth element is always spice-default for me. We can change this line though, https://github.com/openmm/spice-dataset/blob/main/downloader/downloader.py#L81, to be more specific and avoid any such errors.

I ran

datasets = ['SPICE Solvated Amino Acids Single Points Dataset v1.1', 'SPICE Dipeptides Single Points Dataset v1.2', 'SPICE DES Monomers Single Points Dataset v1.1', 'SPICE DES370K Single Points Dataset v1.0', 'SPICE DES370K Single Points Dataset Supplement v1.0', 'SPICE PubChem Set 1 Single Points Dataset v1.2', 'SPICE PubChem Set 2 Single Points Dataset v1.2', 'SPICE PubChem Set 3 Single Points Dataset v1.2', 'SPICE PubChem Set 4 Single Points Dataset v1.2', 'SPICE PubChem Set 5 Single Points Dataset v1.2', 'SPICE PubChem Set 6 Single Points Dataset v1.2', 'SPICE Ion Pairs Single Points Dataset v1.1']
for dataset in datasets:
    ds = client.get_collection("Dataset", dataset)
    spec = ds.list_records().iloc[0].to_dict()
    print(dataset)
    print(spec)

and the output shows:

SPICE Solvated Amino Acids Single Points Dataset v1.1
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE Dipeptides Single Points Dataset v1.2
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE DES Monomers Single Points Dataset v1.1
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE DES370K Single Points Dataset v1.0
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE DES370K Single Points Dataset Supplement v1.0
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default_no_mbis', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default_no_mbis'}
SPICE PubChem Set 1 Single Points Dataset v1.2
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE PubChem Set 2 Single Points Dataset v1.2
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE PubChem Set 3 Single Points Dataset v1.2
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE PubChem Set 4 Single Points Dataset v1.2
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE PubChem Set 5 Single Points Dataset v1.2
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE PubChem Set 6 Single Points Dataset v1.2
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
SPICE Ion Pairs Single Points Dataset v1.1
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}

@leifjacobson Can you please drop the version of qcportal you are using.

cc: @bennybp

leifjacobson · 2022-09-28T21:59:50Z

>>> import qcportal
>>> qcportal.__version__
'v0.15.8'

peastman · 2022-09-28T22:01:50Z

We can change this line though

The downloader script is versioned as part of the repository. Even if we change it for future versions, someone who wants v1.1 of the dataset is going to download or check out that version of the repository and run the corresponding version of the script. So we need to make sure it works correctly with the existing script.

leifjacobson · 2022-09-28T22:07:03Z

Seems consistent within invocations but can change between them?

(schrodinger_venv) -bash-4.2$ cat test.py
from qcportal import FractalClient
dataset = 'SPICE Solvated Amino Acids Single Points Dataset v1.1'
for i in range(10):
    client = FractalClient()
    ds = client.get_collection("Dataset", dataset)
    spec = ds.list_records().iloc[0].to_dict()
    print(spec)
(schrodinger_venv) -bash-4.2$ python test.py 
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
(schrodinger_venv) -bash-4.2$ python test.py 
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
(schrodinger_venv) -bash-4.2$ python test.py 
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
(schrodinger_venv) -bash-4.2$ python test.py 
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd', 'keywords': 'spice_default', 'name': 'WB97M-D3BJ/def2-tzvppd-spice_default'}
(schrodinger_venv) -bash-4.2$ python test.py 
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}
{'driver': 'gradient', 'program': 'psi4', 'method': 'b3lyp', 'basis': 'dzvp', 'keywords': 'openff-default', 'name': 'B3LYP/dzvp-openff-default'}

pavankum · 2022-09-28T22:15:59Z

Thank you for the report @leifjacobson. This is in @bennybp 's territory, I will let him know on slack.

Even if we change it for future versions, someone who wants v1.1 of the dataset is going to download or check out that version of the repository and run the corresponding version of the script.

We should make that version inaccessible and make a minor release then (v1.1.1), unfortunately I don't see any other way. @dotsdl or @jthorton can you take a look and confirm when you can.

jchodera · 2022-09-28T22:18:51Z

@peastman : If the download script does not hard-code the spec, @pavankum's suggestion of hard-coding this to always get the desired version is the right way to go.

The downloader script is versioned as part of the repository. Even if we change it for future versions, someone who wants v1.1 of the dataset is going to download or check out that version of the repository and run the corresponding version of the script. So we need to make sure it works correctly with the existing script.

You can erase the tag and re-release it, but I'd suggest a simple 1.1.1 bugfix with a note in the release notes that this eliminates the non-determinism error would be sufficient.

peastman · 2022-09-28T22:36:35Z

Can you make a PR with the fix? I'm not sure exactly what it is.

peastman · 2022-09-28T23:22:51Z

@pavankum or @jchodera could one of you create a PR with the fix ASAP? This is extremely urgent, since at this moment there could easily be people running the downloader and getting corrupted versions of the dataset.

peastman · 2022-09-29T05:06:33Z

The fix is merged and the 1.1.1 release is up. Thanks @pavankum for the very fast response.

pavankum mentioned this issue Sep 28, 2022

Update spec #46

Closed

peastman mentioned this issue Sep 29, 2022

Ensure downloader selects the correct level of theory #47

Merged

peastman closed this as completed in #47 Sep 29, 2022

peastman mentioned this issue Oct 12, 2022

Cannot reproduce wB97M-D3(BJ) energies and forces #50

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

downloader.py sets method and basis non-deterministically #44

downloader.py sets method and basis non-deterministically #44

leifjacobson commented Sep 28, 2022

peastman commented Sep 28, 2022

pavankum commented Sep 28, 2022

leifjacobson commented Sep 28, 2022

peastman commented Sep 28, 2022

leifjacobson commented Sep 28, 2022 •

edited

Loading

pavankum commented Sep 28, 2022

jchodera commented Sep 28, 2022

peastman commented Sep 28, 2022

peastman commented Sep 28, 2022

peastman commented Sep 29, 2022

downloader.py sets method and basis non-deterministically #44

downloader.py sets method and basis non-deterministically #44

Comments

leifjacobson commented Sep 28, 2022

peastman commented Sep 28, 2022

pavankum commented Sep 28, 2022

leifjacobson commented Sep 28, 2022

peastman commented Sep 28, 2022

leifjacobson commented Sep 28, 2022 • edited Loading

pavankum commented Sep 28, 2022

jchodera commented Sep 28, 2022

peastman commented Sep 28, 2022

peastman commented Sep 28, 2022

peastman commented Sep 29, 2022

leifjacobson commented Sep 28, 2022 •

edited

Loading