Deviation of energies and gradients in test suite

[pultar]

Describe the bug

Values hard-coded in the test suite do not match what the xtb binary produces for me with external embedding.

xtb/test/api/c_api_example.c

Line 276 in 1922dd3

if (!check(pcgrad[0], 0.00000755, 1.0e-6, "pcgrad[0] does not match"))

Deviations are relatively small, but too high to think of rounding errors. The test during build passes, however, and also my minimal example works as before. Upon more investigation, I found that gradients and energies are also inconsistent between C API and xtb binary.

Importantly, I don't get any deviation between C API and xtb binary, when I ran a calculation without embedding (12 significant digits).

This might very well be an input error from my side, however, I would like to ask you to have a look. I will also open a PR with some more tests of the C API that will cover energies and gradients calculated under charge embedding conditions.

To Reproduce

I compiled all files necessary to reproduce this behavior in a repo (https://github.com/pultar/xtb-example). I also attached them for your convenience:

xtb-example-master.zip

XTB binary and library are from a fresh compilation of the master branch with gfortran and openblas.

Output of my program using C API: https://github.com/pultar/xtb-example/blob/master/example.out

Output of xtb binary: https://github.com/pultar/xtb-example/blob/master/test-bohr/xtb.out

Expected behaviour
I expect xtb binary to produce the same energies, gradients, and point charge gradients as the C API does (and I remember, it did).

[pultar]

Did you have a chance to look at my results and have any idea what is going on here? I find it quite concerning that the energy / gradients calculated differ between C API and binary. Especially the point charge gradients are off by a factor of ~20 (!)

Attached are some results as .csv. Could it be how I call xtb (https://github.com/pultar/xtb-example/blob/master/test-bohr/xtb.sh)?

In any case, the unit tests should be adapted accordingly to reflect this behavior.

xtb-comparison.csv

[pultar]

I added two PR's, one with values generated by the C API (#718) and one with values generated by the binary (#719). Please note that the second test fails.

[MtoLStoN]

I can reproduce this Issue using the C-API, the bleeding edge version, and our binary release. This needs some further checking. Thanks for bringing this to our attention.

[pultar]

I added in both PR's a bigger system to demonstrate the gravity of the problem at scale. If there are more point charges, the point charge gradient can even change sign.

Thanks for looking into it! Let me know if I can be of help.

// as a minor notice: I also observed a difference in energies, gradients, ... if I choose atomic symbols instead of atomic numbers in the .pc file. The differences seem less pronounced than they are between C API and binary.

[pultar]

I did some more digging and found more discrepancies, this time between the xtb binary and Fortran unit tests:

xtb/test/unit/test_gfn2.f90

Line 381 in 36202cd

subroutine test_gfn2_pcem_api(error)

I compared values hard-coded in the Fortran unit tests with values obtained after evaluating (identical) inputs from the documentation (https://xtb-docs.readthedocs.io/en/latest/pcem.html) with the binary or with the C API.

Results are saved here: https://github.com/pultar/water-clusters/tree/master/gfn2 and here: https://github.com/pultar/xtb-example/blob/3582e374638b9dd31e570fd7ad5ae8c279350b4f/example.cc#L211

For your convenience, here is a summary: https://github.com/pultar/water-clusters/blob/master/gfn2/xtb-fortran-unit-tests.pdf

I don't know how the hard-coded values were originally determined. I did it like suggested in the documentation with parameters set in the unit tests: https://github.com/pultar/water-clusters/blob/master/gfn2/atomic-symbols/xtb.sh

Conclusion:

• Without external embedding, xtb binary and Fortran unit tests produce identical results. From previous investigation (see above), we know that C API is also consistent with those results
• With external embedding, however, Fortran unit tests and C API produce identical results with respect to each other for all values compared. The output of the xtb binary though deviates from both the Fortran unit tests and the C API (!).
• this issue seems to concern not only the C API but also embedding in general

It seems like embedding causes some downstream effect not captured in the unit tests. I don't know which values are correct - the values produced by the binary or the values hard-coded in the tests.

Let me know what you think.

[MtoLStoN]

Thanks again for reporting this. There was an issue with how the PC coordinates are read in with the binaries (we wrongly assumed angstrom instead of Bohr, even without the interface=orca keyword.)
PR #720 will fix this.