E openmm.OpenMMException: NonbondedForce: The cutoff distance cannot be greater than half the periodic box size.

[mikemhenry]

Two days ago we started getting this error on nightly
https://github.com/choderalab/perses/runs/5367830989?check_suite_focus=true#step:9:293

openmm.OpenMMException: NonbondedForce: The cutoff distance cannot be greater than half the periodic box size

Which is causing these tests to fail:

FAILED perses/tests/test_relative.py::test_position_output - openmm.OpenMMExc...
FAILED perses/tests/test_relative.py::test_generate_endpoint_thermodynamic_states
FAILED perses/tests/test_relative.py::test_HybridTopologyFactory_energies - o...
FAILED perses/tests/test_relative.py::test_RepartitionedHybridTopologyFactory_energies
FAILED perses/tests/test_relative_point_mutation_setup.py::test_PointMutationExecutor
FAILED perses/tests/test_relative_point_mutation_setup.py::test_PointMutationExecutor_endstate_validation
FAILED perses/tests/test_rest.py::test_bookkeeping - openmm.OpenMMException: ...
FAILED perses/tests/test_rest.py::test_energy_scaling - openmm.OpenMMExceptio...
FAILED perses/tests/test_resume.py::test_resume_protein_mutation_no_checkpoint
FAILED perses/tests/test_topology_proposal.py::test_protein_counterion_topology_fix_positive
FAILED perses/tests/test_topology_proposal.py::test_protein_counterion_topology_fix_negitive
FAILED perses/tests/test_topology_proposal.py::test_protein_counterion_topology_fix_zero
FAILED perses/tests/test_examples.py::test_examples[/home/runner/work/perses/perses/examples/protein-neq-switching/run_example.py]

Which is a bit confusing since that line of code has been in openmm for 12 years:
https://github.com/openmm/openmm/blame/fd1cfdd60ccc883d4b811da96b823cb403182635/openmmapi/src/NonbondedForceImpl.cpp#L130

But looking at merged PRs and the dates, this may be the culprit:

openmm/openmm#3480

Any ideas @jchodera
It looks like this is enough to trigger the error:
topology_proposal, old_positions, new_positions = utils.generate_solvated_hybrid_test_topology()

[jchodera]

It's almost certainly due to this change, and the new box sizes we are creating are stochastically shrinking to smaller than twice the cutoff under the action of MonteCarloBarostat or are somehow starting at less than twice the cutoff.

I don't quite understand how this change would cause such a significant difference in box sizes, however.

[jchodera]

@peastman: Am I understanding these lines correctly, such that if we had a dense sphere of radius radius full of atoms, the old method would have set the box width to 2*radius+2*padding, while the current method sets the box width to 2*radius+1*padding, shrinking the box by padding over the previous version?

Do we want to preserve the old behavior in this case? Or do we want to try to force people to update how they use padding in their codes?

[peastman]

That's correct. The new method can produce a box that's either larger or smaller than the previous method. For a dense sphere it's now smaller. But for a dense cube of size d it would previously have chosen a box width of d+2*padding, while now it's sqrt(3)*d+padding, which will usually be larger (depending on d, of course).

Previously padding didn't have a clear meaning, aside from the (somewhat arbitrary) expression for box size it appeared in. Now it has a much clearer meaning: we guarantee that no rotation of the solute can bring any atom of one periodic copy closer than padding to any atom of another periodic copy. Also, previously the box size depended strongly on the orientation of the solute. Now it doesn't.

[jchodera]

What if we also revise the box size error to give useful hints that you should increase your box size or padding, and that the definition recently changed?