stop_when_dft_decayed in single precision

[stevengj]

The current stop_when_dft_decayed implementation seems like it may be problematic in single precision, as @mochen4 noticed in #1830.

In particular, it checks whether | ‖DFT‖ - ‖prev DFT‖ | < tol max(‖DFT‖) for a default tolerance of tol = 1e-11, which seems to be trying to detect a fractional change less than the single precision. On the other hand, while the DFT fields are single precision, the norm is computed in double precision.

We might want to switch to the original approach that I suggested #1740 (comment), which computes the norm of the fields contributing to the DFT, rather than the change of the norm of the DFT. That has the advantage that it avoids the ill-conditioned step of taking a difference of two nearly equal quantities.

cc @smartalecH

[smartalecH]

@mochen4 here is the code for the above implementation. It was originally added in this commit of #1740, but later removed. Feel free to try it out (you'll have to derive a new heuristic for the field convergence).

/* Return the L2 norm of all of the fields used to update DFTs.  This is useful
   to check whether the simulation is finished (whether all relevant fields have decayed).
   (Collective operation.) 
   Note that this is different from above, where we compute the norm of the *actual*
   dft fields. In this case, we are only looking at the *update* (e.g. independent
   of the phase term in the DTFT inner product). This is a more conservative approach
   that by definition looks at *all* the simulation's frequencies (not just the ones
   we care about).  
*/

double fields::dft_time_fields_norm() {
  am_now_working_on(Other);
  double sum = 0.0;
  for (int i = 0; i < num_chunks; i++)
    if (chunks[i]->is_mine()) sum += chunks[i]->dft_fields_norm2();
  finished_working();
  return std::sqrt(sum_to_all(sum));
}

double fields_chunk::dft_time_fields_norm2() const {
  double sum = 0.0;
  for (dft_chunk *cur = dft_chunks; cur; cur = cur->next_in_chunk)
    sum += cur->dft_fields_norm2();
  return sum;
}

double dft_chunk::dft_time_fields_norm2() const {
  if (!fc->f[c][0]) return 0.0;
  int numcmp = fc->f[c][1] ? 2 : 1;
  double sum = 0.0;
  LOOP_OVER_IVECS(fc->gv, is, ie, idx) {
    if (avg2)
      for (int cmp = 0; cmp < numcmp; ++cmp)
        sum += sqr(0.25 * (fc->f[c][cmp][idx] + fc->f[c][cmp][idx + avg1] +
                           fc->f[c][cmp][idx + avg2] + fc->f[c][cmp][idx + (avg1 + avg2)]));
    else if (avg1)
      for (int cmp = 0; cmp < numcmp; ++cmp)
        sum += sqr(0.5 * (fc->f[c][cmp][idx] + fc->f[c][cmp][idx + avg1]));
    else
      for (int cmp = 0; cmp < numcmp; ++cmp)
        sum += sqr(fc->f[c][cmp][idx]);
  }
  return sum;
}

When storing the current update,

meep/python/simulation.py

Line 4549 in 03ea7ad

current_fields = _sim.fields.dft_norm()

just be sure to use this new function instead.

[ianwilliamson]

Would using the decay of the instantaneous energy be a reasonable approach here?

Specifically, at each measurement integrate the instantaneous energy over the full sim volume. Compare this value to the maximum value seen across all measurements so far (i.e. the peak total energy). Terminate once the ratio drops below some threshold.

[stevengj]

The point of my suggested approach is that if you are computing the DFT, you are mainly interested in the field strength in the regions where you are computing the DFT. But, yes, looking at the total energy is not unreasonable too.