Formal Integral: including electron scattering #759

unoebauer · 2017-07-04T12:17:52Z

The goal of this PR is the inclusion of the electron scattering into the Formal Integral scheme, introduced by @yeganer in PR #740.

For this purpose, the mean intensity in the blue and red wings of each line transition have to be calculated (see Eqs. 23, 24 and 28 in Lucy 1999) and passed to the Formal Integral integrator. The most obvious place for this calculation is alongside the determination of the attenuated line source functions att_S_ul in the routine make_source_function in formal_integral.py.

The intensities have to be passed to the formal integrator (alongside att_S_ul). In the integrator, the recursive stepping through the line list has to be adjusted to include the electron scattering contribution (see Eqs. 26, 27 and 28 in Lucy 1999). For this purpose, the electron scattering optical depth accumulated along the constant p paths has to be calculated.

Finally, unit tests for these additions have to be devised, the sanity checks revised and the integration with the configuration system checked.

Note: Even after this change, the Formal Integral will only work with the downbranching scheme for now.

Milestones:

calculate Jblues_lu and Jreds_lu
calculate electron scattering optical depths on integration segments
adjust recursive integration along z for const-p rays
properly account for the edges of the line list
properly treat e-scattering effect across shell interfaces (not only record dtau but dtau * Jkkp)
add unit tests
verification with single and multiline setups
update documentation

* in preparation for the inclusion of electron scattering into the formal integral formalism, the Jblues are calculated * the calculation occurs in the same routine in which att_S_ul is prepared

* WARNING: somehow I broke the integration * Jred and Jblue are now passed to the integrator * some preparatory steps for the e-scattering optical depths have been made

unoebauer · 2017-07-04T15:45:04Z

Somehow, I broke the integration with the last commit. Any ideas why, @yeganer?

yeganer · 2017-07-04T15:59:13Z

tardis/montecarlo/src/integrator.c

@@ -258,11 +270,23 @@ _formal_integral(
                       ++patt_S_ul)
                    {
                      if (*pline < nu_end)
+                        // Calculate e-scattering optical depth to grid cell boundary
+                        zend = storage->time_explosion / C_INV * (1. - nu_end / nu);
+                        dtau += (zend - zstart) * escat_op;
                        break;


This break get's called every loop because the if statement is not followed by a scope {}

Yeah - my bad! Coding too much in python lately...

yeganer · 2017-07-04T15:59:39Z

tardis/montecarlo/src/integrator.c

                      I_nu_r[p_idx] = I_nu_b[p_idx] * (*pexp_tau) + *patt_S_ul;
                      // In the absence of electron scattering, simple recursion as in Lucy 1991
                      // TODO: e-scattering: Replace by Lucy 1999, Eq. 27
                      I_nu_b[p_idx] = I_nu_r[p_idx];
+
+                      // reset e-scattering opacity 
+                      dtau = 0;


I'm not sure why you reset dtau every iteration.

Because dtau measures the accumulated e-scattering optical depth between consecutive resonances. I still have to modify the calculation of I_nu_b, where this quantity is then used. Afterwards it has to be reset.

yeganer · 2017-07-04T16:00:26Z

tardis/montecarlo/src/integrator.c

@@ -258,11 +270,23 @@ _formal_integral(
                       ++patt_S_ul)
                    {
                      if (*pline < nu_end)


This is a second loop end condition because it's not easy to do that nicely

yeganer · 2017-07-04T16:04:09Z

tardis/montecarlo/src/integrator.c

                  // Initialize pointers for inner loop
                  pline = storage->line_list_nu + idx_nu_start;
                  pexp_tau = exp_tau + offset + idx_nu_start;
                  patt_S_ul = att_S_ul + offset + idx_nu_start;
+                  pJred_lu = Jred_lu + offset + idx_nu_start;
+                  pJblue_lu = Jblue_lu + offset + idx_nu_start;

                  for (;pline < storage->line_list_nu + size_line;


maybe I can change pline < storage->line_list_nu + size_line to
(pline < storage->line_list_nu + size_line) & (*pline < nu_end)

Alternatively one could add idx_nu_end and then change the loop condition to pline < storage->line_list_nu + idx_nu_end.

* these modifications do not change the accurate performance of the formal integral method when e-scattering is deactivated * HOWEVER: some safety checks are still missing! Use with caution!

unoebauer · 2017-07-05T10:28:37Z

Seems to work pretty well already:

wkerzendorf · 2017-07-05T12:31:48Z

@unoebauer that looks very good!

* bookkeeping for e-scattering contribution should now be correct

yeganer · 2017-07-06T08:26:15Z

tardis/montecarlo/src/integrator.c


  double R_ph = storage->r_inner[0];
  double R_max = storage->r_outer[size_shell - 1];
  double pp[N];
  double *exp_tau = calloc(size_tau, sizeof(double));
-#pragma omp parallel firstprivate(L, exp_tau)
+#ifdef WITHOPENMP


For the formal integral openmp parallelization I chose a different approach. By providing a dummy header, we can limit #ifdef WITHOPENMP to the header part of the document and I would strongly advise against including it anywhere in the body.

I know this implementation is basically a copy/paste of my progress from cmontecarlo.c but I think we are now able to provide a more readable solution.

yeganer · 2017-07-06T08:29:06Z

tardis/montecarlo/src/integrator.c


  double R_ph = storage->r_inner[0];
  double R_max = storage->r_outer[size_shell - 1];
  double pp[N];
  double *exp_tau = calloc(size_tau, sizeof(double));
-#pragma omp parallel firstprivate(L, exp_tau)
+#ifdef WITHOPENMP
+#pragma omp parallel firstprivate(L, exp_tau, finished_nus)


If we make finished_nus shared we can make the bottom printing much nicer

Streamline the OMP implementation

unoebauer · 2017-07-10T16:01:33Z

Unfortunately, and despite the excellent results obtained with tardis_example, the current formal integral scheme goes completely mad in a simple pure hydrogen atmosphere, in which only the Halpha is active. This is independent of whether electron scattering is active or not.

I'll have a closer look at it tomorrow.

yeganer · 2017-07-10T17:36:41Z

@unoebauer Does this also apply to the version in the current master or only to this PR?

yeganer · 2017-07-10T18:25:19Z

Okay, I think I found two bugs.

We use line_search to determine idx_nu_start. The output of line_search is the index of the next line to the red, or no_of_lines if the value is redder than the reddest line. That means, *pline becomes a pointer to uninitialized memory and ALL 2D arrays point to the first line in the next shell.
We should really think about ending the line list with a 0.
If there is no line interaction AT ALL for one p ray, then I_nu_r never get's updated, that means the intensity is 0 and not related to the value we initialized I_nu_b with.

yeganer · 2017-07-10T18:57:16Z

tardis/montecarlo/src/integrator.c

                    }
                }
-              I_nu[p_idx] *= p;
+              I_nu_r[p_idx] *= p;


If nu is redder than the reddest line I_nu_r[p_idx] will always be 0. So we have to account for the fact, that there may not be a line interaction. I propose to initialize I_nu_r as it physically does make sense in my opinion. I_nu_r is the intensity after a interaction. We assume a photosphere and emit radiation there. I think it makes sense to say "After they leave the photosphere there are at the red wing of some transition that happens at the boundary of the photosphere".
For the case that the ray does not intersect the photosphere, we initialize to 0. Here again I think we can apply the argument that we are at the red wing of a hypothetical transition after which our intensity is 0. Then we propagate through matter and apply the electron scattering.

Does this interpretation make sense? It would at least solve the problem and make the code a little bit easier (only Jkk has to be calculated in the first block)

What's left is to decide what happens in terms of electron scattering if there is no resonance.

Swapping the initialization to I_nu_r and changing the code in first does produce the correct spectrum for disabled electron scattering.

…ardis into formal_integral_add_escatter

unoebauer · 2017-07-12T15:06:05Z

I've now reintroduced I_nu and thus avoided the problem of an uninitialised integration in case of no resonances on the ray. With this change the simple H-atmosphere problem works again when electron scattering is switched off. Once activated, I still get a terrible match between formal integral and virtual packet spectrum.

The problem with the line_list_search is still present, but as I will detail in the next comment, the problems in the simple H-atmosphere with electron scattering are more fundamental and cannot be resolved with the current information we are extracting from the Monte Carlo simulation.

Pragmatically, however, the current state of the Formal Integral simulation should work for all realistic supernova calculations (the formal integral matches the virtual spectrum very well in the tardis example setup, both with and without e-scattering). Since we will typically have a very large line list once a realistic supernova ejecta is set up, we will over the entire spectrum not reach the boundaries of this list and thus never run into the numerical problems associated with the line_list_search and also not into the conceptual problems outline below.

unoebauer · 2017-07-12T15:14:01Z

In order to estimate the effect of electron scattering on the various rays, we need to know the strength of the diffuse radiation field at the LF frequency of the frequency bin we are interested in. Lucy 1999 estimates the diffuse radiation field by considering the mean intensity in the blue and red wings of the next and previous lines. This is justified as long as the current LF frequency is not too far from the frequencies of the lines.

It is however a problem for applications like the H-atmosphere with only H-alpha. Here, most parts of the spectrum are far beyond the frequency of this line and the diffuse radiation field cannot be accurately estimated by considering the intensity in the blue or red wing of the H-alpha line.

To correctly calculate the e-scattering contribution in such cases a J_nu estimator would have to be recorded in the Monte Carlo simulation. In principle this is possible (@chvogl has done this already in his Type IIP branch) but associated with additional effort.

unoebauer · 2017-07-26T15:51:35Z

@wkerzendorf, @yeganer, @chvogl: please have a look at this PR. I think this is as far as we can go without introducing additional estimators.

chvogl · 2017-07-27T08:57:43Z

tardis/montecarlo/src/integrator.c

+                        // conditions) is not in Lucy 1999; should be
+                        // re-examined carefully 
+                        escat_contrib += (zend - zstart) * escat_op * (*pJblue_lu - I_nu[p_idx]) ;
+                        I_nu[p_idx] = I_nu[p_idx] + escat_contrib;


If you want you can move this out of the if statement.

chvogl · 2017-07-27T08:58:07Z

tardis/montecarlo/src/integrator.c

+                        // Account for e-scattering, c.f. Eqs 27, 28 in Lucy 1999
+                        Jkkp = 0.5 * (*pJred_lu + *pJblue_lu);
+                        escat_contrib += (zend - zstart) * escat_op * (Jkkp - I_nu[p_idx]) ;
+                        I_nu[p_idx] = I_nu[p_idx] + escat_contrib;


If you want you can move this out of the if statement.

wkerzendorf · 2017-07-31T12:58:58Z

tardis/montecarlo/formal_integral.py

@@ -124,4 +128,9 @@ def make_source_function(self):
        att_S_ul =  ( wave * (q_ul * e_dot_u) * t  / (4*np.pi) )

        result = pd.DataFrame(att_S_ul.as_matrix(), index=transitions.transition_line_id.values)
-        return result.ix[atomic_data.lines.index.values].as_matrix(), e_dot_u
+        att_S_ul = result.ix[atomic_data.lines.index.values].as_matrix()


if you just do .values that is maybe clearer that you just want the array and not as a real matrix.

actually, this is on you :) You've implemented this ages ago (SOCIS 2016) and I've just shuffled things aroun

Add calculation of Jblues to Formal Integral

c9748a7

* in preparation for the inclusion of electron scattering into the formal integral formalism, the Jblues are calculated * the calculation occurs in the same routine in which att_S_ul is prepared

unoebauer added the work in progress 🚧 label Jul 4, 2017

unoebauer requested a review from wkerzendorf July 4, 2017 12:22

unoebauer added 3 commits July 4, 2017 14:46

Fix units bug, add calculation of Jreds

628418f

Introduce I_nu_r and I_nu_b in prep. for e-scat.

793690b

Pass Jred and Jblue to integrator

6b9ad27

* WARNING: somehow I broke the integration * Jred and Jblue are now passed to the integrator * some preparatory steps for the e-scattering optical depths have been made

yeganer reviewed Jul 4, 2017

View reviewed changes

unoebauer added 2 commits July 5, 2017 11:28

Fix bug in integrator.c from in previous commit

c530e4b

Modify calculation of I_nu_b to include e-scat.

059b6c6

* these modifications do not change the accurate performance of the formal integral method when e-scattering is deactivated * HOWEVER: some safety checks are still missing! Use with caution!

Remove check for e-scattering opacity in FI

c495561

unoebauer added 3 commits July 5, 2017 16:59

Improve and reorganize formal integral integration

f7b1bd7

* bookkeeping for e-scattering contribution should now be correct

Add status printout for formal integral

62125c4

Correct status printout for FI

5753352

yeganer reviewed Jul 6, 2017

View reviewed changes

yeganer and others added 2 commits July 6, 2017 10:53

Streamline the OMP implementation

a3165bb

Merge pull request #1 from yeganer/fix_omp_progress

1f1593f

Streamline the OMP implementation

yeganer reviewed Jul 10, 2017

View reviewed changes

unoebauer added 2 commits July 12, 2017 16:56

Reintroduce I_nu to avoid problems

e9a39e8

Merge branch 'formal_integral_add_escatter' of github.com:unoebauer/t…

4b6e988

…ardis into formal_integral_add_escatter

unoebauer requested a review from chvogl July 12, 2017 15:17

unoebauer added 2 commits July 26, 2017 16:38

Add Jkk to escatter_contrib calculation

28006f0

Update documentation

9c334e0

chvogl approved these changes Jul 27, 2017

View reviewed changes

Remove redundancies in integrator.c

d6a27de

wkerzendorf reviewed Jul 31, 2017

View reviewed changes

Remove as_matrix following @wkerzendorf

5863e4b

wkerzendorf merged commit 96a94b5 into tardis-sn:master Aug 1, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Formal Integral: including electron scattering #759

Formal Integral: including electron scattering #759

unoebauer commented Jul 4, 2017 •

edited

Loading

unoebauer commented Jul 4, 2017

yeganer Jul 4, 2017

unoebauer Jul 5, 2017

yeganer Jul 4, 2017

unoebauer Jul 5, 2017

yeganer Jul 4, 2017

yeganer Jul 4, 2017

unoebauer commented Jul 5, 2017

wkerzendorf commented Jul 5, 2017

yeganer Jul 6, 2017

yeganer Jul 6, 2017

unoebauer commented Jul 10, 2017

yeganer commented Jul 10, 2017 •

edited

Loading

yeganer commented Jul 10, 2017

yeganer Jul 10, 2017

unoebauer commented Jul 12, 2017

unoebauer commented Jul 12, 2017 •

edited

Loading

unoebauer commented Jul 26, 2017

chvogl Jul 27, 2017

chvogl Jul 27, 2017

wkerzendorf Jul 31, 2017

unoebauer Jul 31, 2017

Formal Integral: including electron scattering #759

Formal Integral: including electron scattering #759

Conversation

unoebauer commented Jul 4, 2017 • edited Loading

unoebauer commented Jul 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unoebauer commented Jul 5, 2017

wkerzendorf commented Jul 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unoebauer commented Jul 10, 2017

yeganer commented Jul 10, 2017 • edited Loading

yeganer commented Jul 10, 2017

Choose a reason for hiding this comment

unoebauer commented Jul 12, 2017

unoebauer commented Jul 12, 2017 • edited Loading

unoebauer commented Jul 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unoebauer commented Jul 4, 2017 •

edited

Loading

yeganer commented Jul 10, 2017 •

edited

Loading

unoebauer commented Jul 12, 2017 •

edited

Loading