Adding possibility to turn off covariance matrix calculation also for least squares fitting

[Andrea-gm]

At the moment, from the documentation it seems that it is not possible to turn off the calculation of the covariance matrix at the end of a fitting process for the least square methods. This increases significantly the fitting time when your model has tens of parameters, and it would be desirable to be able to turn this option off.

The calc_covar = False option is already there, but it works only for fitting methods other than least squares. Any suggestion on this? Thanks!

[reneeotten]

Hi Andrea, the covariance matrix is returned by SciPy when using the methods “leastsq” or “least_squares”. So you’ll have to convince SciPy to not calculate that; it seems at least for the “leastsq” method that you can suppress the calculation and return by setting “full_output=0”; see https://github.com/scipy/scipy/blob/92d2a8592782ee19a1161d0bf3fc2241ba78bb63/scipy/optimize/_minpack_py.py#L479 You can pass that option to the solver when you initialize the Minimizer class ; check the documentation and/or examples for more information. Hope that helps, Renee

…

On Dec 6, 2024, at 2:02 PM, Andrea Giunto ***@***.***> wrote: At the moment, from the documentation it seems that it is not possible to turn off the calculation of the covariance matrix at the end of a fitting process for the least square methods. This increases significantly the fitting time when your model has tens of parameters, and it would be desirable to be able to turn this option off. The calc_covar = False option is already there, but it works only for fitting methods other than least squares. Any suggestion on this? Thanks! — Reply to this email directly, view it on GitHub <#977>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC45CIDPXNOADA6MGBSIYBT2EHYD5AVCNFSM6AAAAABTFHPUXCVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXGYZDKMJQHE>. You are receiving this because you are subscribed to this thread.

[Andrea-gm]

Hi Renee,

Thanks so much for the prompt response! It turns out that the long fitting times actually are not coming from the covariance matrix calculation within scipy, but in fact they are coming from the function Minimiser._calculate_uncertainties_correlations()

I added a workaround within the leastsq() method in my pc, but it would be ideal to add an option to avoid calculating the uncertainties.

Also, setting full_output = False throws an error, because in the Minimizer.leastsq() a tuple of 5 elements is expected as output from scipy_leastsq(), while the output only has length of 2 when full_output = False. Probably this should be fixed?

Thanks a lot for your help!

[newville]

@Andrea-gm @reneeotten Yes, to be clear, calc_covar is not used with leastsq and least_squares.

I don't see any reason to use full_output=False.

The added runtime from _calculate_uncertainties_correlations should be small. It is necessary to estimate uncertainties of Parameters that are constraint expressions. I think uncertainties should be estimated if at all possible.

Why is squeezing ultimate performance for any individual fit a higher priority than estimating uncertainties?

[newville]

@Andrea-gm Posting code and timing information would entail posting actual example code that could be run and timed by someone else. Images of code is not code.

Sorry, but your reports are simply not reliable. You said fitting time went from one to five minutes, and then 45 seconds, and then 13 seconds.

It turns out that I fit XRF data from an EDS detector regularly (I run an X-ray microprobe) - I don't typically need 50 variables. There just are not 50 separate quantities that show up in most EDS spectra.

I also (for other X-ray spectroscopies) fit data with numerous constraint expressions (~2x to 3x the number of fitting variables). I don't see such problems. I cannot tell what you are doing (because you have deliberately not shown what you are doing when complaining about performance).

But, if you have 150 constraint expressions, then, yes, you need to get the calculated uncertainties for those. Turning those off would be irresponsible. We are not going to assist you in doing that or condone you doing it on your own. That's an abuse of our work.

[Andrea-gm]

I do not appreciate the aggressive tone that has been present since the early replies. I want to emphasize that I am in no way complaining about the code. On the contrary, I am extremely grateful for your work—lmfit works very well and is incredibly useful.

I did not understand that you wanted the actual code, sorry about that. Please understand that my work is unpublished. With the help of Chatgpt, I have reproduced the behaviour on a little script that uses 15 independent gaussians + 50 dependent ones, which you will find in the next reply. This is very similar to what happens in my code. You will see that scipy_leastsq takes 2.5 secs, while calc_uvar takes 4.5 secs, bringing the total of Model.fit() to 11.7 s. It is easy to scale that to understand why my fit increases from 1 minute to 5 (or from 7.7 secs to 37.7 secs, the scaling is the same for any spectrum). Of course, the timings on your machine may be different, but I would expect the behaviour to be similar. This snippet confirms that the long calculation times in calc_uvar are due to the large number of dependent parameters.

Regarding the EDS spectra, here's an example from a 7-element sample. Each single label corresponds to 1 gaussian peak, which is defined by 3 parameters (center, area, sigma). As such, there are in total some 200 parameters used in this spectrum considering also those used for the background.

On the other hand, I’m having difficulty understanding the strong opposition to allowing users the option to disable uncertainty calculations. While I recognize their utility, in my specific case, they don’t add value since the uncertainties are always smaller than my measurement error. For example, what difference does a 3% fitting error (i.e., worst case scenario I have seen in my peak fits) make when my overall accuracy is only 10%? My work will be published in the coming months and you will be more than welcome to take a look at what metric I am using to evaluate the measurement errors if you are curious.

I would really appreciate if you could reconsider the option of disabling uncertainty calculations, thank you.

[Andrea-gm]

import numpy as np
import matplotlib.pyplot as plt
from lmfit import Model
import time

Gaussian function definition

def gaussian(x, center, sigma, area):
return area / (sigma * np.sqrt(2 * np.pi)) * np.exp(-0.5 * ((x - center) / sigma) ** 2)

Generate the synthetic spectrum

np.random.seed(42) # For reproducibility
x = np.linspace(-10, 50, 1000)

Parameters for the 10 independent Gaussians

num_independent = 15
centers = np.random.uniform(0, 40, num_independent)
sigmas = np.random.uniform(0.5, 2.0, num_independent)
areas = np.random.uniform(5, 20, num_independent)

Generate the independent Gaussian peaks

independent_peaks = [gaussian(x, c, s, a) for c, s, a in zip(centers, sigmas, areas)]

Parameters for 20 dependent Gaussians (linked to independent ones)

num_dependent = 50
dependent_links = [
(np.random.randint(0, num_independent), # Index of linked independent Gaussian
np.random.uniform(-1, 1), # Center offset
np.random.uniform(0.8, 1.2), # Sigma scale factor
np.random.uniform(0.5, 1.5)) # Area scale factor
for _ in range(num_dependent)
]

Generate the dependent Gaussian peaks

dependent_peaks = [
gaussian(x, centers[idx] + offset, sigmas[idx] * scale_sigma, areas[idx] * scale_area)
for idx, offset, scale_sigma, scale_area in dependent_links
]

Combine all peaks and add noise

spectrum = sum(independent_peaks) + sum(dependent_peaks)
noise = np.random.normal(0, 0.2, size=x.size)
spectrum += noise

Define a model with explicitly named parameters

def total_gaussian(x, **params):
y = np.zeros_like(x)
for i in range(num_independent): # Independent Gaussians
center = params[f'center_{i}']
sigma = params[f'sigma_{i}']
area = params[f'area_{i}']
y += gaussian(x, center, sigma, area)
for i in range(num_dependent): # Dependent Gaussians
center = params[f'center_dep_{i}']
sigma = params[f'sigma_dep_{i}']
area = params[f'area_dep_{i}']
y += gaussian(x, center, sigma, area)
return y

Create an lmfit model

model = Model(total_gaussian, independent_vars=['x'])

Add parameters for independent Gaussians

params = model.make_params()
for i, (c, s, a) in enumerate(zip(centers, sigmas, areas)):
params.add(f'center_{i}', value=c)
params.add(f'sigma_{i}', value=s, min=0.1)
params.add(f'area_{i}', value=a, min=0)

Add parameters for dependent Gaussians (defined as expressions)

for i, (linked_idx, offset, scale_sigma, scale_area) in enumerate(dependent_links):
params.add(f'center_dep_{i}', expr=f'center_{linked_idx} + {offset}')
params.add(f'sigma_dep_{i}', expr=f'sigma_{linked_idx} * {scale_sigma}')
params.add(f'area_dep_{i}', expr=f'area_{linked_idx} * {scale_area}')

Fit the spectrum

print('Fitting model')
start_time = time.time()
result = model.fit(spectrum, x=x, params=params)
fitting_time = time.time() - start_time
print(f"Model.fit() took a {fitting_time:.1f} s")

Plot the results

plt.figure(figsize=(12, 6))
plt.plot(x, spectrum, label="Synthetic Spectrum", color='blue', linewidth=1)
plt.plot(x, result.best_fit, label="Best Fit", color='red', linestyle='--', linewidth=1)
plt.legend()
plt.xlabel("X")
plt.ylabel("Intensity")
plt.title("Spectrum Fitting with Independent and Dependent Gaussians")
plt.show()

Print fit report

print(result.fit_report())

[newville]

@Andrea-gm We are all volunteers here and professional scientists. Who we are is pretty easy to look up. You are welcome to accuse us of being aggressive or unkind or whatever you want. It simply is not going to work. You can be mad at us for whatever you think we did or didn't do to you, and are welcome to have any opinion you would like to have. We have written this software, made it freely available to anyone who wants to use it, and tried to support it and answer questions about it. For all of us, this project is a side project. We are not paid to develop or support this software. We all have day jobs and families and many other side projects. In a recent presentation to grant funding agents, I did mention this project once, in passing, as being an example of outreach and "synergistic activities" as a spin-off of a more on-topic project. In its entirety, that was one of eight bullet points on one of twenty slides that read "projects like lmfit that benefit the wider scientific community".

And, if you are unhappy with your perception of our "attitude", then you can walk away.
I will try one last time to explain the responses here.

To be clear, you got substantive replies from two different developers on this project within days of posting a not-very-specific message to a GitHub project, and one that you had never posted to before. If you think that the response is anything short of "astonishingly fast" and "highly respectful of your input" then I take it that you have not had much experience with GitHub or any internet forum in the past.

You said, "I did not understand that you wanted the actual code, sorry about that". You posted to Github about Python code. This project is a pretty deep "scientific programming" sort of code. We expect professional scientists, not beginners here. Indeed, you explicitly indicated that you a) understood what this code does, and b) had played around with altering the code.
An early response from me was

It might seem obvious, but posting an example that shows the timing would be helpful. If changing code is a goal, examples demonstrating a problem would be needed. If those didn't seem obvious, please try to make that sort of thing seem obvious.

So, yes, we expect that posting real timing code and real code would be something that most people posting here would understand. You might have read such advice in our documentation or any forum such as StackOverflow to provide a Minimal Working Example. If you are asking a question about code not working as you would like, then providing example code is a perfectly normal expectation.

You asked several times to turn off the calculation of uncertainties, and "I’m having difficulty understanding the strong opposition to allowing users the option to disable uncertainty calculations.". Our answer remains, "No, we are not going to do this", and the reason for the opposition is only "because we are professional scientists". If you do not like it, you are welcome to use a different library. If you hack at our code or go out of your way to circumvent the estimate of uncertainties, please do not use or cite our work. We want nothing to do with that.

Your entire point of discussion was the runtime for a fit. We are thoroughly underwhelmed by such discussions. If runtime is that big of a concern for you, why the heck are you using Python? We invite you to re-write your fit in Fortran or C. That will probably give you a nice demonstration of how we value performance.

Your example creates ~50 Gaussians. You have ~45 variables and ~150 constraint expressions. Your code runs and gives a good fit, with calculated uncertainties in all parameters in under 30 seconds. What the heck are you complaining about?

Now, that is not how I would model XRF spectra from an EDS detector, but you're welcome to do your own analysis. A typical fit for XRF data from me might look like this (the plot shows only a selection of elements included in the model), using 22 variables and having a run time of 3 to 5 seconds:

a typical fit report (removing correlations) would be:

[[Fit Statistics]]
    # fitting method   = leastsq
    # function evals   = 142
    # data points      = 1010
    # variables        = 22
    chi-square         = 265040.135
    reduced chi-square = 268.259246
    Akaike info crit   = 5669.63025
    Bayesian info crit = 5777.81977
[[Variables]]
    det_thickness:        0.45 (fixed)
    det_noise:            0.04426878 +/- 8.0075e-04 (1.81%) (init = 0.04)
    cal_offset:           0.01852466 +/- 8.4327e-04 (4.55%) (init = 0)
    cal_slope:            0.00996534 +/- 1.5243e-06 (0.02%) (init = 0.01)
    cal_quad:             0 (fixed)
    peak_step:            0.025 (fixed)
    peak_tail:            0.2 (fixed)
    peak_beta:            1.30981596 +/- 0.07885708 (6.02%) (init = 1)
    peak_gamma:           0.05 (fixed)
    elastic_amp:          317.199196 +/- 9.85447683 (3.11%) (init = 1000)
    elastic_center:       11 (fixed)
    elastic_step:         0.02 (fixed)
    elastic_tail:         0.1 (fixed)
    elastic_beta:         0.5 (fixed)
    elastic_sigmax:       1 (fixed)
    compton1_amp:         416.326242 +/- 7.43944903 (1.79%) (init = 1000)
    compton1_center:      10.7873125 +/- 0.00366787 (0.03%) (init = 10.77)
    compton1_step:        0.025 (fixed)
    compton1_tail:        1 (fixed)
    compton1_beta:        2 (fixed)
    compton1_sigmax:      1.25 (fixed)
    compton2_amp:         6.77446046 +/- 0.61963845 (9.15%) (init = 10)
    compton2_center:      11.5596774 +/- 0.01833362 (0.16%) (init = 10.55)
    compton2_step:        9.54953477 +/- 2.64941224 (27.74%) (init = 0.025)
    compton2_tail:        0.5 (fixed)
    compton2_beta:        2.5 (fixed)
    compton2_sigmax:      1.5 (fixed)
    filterlen_beryllium:  0.025 (fixed)
    filterlen_air:        50 (fixed)
    pileup_amp:           0.01 (fixed)
    escape_amp:           0.51523209 +/- 0.04052627 (7.87%) (init = 0.25)
    amp_s:                178.956484 +/- 7.68319656 (4.29%) (init = 300)
    amp_ar:               14.6599600 +/- 1.01439005 (6.92%) (init = 25)
    amp_k:                155.547407 +/- 3.15357353 (2.03%) (init = 300)
    amp_ca:               150.579447 +/- 2.39645087 (1.59%) (init = 300)
    amp_v:                0.14823544 +/- 0.04278643 (28.86%) (init = 10)
    amp_mn:               2.26420713 +/- 0.08991971 (3.97%) (init = 5)
    amp_fe:               7.75415156 +/- 0.16410495 (2.12%) (init = 15)
    amp_ni:               0.33903225 +/- 0.02293475 (6.76%) (init = 1)
    amp_cu:               1.10492389 +/- 0.04149199 (3.76%) (init = 2)
    amp_zn:               7.78857708 +/- 0.12674698 (1.63%) (init = 15)
    amp_ba:               2.68393460 +/- 0.18876970 (7.03%) (init = 4)

And, yes the uncertainties are important here. V is not very significant, and Ba L lines give a better fit in that region. Mn, Cu, Ni, and Ba are all significant even though their amplitudes (proportional to concentration, though not yet calibrated to "ppm" or "wt %" here) are smaller than the uncertainty in the concentration of K. That's a common feature in XRF analysis. Cu and Ba are both significant here, but that is not because of their amplitudes, it is because of their uncertainty.

Now, one might imagine that you might have started by asking, "Hey, this fit seems sort of slow, Does anyone have any ideas how this could be improved?". Instead, you asked about how to remove the calculation of uncertainties, apparently assuming that this is the only way to improve performance. Sorry, but we are not going to help you with that.

[Andrea-gm]

For the record, again, I am in no way complaining about your work or that of others, nor do I expect the modification I proposed to be implemented. I am simply asking. Stop accusing me of things I do not say, nor intend to say. I am highly appreciative of all the time spent by the volunteers in putting together this excellent package.

“It might seem obvious”, but when I said I did not appreciate the aggressive tone, I referred only to you. @reneeotten has been, on the other hand, very kind and helpful before you decided to take over and dismiss everything I had to say. Both of you have been “astonishingly fast”, but only one person has been “highly respectful of my input”. “It might seem obvious”, but I am not referring to you with the latter.

From the start, you questioned every single thing I said—which is understandable and expected in a scientific discussion, so no issue there. However, when I showed that I was correct, there was no acknowledgment of that fact. Do you realize that calculating uncertainties is tripling or even quadrupling the time needed for a single fit? Is it really so difficult to understand why I’d want to cut this time, especially since I do not need the uncertainties? I am working with an automated lab, with an output of 150 EDS spectra per hour. Switching to Fortran or C isn’t a practical solution, at least as an initial approach. As an experienced coder, I’m sure you understand that it’s far simpler to turn off an option than to rewrite thousands of lines of code in another language.

Yes, I am new to GitHub, but I do not see how not readily understanding that you wanted actual code, rather than just an example of the timings, makes me less of a “professional scientist”. Indeed, had you asked about a “Minimal Working Example” instead of an ‘Example”, I would have realised directly you were expecting some code. And, by the way, what difference did it make? You still decided that taking 4 times the time for fitting was not going to be an issue for me, without even knowing what problem I am trying to solve.

You keep assuming that you know the scientific problem I am trying to solve better than me, but quite frankly, you do not. You keep giving unsolicited scientific advice. Maybe it’s because I am new to GitHub, but I thought this Discussion section was to ask about the code, rather than the science.
Still, if you know how to bring down the number of gaussians I have to use for each element, I’ll be happy to hear it. That said, even if it cuts the calculation time of uncertainties, I’m still not going to use them, because I do not need them. If you had been “highly respectful of my input” you would know that my measurement accuracy is 10%, so a fitting uncertainty of 3% is not going to make any difference. Probably in the future I will find a way to make a use of the fitting uncertainty, but for the moment it’s not needed.

In addition, had you “been respectful of my input”, you would have seen that create_uvars is being called twice, once within _calculate_uncertainties_correlations inside Minimizer.minimize(), and once right after Minimizer.minimize() returns the result within Model_Result.fit(). Is this really intended? It seems to me that the same long calculations are being repeated for no reason, but maybe I am wrong on this. An actual reply considering my input would have been appreciated.

One last thing—does being new to GitHub mean I have no right to post here? It would be more helpful if you refrained from being presumptuous, dismissive and judgmental. Please get down from your pedestal.