weights in curve_fit

[jcrbloch]

I should have opened a new issue for this, so here it comes:
The parameter w for weights in curve_fit is not very thoroughly documented. What I see is:
w: (optional) weight applied to the residual; can be a vector (of length(x) size or empty) or matrix (inverse covariance matrix)
From this I assumed that when providing w as a vector the routine expects the inverse of the variance, as this would be in line with the concept of covariance matrix if w is given as a matrix.

However from my application, and after comparing my results with fits done in C and with gnuplot, it looks as if curve_fit uses the weight vector as inverse standard deviations rather than inverse variances. I had a quick non-expert look in the curve_fit source, and that is also what I think I see in the source code. Could you confirm this and explain the logics?

Kind regards,
Jacques Bloch

[iewaij]

Hi @jcrbloch thanks for reporting the issue!

Could you elaborate on why "curve_fit uses the weight vector as inverse standard deviations"? In LsqFit.jl, the weight is assumed as 1/var(ε_i) and the covariance matrix is calculated as inv(J' * fit.wt * J), in the same way defined in GNU Scientific Library. According to Weighted and General Least Squares, I think essentially the logic behind is:

Under heteroskedastic error where Ω is a diagonal matrix, the GLS estimator has . If , then and β is B.L.U.E.

[jcrbloch]

Unfortunately that is not what the routine is doing. When passing a vector of errors instead of a matrix, the user is implicitly expected to give the inverse of the standard deviations and not the inverse of the variances. This is fundamentally different from what is expected when passing an error matrix, which is expected to be the inverse of the covariance matrix. I tried this by passing a either vector or a diagonal matrix, and the results are only the same when the vector contains the square root of the diagonal matrix entries. In curve_fit.jl lines 89 and 96: when computing f(p) the difference between model and data are multiplied with the vector of weights. When computing the chi squared that will be minimized, this will then be squared. So clearly the routines expect the inverse of the standard deviations and not of the variance. This can also be verified by looking at lines 108-110 for the case where an error matrix is passed. There the Cholesky decomposition is used, which amounts to effectively taking the square root of the covariance matrix. When an error vector is passed, no square root is used in the routines. That is were the mistake resides.

…

On 19. Jun 2018, at 03:39, Jiawei Li ***@***.***> wrote: Hi @jcrbloch <https://github.com/jcrbloch> thanks for reporting the issue! Could you elaborate on why "curve_fit uses the weight vector as inverse standard deviations"? In LsqFit.jl, the weight is assumed as 1/var(ε_i) and the covariance matrix is calculated as inv(J' * fit.wt * J), in the same way defined in GNU Scientific Library <https://www.gnu.org/software/gsl/manual/html_node/Nonlinear-Least_002dSquares-Weighted-Overview.html#Nonlinear-Least_002dSquares-Weighted-Overview>. Here's a great note on Weighted and General Least Squares <http://www.stat.cmu.edu/%7Ecshalizi/mreg/15/lectures/24/lecture-24--25.pdf>, I think essentially the logic behind is: Under heteroskedastic error <https://camo.githubusercontent.com/7c651b1ab90e7689e4033ef214174ba9a42439e4/687474703a2f2f6c617465782e636f6465636f67732e636f6d2f6769662e6c617465783f636f76253238253543657073696c6f6e2532392532302533442532302535437369676d61253545322532302535434f6d656761> where <https://camo.githubusercontent.com/541cba9748da394b56e923505972efa3d70b281c/687474703a2f2f6c617465782e636f6465636f67732e636f6d2f6769662e6c617465783f2535434f6d656761> is a diagonal matrix, the GLS estimator <https://camo.githubusercontent.com/804635a768cdfad07fd64c0ea6c92b65072d951f/687474703a2f2f6c617465782e636f6465636f67732e636f6d2f6769662e6c617465783f253543686174253742253543626574612537442532302533442532302532384a2532372532302535434f6d6567612535452537422d312537442532304a2532392535452537422d312537444a2532372532302535434f6d6567612535452537422d3125374425323059> has <https://camo.githubusercontent.com/73c444002efc82f0c1123f649478eb87eb098d9e/687474703a2f2f6c617465782e636f6465636f67732e636f6d2f6769662e6c617465783f636f76253238253543686174253742253543626574612537442532392532302533442532302535437369676d61253545322532302532384a2532372532302535434f6d6567612535452537422d312537442532304a2532392535452537422d31253744>. If <https://camo.githubusercontent.com/f37e952180f5da985cce701e096c0ffe68baeb04/687474703a2f2f6c617465782e636f6465636f67732e636f6d2f6769662e6c617465783f77253230253344253230636f76253238253543657073696c6f6e2532392535452537422d312537442532302533442532302535434f6d6567612535452537422d312537442532302f2532302535437369676d6125354532>, then <https://camo.githubusercontent.com/30c59ed7b7cb2f4d0a416e19259fbecc1e80c273/687474703a2f2f6c617465782e636f6465636f67732e636f6d2f6769662e6c617465783f636f76253238253543686174253742253543626574612537442532392532302533442532302535437369676d61253545322532302532384a2532372532302535434f6d6567612535452537422d312537442532304a2532392535452537422d312537442532302533442532302532384a253237253230772532304a2532392535452537422d31253744>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#69 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFhuZLKkAgZSW2U7MFyUZAXgUbChQxxiks5t-FZggaJpZM4TvMt2>.

[jcrbloch]

To comment further, after looking at issue 71: I think the implementation of the fit with error matrix was perfectly fine, and there is no need to change it. The problem is only when passing an error vector, as i explained in my detailed post above.

[iewaij]

Yes it is a mistake. Thanks very much for pointing out! #72 should fix this.

[iewaij]

I think a possible explanation for passing the reciprocal of standard deviation (σ) to residual function is to reduce the computation needed for sqrt() if the weight is estimated using fit.resid.^2 (but current implementation is still wrong either way). I don't know much about computational complexity but here is my reasoning.

Scenario 1: pass weight vector as the reciprocal of estimated variances of error, which is PR #72:

var_error =fit.resid.^2
wt = 1./var_error
curve_fit(model, tdata, ydata, wt, p0)

# behind the code
sqrt_wt = sqrt.(wt) # which costs a lot of time
f(p) = sqrt_wt .* ( model(xpts, p) - ydata ) # the residual function for least squares algorithm
fit = lmfit(f, p0, wt; kwargs...)
covar = inv(J'*Diagonal(fit.wt)*J)

Scenario 2: Pass weight vector as the reciprocal of estimated standard deviation of error, which is essentially PR #74:

std_error  = abs.(fit.resid) # which doesn't cost much time
sqrt_wt = 1./std_error
curve_fit(model, tdata, ydata, sqrt_wt, p0)

# behind the code
f(p) = sqrt_wt .* ( model(xpts, p) - ydata ) # the residual function for least squares algorithm
wt = sqrt_wt.^2
fit = lmfit(f, p0, wt; kwargs...)
covar = inv(J'*Diagonal(fit.wt)*J)

Just for reference, I ran through 2 different scenarios, Scenario 1 costs nearly 2x time of Scenario 2. The notebook could be viewed and reproduced here, make sure you've restarted kernel.

[pkofod]

@jcrbloch this took a while, but sqrt is now applied to the weights for consistency with the matrix form. It's not in the latest release, but it's in the master versions. Documentation is not up to date though.