error: The fitting function returned NaN for at least one X value.

paperlovestory
Posts: 36
Joined: 2016-12-28
Location: United Kingdom

What might the above error message mean? I'm using the FuncFit operation.

This is my fit function:

function Rtot_line(w, x) : FitFunc 
	wave w
	variable x
 
	variable val
	val = w[0] * (x - w[1]) + w[2]			//w[1] < 0
	return val
end

The x-wave is just a series of integers, 5,10,15,20... and when I print the coefficients w[0] w[1] and w[2] in the main body of the procedure that runs the FuncFit operation, I get an integer (not NaN). Wondering what exactly is the NaN which is being returned here?


Posts: 1966
Joined: 2007-06-29
Location: United States

It's possible that error is being returned erroneously instead of Singular Matrix error. Your fit function features a linear dependency. If you multiply out, you get

val = w[0] * x - w[0]*w[1] + w[2]

That means that you have a constant term composed of a combination of all three fit coefficents. Any change in one can be compensated for by a change in another, so there is not locally unique solution. You need to just fit a line.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


paperlovestory
Posts: 36
Joined: 2016-12-28
Location: United Kingdom

johnweeks wrote:
It's possible that error is being returned erroneously instead of Singular Matrix error. Your fit function features a linear dependency. If you multiply out, you get
val = w[0] * x - w[0]*w[1] + w[2]

That means that you have a constant term composed of a combination of all three fit coefficents. Any change in one can be compensated for by a change in another, so there is not locally unique solution. You need to just fit a line.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

Thanks John! The reason why the fit function was composed as such is because w[1] and w[2] are terms which I'd like to hold constant using the /H="011" flag; is there a way to get around this since I cannot use the standard "y=K0+K1x" fit function?


Posts: 1966
Joined: 2007-06-29
Location: United States

Ah. The technique of including constants in the fit function coefficients and then holding them is the easiest way to achieve your goal. If you hold w[1] and w[2] does the error go away?

The much more difficult way to do that is to define a structure fit function. The constants would be part of the "extra" members of the structure. This requires that you write a driver for the fit that sets up the structure.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


paperlovestory
Posts: 36
Joined: 2016-12-28
Location: United Kingdom

johnweeks wrote:
Ah. The technique of including constants in the fit function coefficients and then holding them is the easiest way to achieve your goal. If you hold w[1] and w[2] does the error go away?

The much more difficult way to do that is to define a structure fit function. The constants would be part of the "extra" members of the structure. This requires that you write a driver for the fit that sets up the structure.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

Indeed, if I hold w[1] and w[2], the error is still present. Using the same set of numbers and attempting to fit them somewhere else with the expression w[0]*(x-w[1]) + w[2], where w[1] and w[2] were fixed and pre-determined by an earlier procedure in IGOR did not give rise to the singular matrix error, i.e. my data should fit when there is only one unknown fit coefficient. However when the FuncFit is run here, the error message as above still appears.

Could it mean something else other than the singular matrix error?


jjweimer
jjweimer's picture
Posts: 1334
Joined: 2007-08-14
Location: United States

paperlovestory wrote:

Thanks John! The reason why the fit function was composed as such is because w[1] and w[2] are terms which I'd like to hold constant using the /H="011" flag; is there a way to get around this since I cannot use the standard "y=K0+K1x" fit function?

Since you hold the terms constant, you know their values in advance. Fit the straight line y = m*x + b. Extract the unknown w[0] by algebra using m, b and your known w[1] and w[2]. Use linear uncertainty propagation to obtain the uncertainty on w[0] from the uncertainties on m and b (since w[1] and w[2] are constant in this case, their uncertainties are zero).

Alternatively, define your fit function as w[0] * (x - alpha) + beta where alpha and beta are predefined (rather than held during the fitting procedure).

I am interested as well to know what might be behind the error when you hold the terms constant in advance. I would wonder what happens when you predefine them as above to avoid using a hold flag.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH


Posts: 1966
Joined: 2007-06-29
Location: United States

Please post an Igor experiment file with a data set, fit function, initialization and FuncFit command that reproduces your problem. I will see if I can figure out what your problem is.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


paperlovestory
Posts: 36
Joined: 2016-12-28
Location: United Kingdom

johnweeks wrote:
Please post an Igor experiment file with a data set, fit function, initialization and FuncFit command that reproduces your problem. I will see if I can figure out what your problem is.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

I have attached a (simplified) experiment file here with a real data set and for which the error occurs.

AttachmentSize
20180321_lineFit_test.pxp10.88 KB

paperlovestory
Posts: 36
Joined: 2016-12-28
Location: United Kingdom

jjweimer wrote:

Alternatively, define your fit function as w[0] * (x - alpha) + beta where alpha and beta are predefined (rather than held during the fitting procedure).

I am interested as well to know what might be behind the error when you hold the terms constant in advance. I would wonder what happens when you predefine them as above to avoid using a hold flag.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH

I have thought of this alternative to solve the problem described above as well, and this is what I've written:

for (aa = 0; aa < 13; aa += 1)
	D1D[] = D2D[p][aa]
	wavestats /M=1 /Q D1D
	make/o/n=1 W_coef
	W_coef = {1}			//"1" arbitrarily chosen, less likely to give error than "0"
	if (V_npnts > 2)
		FuncFit lineFit, kwcWave=W_coef, D1D /X=waveX
	else
		printf "%s\r", "Insufficient data points"
		continue
	endif
endfor
 
//--------------------------------------------------------------------------------
 
function lineFit(w, x) : FitFunc 
 
	wave w
	variable x
 
	variable val, alpha, delta
	nvar X_ave = X_ave
	nvar Y_ave = Y_ave
	alpha = X_ave
	delta = Y_ave
 
	val = w[0] * (x - alpha) + delta			//w[1] < 0
 
	return val
 
end

Seems to get the job done, does not give an error (although I must admit my code seems quite poorly written?) but the set of coefficients generated using this, from the same data set, differs from the fitting done using another graphing software (manually) for the same X_ave and Y_ave.


jjweimer
jjweimer's picture
Posts: 1334
Joined: 2007-08-14
Location: United States

paperlovestory wrote:

I have thought of this alternative to solve the problem described above as well, and this is what I've written:

for (aa = 0; aa < 13; aa += 1)
	D1D[] = D2D[p][aa]
	wavestats /M=1 /Q D1D
	make/o/n=1 W_coef
	W_coef = {1}			//"1" arbitrarily chosen, less likely to give error than "0"
	if (V_npnts > 2)
		FuncFit lineFit, kwcWave=W_coef, D1D /X=waveX
	else
		printf "%s\r", "Insufficient data points"
		continue
	endif
endfor
 
//--------------------------------------------------------------------------------
 
function lineFit(w, x) : FitFunc 
 
	wave w
	variable x
 
	variable val, alpha, delta
	nvar X_ave = X_ave
	nvar Y_ave = Y_ave
	alpha = X_ave
	delta = Y_ave
 
	val = w[0] * (x - alpha) + delta			//w[1] < 0
 
	return val
 
end

Seems to get the job done, does not give an error (although I must admit my code seems quite poorly written?) but the set of coefficients generated using this, from the same data set, differs from the fitting done using another graphing software (manually) for the same X_ave and Y_ave.

A more compact expression that avoids extra terms (local variables) and nomenclature issues (w and x terms in the call) is below.

Function lineFit(ww,xx)
     wave ww
     variable xx
 
     NVAR x_ave, y_ave
 
     return (w[0]*(xx- x_ave) + y_ave)
end

I really dislike the use of globals in the fit function. The only two ways around this are to pass parameters that are held constant or to use a structure fit function. I believe also that you can eliminate x_ave and y_ave as global inside the fit function in this way.

make/O/N=1 w_coeff
for (...)
    DID = D2D[p][aa]
    WaveStats/M=1/Q DID
    w_coef = 1
    DID += - y_ave
    waveX +=  x_ave
    FuncFit lineFitNew, kwcWave=w_coef, DID /X=waveX
end for
 
Function lineFitNew(ww,xx)
     wave ww
     variable xx
 
     return (w[0]*xx)
end

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH


[ last edited March 21, 2018 - 08:42 ]
Posts: 1966
Joined: 2007-06-29
Location: United States

paperlovestory wrote:
I have attached a (simplified) experiment file here with a real data set and for which the error occurs.

Thanks. You found a bug!
The results of one iteration of the fit were checking for NaN or Inf in elements of an internal matrix that corresponded to your fixed coefficients. It should only have checked elements that were actively being fit. The contents of those cells would be whatever was in memory when the matrix was allocated, so it usually didn't manifest. I don't know why that one fit had a problem; perhaps there were actual NaNs lying about in memory because the data set had several NaNs.

In any case, the problem is fixed for the next nightly build of Igor. Tomorrow, select Help->Igor Pro Nightly Builds to get the fix. Be sure the build number is at least 31485.

The bug has been there for many years. You had just the right combination to trigger it.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


paperlovestory
Posts: 36
Joined: 2016-12-28
Location: United Kingdom

johnweeks wrote:
paperlovestory wrote:
I have attached a (simplified) experiment file here with a real data set and for which the error occurs.

Thanks. You found a bug!
The results of one iteration of the fit were checking for NaN or Inf in elements of an internal matrix that corresponded to your fixed coefficients. It should only have checked elements that were actively being fit. The contents of those cells would be whatever was in memory when the matrix was allocated, so it usually didn't manifest. I don't know why that one fit had a problem; perhaps there were actual NaNs lying about in memory because the data set had several NaNs.

In any case, the problem is fixed for the next nightly build of Igor. Tomorrow, select Help->Igor Pro Nightly Builds to get the fix. Be sure the build number is at least 31485.

The bug has been there for many years. You had just the right combination to trigger it.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

Thanks John.

After getting the fix, my build number was 31475. I'm not sure if it solved the problem because it caused many of my earlier codes to no longer work (or, are experiment files supposed to be built from scratch after replacing the files?). I have since reverted back to my original one (31118) and my earlier codes work again.


Posts: 1966
Joined: 2007-06-29
Location: United States

paperlovestory wrote:
After getting the fix, my build number was 31475.

No, you didn't get a sufficiently recent build. Try again today, please.
Quote:
I'm not sure if it solved the problem because it caused many of my earlier codes to no longer work (or, are experiment files supposed to be built from scratch after replacing the files?). I have since reverted back to my original one (31118) and my earlier codes work again.

What went wrong? I don't think we have changed that much since 31118.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


paperlovestory
Posts: 36
Joined: 2016-12-28
Location: United Kingdom

johnweeks wrote:
paperlovestory wrote:
After getting the fix, my build number was 31475.

No, you didn't get a sufficiently recent build. Try again today, please.
Quote:
I'm not sure if it solved the problem because it caused many of my earlier codes to no longer work (or, are experiment files supposed to be built from scratch after replacing the files?). I have since reverted back to my original one (31118) and my earlier codes work again.

What went wrong? I don't think we have changed that much since 31118.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

Thank you. It is 31485 right now and the earlier codes work again.


Posts: 1966
Joined: 2007-06-29
Location: United States

That's a relief! Thanks. I think the nightly build failed after my fix, and that would explain why you got a too-early build.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


Back to top