Automate curve fitting

elliottmclean
Posts: 10
Joined: 2017-06-22
Location: Australia

I've got a procedure with a bespoke curve fitting function which I need to somehow make automatic for a number of files which need be loaded. At the moment the fitting function and the load/graph are in a seperate procedure files so my first question is as to whether that can be called from another procedure or whether I should combine it all into the one for simplicity?

For the first fit the fitting cursors need to be set to disregard noise at the end of the graph; and the initial guesses must be set to check as to whether the fit actually looks okay. Is it possible to call from a procedure to open the curve fitting dialog? Once the coefficients have been found for the first, they are fed into the function to fit the second, and then the coefficients from the second are used for the third and so on.

Any help on the matter would be greatly appreciated as I'm not sure where to even start.
Cheers.


[ last edited October 4, 2017 - 01:40 ]
Posts: 1823
Joined: 2007-06-29
Location: United States

If you already have something that will load and graph the files, you might want to look at the Batch Curve Fitting package. See Analysis->Packages->Batch Curve Fitting. There is a demo experiment to introduce you to it as well: File->Example Experiments->Curve Fitting->Batch Curve Fit Demo. I believe it has a mode to use each fit as the initial guess for the next.

It is possible to trigger the Curve Fit menu item using DoIgorMenu, but I'm not sure it's particularly useful. You could set up your workflow to require initial guesses as an input, and then the method you use to get those initial guesses would be up to you to decide each time you do this. You would also determine the position of the start and end points at the same time. Quite likely that you would use the dialog to do the first fit and use that as the initial guess.

Yes- you can string together different procedures by simply calling one function from another. You may wish to design the pieces with the thought that they would eventually work together. If a task has multiple steps, each of which is relatively independent of the others, it is always easier to develop each step separately and then finally work out the problems associated with stringing them together. Hint- error handling is something you need to think about up front. An error in an early step can lead to chaos down the chain.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


elliottmclean
Posts: 10
Joined: 2017-06-22
Location: Australia

johnweeks wrote:
If you already have something that will load and graph the files, you might want to look at the Batch Curve Fitting package. See Analysis->Packages->Batch Curve Fitting. There is a demo experiment to introduce you to it as well: File->Example Experiments->Curve Fitting->Batch Curve Fit Demo. I believe it has a mode to use each fit as the initial guess for the next.

Great! Thanks John I'll check out batch curve fitting and see how I go.


wings
Posts: 45
Joined: 2016-04-27
Location: China

elliottmclean wrote:
I've got a procedure with a bespoke curve fitting function which I need to somehow make automatic for a number of files which need be loaded. At the moment the fitting function and the load/graph are in a seperate procedure files so my first question is as to whether that can be called from another procedure or whether I should combine it all into the one for simplicity?

For the first fit the fitting cursors need to be set to disregard noise at the end of the graph; and the initial guesses must be set to check as to whether the fit actually looks okay. Is it possible to call from a procedure to open the curve fitting dialog? Once the coefficients have been found for the first, they are fed into the function to fit the second, and then the coefficients from the second are used for the third and so on.

Any help on the matter would be greatly appreciated as I'm not sure where to even start.
Cheers.

Very typical use of Igor!

1 make a wave to store the initial guesses, this wave will act as the coefficient wave.
2 make several waves to store fitted results. you can get the result from the coefficient wave after each fitting process.
3 get the name list( or wave reference list) of the waves to be fitted
4 use for loop (or do-while loop). in each loop, get the wave from the name list and fit it using funcfit. in each loop the same coefficient wave is used, so you dont need to fill in initial guess, just as you want, the coefficients (fitting results of the last fitting) are automatically used in the next fitting.


[ last edited October 6, 2017 - 05:10 ]
Posts: 1823
Joined: 2007-06-29
Location: United States

wings wrote:
in each loop, get the wave from the name list and fit it using funcfit. in each loop the same coefficient wave is used, so you dont need to fill in initial guess, just as you want, the coefficients (fitting results of the last fitting) are automatically used in the next fitting.

You need to be careful, though. If a fit fails, the contents of the coefficient wave may be garbage.
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


elliottmclean
Posts: 10
Joined: 2017-06-22
Location: Australia

At the moment I've got a function that opens the select files to be analysed and retrieves the paths which is then passed as a global variable to other functions to either be graphed normally or as a waterfall plot. Is there an easier way to pass the waves between functions rather than passing a global variable of the paths and loading the files into each function? Would it be easier to load the waves into a data folder in a function and then access that data folder in each other function?


[ last edited October 6, 2017 - 22:06 ]
wings
Posts: 45
Joined: 2016-04-27
Location: China

elliottmclean wrote:
At the moment I've got a function that opens the select files to be analysed and retrieves the paths which is then passed as a global variable to other functions to either be graphed normally or as a waterfall plot. Is there an easier way to pass the waves between functions rather than passing a global variable of the paths and loading the files into each function? Would it be easier to load the waves into a data folder in a function and then access that data folder in each other function?

//save the current work datafolder
string curr=getdatafolder(1)

//set the new data folder as the work datafolder
setdatafolder theFolderYouWantToLoadDataInto

//do anythings to the data just loaded

//restore previous datafolder
setdatafolder curr


[ last edited October 6, 2017 - 22:32 ]
wings
Posts: 45
Joined: 2016-04-27
Location: China

johnweeks wrote:
wings wrote:
in each loop, get the wave from the name list and fit it using funcfit. in each loop the same coefficient wave is used, so you dont need to fill in initial guess, just as you want, the coefficients (fitting results of the last fitting) are automatically used in the next fitting.

You need to be careful, though. If a fit fails, the contents of the coefficient wave may be garbage.
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com

yes, examine the fitted result is a necessary step of fitting process


Posts: 1823
Joined: 2007-06-29
Location: United States

You really should avoid global variables like the plague unless you need them to store information that needs to persist from one invocation of your code to the next. The best way to work is to pass the waves via function WAVE parameters. Something like this:

Function loadAndAnalyse()
	LoadWave ...
	Wave w = $StringFromList(0, S_WaveNames)	// assumes you only ever load one wave, but that limitation is easily overcome.
	AnalyseMyData(w)
end
 
Function AnalyseMyData(WAVE w)				// Uses Igor 7 inline declaration style!
	... Do something with the data in the wave ...
end

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


elliottmclean
Posts: 10
Joined: 2017-06-22
Location: Australia

johnweeks wrote:
You really should avoid global variables like the plague unless you need them to store information that needs to persist from one invocation of your code to the next. The best way to work is to pass the waves via function WAVE parameters. Something like this:
Function loadAndAnalyse()
	LoadWave ...
	Wave w = $StringFromList(0, S_WaveNames)	// assumes you only ever load one wave, but that limitation is easily overcome.
	AnalyseMyData(w)
end
 
Function AnalyseMyData(WAVE w)				// Uses Igor 7 inline declaration style!
	... Do something with the data in the wave ...
end

Thanks John I figured that would be the case. I'll just do what you mentioned.

In regards to the batch fitting I have 2 questions. At the moment I load up the selected files and extract the data into waves and graph the first set of data so I can do an initial fit to test initial guesses. I'm trying to feed W_coef from this inital fit into the batch fitting as the inital guesses for the fit via the Functions and Coefficients sub heading and in the "2D initial guess wave" drop menu. However when I try to run the batch fitting it does not work. It's not a huge inconvenience but it would save me individually typing the guesses into all the boxes.

Secondly, on a number of fit that I have run I keep getting an error which reads: "Error on batch X: Limit of passes without decreasing chi-square was reached" and the fit isn't applied to these sets of data and then each data set that follows has an incorrect fit. Now I'm not sure whether this was an error unique to my fitting function or whether it was common for batch fitting in general but I couldn't find a chi-square in my function.

Cheers.


Posts: 1823
Joined: 2007-06-29
Location: United States

elliottmclean wrote:
In regards to the batch fitting I have 2 questions. At the moment I load up the selected files and extract the data into waves and graph the first set of data so I can do an initial fit to test initial guesses. I'm trying to feed W_coef from this inital fit into the batch fitting as the inital guesses for the fit via the Functions and Coefficients sub heading and in the "2D initial guess wave" drop menu. However when I try to run the batch fitting it does not work. It's not a huge inconvenience but it would save me individually typing the guesses into all the boxes.

I'm not sure exactly what you mean- there is nothing in the dialog called "2D initial guess wave". You should be able to make a coefficient wave before you open the dialog, then select that wave from the Coefficient Wave menu. When you do that, the Initial Guess column in the list will be filled in with the contents of your wave. The pre-made coefficient wave must have exactly the right number of elements. Be sure you make a double-precision wave.

Quote:
Secondly, on a number of fit that I have run I keep getting an error which reads: "Error on batch X: Limit of passes without decreasing chi-square was reached" and the fit isn't applied to these sets of data and then each data set that follows has an incorrect fit. Now I'm not sure whether this was an error unique to my fitting function or whether it was common for batch fitting in general but I couldn't find a chi-square in my function.

That means that nine iterations went by without any improvement in the fit. One way that can happen is if you are fitting fake data that has no noise- you get a "perfect" fit with no room for improvement. It could also happen if you manage to give the fit the absolute best solution to start from. The V_chisq and the current solution are probably OK. Another way this can happen is if your initial guesses are actually really bad, or the fit function simply doesn't resemble your data at all, and the chi-square can't improve.

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


elliottmclean
Posts: 10
Joined: 2017-06-22
Location: Australia

johnweeks wrote:

I'm not sure exactly what you mean- there is nothing in the dialog called "2D initial guess wave". You should be able to make a coefficient wave before you open the dialog, then select that wave from the Coefficient Wave menu. When you do that, the Initial Guess column in the list will be filled in with the contents of your wave. The pre-made coefficient wave must have exactly the right number of elements. Be sure you make a double-precision wave.

I've attached a photo to show you what I mean. The "2D Initial Guess Wave" is on the left hand side which I assumed would be where I could input my coefficient wave from an initial seperate fit. I have a feeling the one you're directing me to is in the curve fitting however I was talking about batch fitting.

Cheers.

AttachmentSize
Screen Shot 2017-10-12 at 11.47.01 am.png163.13 KB

Posts: 1823
Joined: 2007-06-29
Location: United States

My apologies- I lost track of the fact that you are using the Batch Curve Fit package.

The 2D Initial Guess Wave menu is only available if you select 2D Wave, One Column Per Fit from the Initial Guess Mode menu just above 2D Initial Guess Wave.

Quote:
Secondly, on a number of fit that I have run I keep getting an error which reads: "Error on batch X: Limit of passes without decreasing chi-square was reached" and the fit isn't applied to these sets of data and then each data set that follows has an incorrect fit. Now I'm not sure whether this was an error unique to my fitting function or whether it was common for batch fitting in general but I couldn't find a chi-square in my function.

Now that I'm on board, let's try this one again.
The author of the Batch Curve Fit package has left WaveMetrics, making support tricky... What happens if you use the Same Initial Guesses for Each Batch item in the Initial Guess Mode menu? That sounds like it shouldn't get screwed up by a failure of one of the fits.

And the mode, "Use Coefficients from Last Successful Fit" seems like it should ignore errors, using the last solution that finished without a problem. It might be a bug.

Can you post an Igor experiment with a selection of data that illustrates your problem?

John Weeks
WaveMetrics, Inc.
support@wavemetrics.com


Back to top