loading and combining many .dat files

Share icon

Thu, 05/08/2014 - 06:55 am

Hello. I have a few years of data files that all have the same structure that I would like to load at once and combine all the similar waves (there is a separate .dat files for each month for three years). For example, the first three columns are always date, ozone and SO2 (I have 42 columns in total). I would like to load all the files that I have and combine the date, ozone, SO2 (and rest of the columns) for each file into one date, ozone, SO2 column etc. The data files do not have any names for the columns, which is fine, I figure I can rename at beginning or end.

I've figured out how to load multiple files using DoLoadMultipleFiles() and concatenate one by one using the Concatenate command, but I can't figure out how to do it automatically and all together. I've been trying to do this myself using this example: http://www.igorexchange.com/node/5337 and I am not quite getting there.

I understand that I must load the first one in the series myself, which is fine, and then append. But I can't seem to figure out how to concatenate my remaining files onto the original one. I get a little lost in the ChemGoof code and how to make it work with my data. Can someone help, please? I "think" my problem is not following how the creating a temp data folder works and thus can't figure out how to modify it, but perhaps I have something else wrong too.

Also, I am worried about being certain that the files load in the correct order. I know I can sort at the end, so there won't be an issue with the ozone, SO2, etc columns being associated with the wrong date column?

Thanks.
Starting after your first loaded file, I'm going to assume you have wavenames wave0, wave1, wave2,....wave41

Now, when you load a second file, it will autoname the new columns starting from 42. But you don't really have to worry about the names it gives because they will be in S_Wavenames.

1. Check out the LoadandGraphAll(pathname) example in the manual. Following this methodology (skipping the graphing) iterate through the files within a folder loading them individually.... I used a similar process to compile solar panel generation data from 4 years of files.

2. The new waves loaded will be named in S_wavenames

3. Iterate through the loaded waves, concatenating them to those that came before

4. replace the original wave with the concatenated one (following code is not tested and is just for a guide)

//section for concatenating new columns to those which were loaded before
variable i
for (i=0; i<itemsinlist(s_wavenames); i+=1)
     wave previouswave = $("wave" + num2str(i))
     wave newwave = $(stringfromlist(i, s_wavenames,";"))

    concatenate {previouswave, newwave}, concatenatedwave
    killwaves/z previouswave, newwave       //kill partial waves
    wave concatenatedwave
    rename concatenatedwave, $("wave" + num2str(i))   //rename concatenated wave as the original... This is to ensure the next file can find it
endfor


//after all files are loaded feel free to rename the waves


5. So long as the files within the folder are in the proper order, the LoadandGraphAll example of indexing the files will ensure they are concatenated in the proper order

May 8, 2014 at 07:50 am - Permalink

Whenever wave names are known in advance you should use the LoadWave /B flag to name the waves as is done in the LoadChemGoofFile example.

Quote:
I am worried about being certain that the files load in the correct order.

What is the rule for determining the correct order?

If you can't figure it out attach a zip archive of three files and post it along with the code you have to this point. Also specify your OS and Igor version.

May 8, 2014 at 09:05 am - Permalink

Thanks for the advice. I'll work on what you both suggest and get back if I am still stuck.
Thanks

May 9, 2014 at 06:57 am - Permalink