Load several data files
tooprock
Tue, 03/31/2015 - 06:57 am
I have some data with irregular format. What I'm trying to do is writing some functions to read only selected files into separate data folders. I put an example below and this format repeats itself in every file. Attached you can see an example data file. I have so far:
- First row is the Date/Time
- Second, third and fourth rows are data points for three main groups (each one has 10 parameters). My final wave will have the following dimensions (Rows = X; Columns = 10; Layers = 3)
- Fifth row has different number of columns therefore I thought they can be saved in a 2D wave (Rows = X; Columns = 8)
- Sixth row has also different number of columns than other five rows. So again a 2D wave (Rows = X; Columns = 9)
Example Data:
-----------------------
T,2011,06,09,10,26,11
B,319927,13990,111,689,162065,6511,113,690,1003.6,303.2
G,572177,12878,240,689,295490,4997,241,690,1003.6,303.2
R,699138,21745,1437,689,366300,10369,1350,690,1003.6,303.2
D,BBXX,234,2.890e-5,1.771e-5,1.705e-5,1.291e-5,6.409e-6,6.479e-6
Y,168092,1003.6,303.2,298.2,62.6,9.3,7.9,0,0000
T,
B,
G,
R,
D,
Y,
...
I thought I may write two functions. One to read data files in order and save the data in a separate data folder. A second function to do the main job and read/load individual data points.
Function LoadSelectedNephFiles()
Variable refNum
String message = "Select one or more .dat files"
String outputPaths
String fileFilters = "Data Files (*.dat):.dat;"
SVAR workdir = root:workdir
fileFilters += "All Files:.*;"
Open /D /R /MULT=1 /F=fileFilters /M=message refNum
outputPaths = S_fileName
if (strlen(outputPaths) == 0)
Print "Cancelled"
return -1
endif
Variable numFilesSelected = ItemsInList(outputPaths, "\r")
NVAR numberofAnalFiles = root:numberofAnalFiles
numberofAnalFiles = numFilesSelected
Variable i
for(i=0; i<numFilesSelected; i+=1)
String path = StringFromList(i, outputPaths, "\r")
// Printf "%d: %s\r", i, path // For debugging only
Variable err = LoadNephFile("", path) // It is OK to pass "" for pathName because we are passing a full path for fileName
if (err != 0)
return err
endif
endfor
return 0
// Success
End
Variable refNum
String message = "Select one or more .dat files"
String outputPaths
String fileFilters = "Data Files (*.dat):.dat;"
SVAR workdir = root:workdir
fileFilters += "All Files:.*;"
Open /D /R /MULT=1 /F=fileFilters /M=message refNum
outputPaths = S_fileName
if (strlen(outputPaths) == 0)
Print "Cancelled"
return -1
endif
Variable numFilesSelected = ItemsInList(outputPaths, "\r")
NVAR numberofAnalFiles = root:numberofAnalFiles
numberofAnalFiles = numFilesSelected
Variable i
for(i=0; i<numFilesSelected; i+=1)
String path = StringFromList(i, outputPaths, "\r")
// Printf "%d: %s\r", i, path // For debugging only
Variable err = LoadNephFile("", path) // It is OK to pass "" for pathName because we are passing a full path for fileName
if (err != 0)
return err
endif
endfor
return 0
// Success
End
Second function is still not clear to me and at this point I would appreciate any helps/suggestions. Some initial thoughts:
- Can I use LoadWave function? If yes, how should I design it work with this non-ideal data structure?
- How should I deal with the first block and put that in a loop?
Cheers,
Emre
I think you mean "How can I loop through all of the files in a folder". Use IndexedFile. For and example, see http://www.igorexchange.com/node/5843
LoadWave can only load a regular block of numbers. You can tell LoadWave to load a block of lines of text using the /L flag. So you can tell LoadWave to load lines 1, 2 and 3 (zero-based). You can then parcel the data into your 3D output wave and repeat for the next block of RGB lines. To do this you need to be able to calculate the line numbers in each RGB block. If the files are regular, that is doable. However you also need to know when to stop. For this you need to know the total number of lines in the file, which can note be determined except by examining the file line-by-line.
Because of this last issue, I think I would dispense with LoadWave and use FReadLine instead. There is an example at http://www.igorexchange.com/node/880. You would then use sscanf to parse a given line and store the values in the appropriate place in the output wave.
March 31, 2015 at 10:16 am - Permalink
String pathName // Igor symbolic path name or "" for dialog.
String fileName // file name, partial path and file name, or full path or "" for dialog.
Variable err
// This puts up a dialog if the pathName and fileName do not specify a file.
String message = "Select a data file"
Variable refNum
String fileFilters = "Data Files (*.dat):.dat;"
Open /R /Z=2 /P=$pathName /F=fileFilters /M=message refNum as fileName
// Save outputs from Open in a safe place.
err = V_Flag
String fullPath = S_fileName
SVAR workdir = root:workdir
workdir = S_fileName
if (err != 0)
return err // -1 means user canceled.
endif
String fileNameFromFile // File name as read from the first line of the file
// Create data folder
String dfName = ParseFilePath(3, fullPath, ":", 0, 0) // Get file name without extension
dfName = CleanupName(dfName, 0)
NewDataFolder /O /S $dfName
String targetText
String format
Variable year, month, day, hour, minute, second
Variable foundDateTime
Make/D/O/N=0 TimeWave
Variable thRow = 0
do
FReadLine refNum, targetText
if(stringmatch(targetText,"T,*"))
format = "T,%d,%d,%d,%d,%d,%d"
sscanf targetText, format, year, month, day, hour, minute, second
foundDateTime = Date2Secs(year,month,day) // Convert year. month and day into seconds
foundDateTime += second + 60*minute + 60*60*hour
InsertPoints thRow, 1, TimeWave
TimeWave[thRow] = foundDateTime
thRow += 1
endif
while(strlen(targetText)!=0)
SetScale d, 0, 0, "dat", TimeWave
Close refNum
SetDataFolder ::
return 0
End
Cheers,
Emre
March 31, 2015 at 10:28 am - Permalink