Opening 2500 files at once
mvlee
Wed, 03/30/2011 - 12:27 am
I'm currently importing data from text files and turning it into images. I think that's a fairly common problem.
I wrote a procedure to open each of the files and import the data into multidimensional waves.
I'm using the Open Dialog for multiple files.
String message = "Select one or more files"
String outputPaths
String fileFilters = "Data Files (*.txt):.txt;"
fileFilters += "All Files:.*;"
Open /D /R /MULT=1 /F=fileFilters /M=message refNum
outputPaths = S_fileName
When I only have 900 files, it works fine, but when I try to open 2500 files, the open command seems to just abort and S_filename is empty.
I think maybe I'm hitting the limit for the number of files the dialog can handle, but I haven't seen a documented limit.
Is there a limit to the number of files? Is there something obvious that I'm missing?
Thank you.
Try these steps:
1) used IndexedFile to select all the files you're interested in (not interactive)
2) Iterate through that list to open each of the files in turn, perform your analysis and close the file.
March 30, 2011 at 04:45 am - Permalink
The number of _simultaneously_ open files is usually limited [1]. In your case it might be easier to open the files one by one. So something along the lines of
variable i, refNum
string filePath = "c:"
string fileNames = "file1.txt;file2.txt"
for(i=0; i < ItemsInList(fileNames);i+=1)
Open/R refNum as filePath + StringFromList(i,fileNames)
// handle file handles refNum
endfor
End
should to the trick. You can also use NewPath to let the user select a directory and then read all files from this folder.
March 30, 2011 at 05:48 am - Permalink
The post: http://www.igorexchange.com/node/1890 shows some code for this.
March 30, 2011 at 09:06 am - Permalink
I looked at the source code for the Open File dialog for multiple files on Windows. The Windows OS call we use requires that the buffer that receives the file names must be allocated before the call that puts up the Open File dialog. We allocate 50000 bytes for this buffer. That would leave room for 2500 20-character file names, except that it also has to store the path to the folder once. So if the path to the folder has, say, 100 characters, it would leave room for fewer than 2500 20-character file names. Naturally, it would be a different number depending on the number of characters in the individual file names. Oh, and there has to be room for a NULL byte between each name...
Note that this is not a limit on the number of simultaneously open files. This is a limit on the number of file names that can be returned from an Open File dialog with the /MULT flag. The allowed number of simultaneously open files is likely smaller.
As Thomas says, that's a lot of files. You might be better off with our older method of dealing with this sort of thing- if you can arranged for a single folder to contain all the files, and nothing but the files, then use IndexedDir to get file names. Use the NewPath command to get the path to a user-selected folder instead of requiring the selection of that many files in a dialog.
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
March 30, 2011 at 09:26 am - Permalink
variable i, refNum
string filePath = "c:"
string fileNames = "file1.txt;file2.txt"
for(i=0; i < ItemsInList(fileNames);i+=1)
Open/R refNum as filePath + StringFromList(i,fileNames)
// handle file handles refNum
Close refNum
endfor
End
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
March 30, 2011 at 09:28 am - Permalink
Yeep, missed that one. Not closing the file handle will not help in at all ...
bye,
Thomas
March 31, 2011 at 01:36 am - Permalink