Image Plot from matrix data
MarkusG
Sat, 09/24/2011 - 02:25 am
I would like to start by explaining how far I got on my own before I ran into the problems I am addressing here. Maybe there is some alternative along the way that will make the actual problem not appear in the first place.
I am starting with data that is in what I call "gnuplot format":
x1 y1 z11
x1 y2 z12
x1 y3 z13
x2 y1 z21
x2 y2 z22
...
I didn't find out whether this can be processed directly by Igor so I extracted three files from it with python which are (x1,x2,...), (y1,y2,...), and the matrix with the z-values. These I load into Igor which allows me to plot the matrix as an image with the x- and y-wave (after adding one additional point) as the corresponding axes.
The first problem is that I think the z-values in the matrix lose the knowledge about the x- and y-values they belong to. So it is up to me to set the right order for the axes. In this case I know how the image is supposed to look like but once I don't, how can I be sure which order of x- and y-values is correct?
The second and more important thing to me right now is, that I have two such matrices, one running from y = 0 to 3 Tesla the other one from y = 0 to -3 Tesla. I would like to plot them in the same image plot. Is there a way to correctly join the two matrices in Igor? For that I would have to mirror the second matrix so that it runs from -3 to 0 Tesla. Only changing the order in the y-wave doesn't help because as I said above, the z-values are not connected to their "real" coordinates any more.
Ok, I hope I followed all the forum rules correctly and there aren't too many typos.
Thanks for any help,
Markus
X (horizontal) values are assigned to rows (just like 1D waves/vectors), and Y values are assigned to columns.
Why not use two image plots in the same graph? You'd have two left axes, one that is drawn from 0 to 50% of the plot area (for the 0 to -3 values) and one that is drawn from 50% to 100% (for the 0 to +3 values). Perhaps you will want to reverse the left axis for the 0 to -3 values.
--Jim Prouty
Software Engineer, WaveMetrics, Inc.
September 24, 2011 at 09:04 am - Permalink
In Igor image processing, one of the main questions is always whether the x and y values are linearly spaced, that is, is there a constant (non-zero) difference between adjacent x and/or y pixels? If so then you can use wave scaling to your advantage. You can read up on this by executing
DisplayHelpTopic "Waveform Model of Data".The solution to your second question is also made more straightforward if you have linearly spaced data.
So, is it? If so you'll want to load your data into a single 2D wave and adjust the scaling using
SetScale. If you need help with this then it would be easier if you posted some sample data.September 25, 2011 at 06:24 pm - Permalink
Thank you for your advice. This sounds very promising although I would have to find out first how to plot two image plots in one graph. For the moment I have to admit that I did the quick and dirty approach and just created two graphs, one twice as wide as the other. After exporting them into Quartz-pdfs I copied the image points with Illustrator out of the smaller graph and placed them into the other. With the amount of data points that are all represented by ungrouped rectangles this process was very annoying and brought my computer down several times. So rest assured that on the next occasion I will certainly try your suggestion.
Thank you,
Markus
October 2, 2011 at 11:48 am - Permalink
I am sorry but I am getting an error message (topic not found after searching all Igor help files ...) when I enter this command.
Yes, the data is linearly spaced. Here comes an example
4 -2 -9.58591602572802E-9
4 -1.998 1.35256066176515E-6
4 -1.996 2.15606204044056E-6
4 -1.994 1.26551470588236E-6
4 -1.992 -1.03785615808816E-6
4 -1.99 -1.78779044117607E-6
4 -1.988 -1.64048161764687E-6
4 -1.986 -2.75199218750001E-6
4 -1.984 -3.85680652573536E-6
4 -1.982 -4.74735340073543E-6
4 -1.98 -4.58665349264731E-6
...
4 1.986 -3.43295955882352E-5
4 1.988 -3.34323529411777E-5
4 1.99 -2.88925689338227E-5
4 1.992 -2.29801378676466E-5
4 1.994 -2.26587362132363E-5
4 1.996 -2.55714292279415E-5
4 1.998 -2.76270542279411E-5
4 2 -2.78480147058819E-5
3.95 -2 -7.71459031640625E-6
3.95 -1.998 -7.09759650735233E-7
3.95 -1.996 -1.35256020220626E-6
3.95 -1.994 -2.26989154411786E-6
3.95 -1.992 -2.40380790441147E-6
3.95 -1.99 -2.2297164522063E-6
3.95 -1.988 -2.87251700367581E-6
3.95 -1.986 -4.26525275735328E-6
...
3.95 1.988 -2.00071829044128E-5
3.95 1.99 -2.10785202205877E-5
3.95 1.992 -2.17347104779417E-5
3.95 1.994 -2.13396553308819E-5
3.95 1.996 -1.8614448529412E-5
3.95 1.998 -1.88287132352939E-5
3.95 2 -1.9029590992647E-5
3.9 -2 -7.52435803079044E-6
3.9 -1.998 3.60906020220555E-6
3.9 -1.996 1.56013143382374E-6
3.9 -1.994 4.68708639704682E-8
3.9 -1.992 -1.62039430147015E-6
3.9 -1.99 -3.08008823529405E-6
...
I think you get it ;-)
Thank you,
Markus
October 2, 2011 at 11:53 am - Permalink
Thanks once more,
Markus
October 2, 2011 at 12:13 pm - Permalink
The following function should help you out:
wave xWave, yWave, zWave
// assumptions: the x values are monotonic
// equivalent: the data is row-major (if x dimension = rows)
variable nPoints = DimSize(zWave, 0)
// get the wave scaling and the dimensions of the image
variable dx, dy = yWave[1] - yWave[0]
variable i, ySize = 1
for (i = 1; i < nPoints; i+=1)
if (xWave[i] == xWave[0])
ySize += 1
else
dx = xWave[i] - xWave[0]
break
endif
endfor
variable xSize = nPoints / ySize
// make a copy of the z-data
Duplicate /O zWave, M_Matrix
// redimension the data to a 2D matrix (image)
// there's a catch: Igor stores its data in column-major
// format, but yours is row-major. So we solve that by doing
// a transpose
Redimension /N=(ySize, xSize) M_Matrix
MatrixOP /O M_Matrix = M_Matrix^t
// finally set the wave scaling
SetScale /P x, xWave[0], dx, M_Matrix
SetScale /P y, yWave[0], dy, M_Matrix
End
I see now that it only works when "Using Igor.ihf" is open. Interesting. Anyway, to read it, type "SetScale" in the command line, right click on it, select "help for setscale", then scroll down and click on the link.
I think that the closest built-in Igor functionality is in the 'image line profiles' panel. To activate these, go to Analysis > Packages > Image Processing. A new menu, "Image", will appear, and you can select 'image line profiles' from that.
However, I think this is probably not quite what you are looking for. Programming something in Igor to mirror the Origin functionality is not difficult, but since you appear to be a new user I'll see if I can cook up something.
I think that Igor has the 'fiddle' colorscale for this type of situation, but I haven't used it, so I cannot comment on its usefulness. Sometimes selecting log scaling for the colors can help as well.
I've also found that sometimes I can uncover more detail by being able to play with the color scales more precisely and interactively. I made a procedure that allows me to do this a couple of months ago. I've now uploaded it to IgorExchange here: http://www.igorexchange.com/project/ColorScaleSliders. Give it a go, you may find it useful.
October 3, 2011 at 09:56 am - Permalink
Try
--Jim Prouty
Software Engineer, WaveMetrics, Inc.
October 4, 2011 at 12:07 am - Permalink
Try
Analysis->Packages->Image Processingand after it has loaded and compiled,Image->Image Line Profiles.John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
October 5, 2011 at 10:23 am - Permalink
Thanks! As you have probably realized by now, I am a real newbie to Igor, especially when it comes to programming and executing functions. I will read up on this in the manual and give it a try.
On the first glance this looked exactly like the feature I was looking for. However, the line trace that it displayed is not quite what I expected from the image. For instance where there is a peak visible in the image, the line trace does not show that peak. For instance, I do not understood what is the meaning of the two lines when the width parameter is set to a finite value. But here the same thing applies as already said above. I will have to read up on that feature in the manual to understand what it does.
Great! This is very nice! Being able to adjust the color scale and also the map with the live preview is very very handy! It really looks like I have to learn something about the programming possibilities of Igor. They sure seem to be very powerful!
October 8, 2011 at 03:29 am - Permalink
Thank you, that did the trick! Now some reading has to be done...
October 8, 2011 at 03:31 am - Permalink
Thank you for the tip! As said above, it doesn't exactly do what I expected it to do, but that is most certainly my fault. It sure looks promising.
October 8, 2011 at 03:39 am - Permalink
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
October 10, 2011 at 09:15 am - Permalink
Thank you once more,
Markus
October 11, 2011 at 05:46 am - Permalink
Perhaps you could send a copy of your experiment file (the .pxp file) to support@wavemetrics.com, along with a description of the steps you took to get to that screen shot.
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
October 11, 2011 at 09:03 am - Permalink
Hi!
I sent the mail out today. Hope this helps. Sorry for the long breaks between my post but I am finishing my phd-thesis at the moment and should not devote too much times to other things.
I'll look forward to your reply!
Cheers,
Markus
October 18, 2011 at 08:12 am - Permalink
Here is the content of an e-mail I sent Markus this morning:
Your file brings up a couple of issues.
1) It seems that the ImageLineProfile operation (which is the basis for the package I suggested that you use) doesn't support the use of auxiliary X and Y waves with the image. If you think about it, the concept of a width in the face of pixels of uneven width is not well defined. BUT- I think the procedure should detect the use of auxiliary waves and issue an informative error message instead of simply doing the wrong thing. Another programmer is working to add such a message.
2) You don't actually need the auxiliary waves, since your X and Y increments are all equal. If you use wave scaling instead, then the image line profile will work as advertised. I use these commands to apply wave scaling to your image wave:
setscale/P x -5.02, .005,'dI/dV (2.044V)'
setscale/P y 0, .01,'dI/dV (2.044V)'
The first command sets an X0 value of -5.02 with a delta X of 0.005. If the concept of wave scaling is not familiar, you can read about it here:
DisplayHelpTopic "The Waveform Model of Data"
I have attached a copy of your experiment file with the image scaled and re-plotted.
3) the image axes have reduced ranges, and the image profile shows the entire range. That includes a band of a couple pixels width where your image apparently has bad data. In the attached experiment file, I set the ranges of the profile graph to show just the range you had originally selected.
John Weeks
WaveMetrics, Inc.
support@wavemetrics.com
October 18, 2011 at 11:21 am - Permalink
I finished my thesis by now and would like to express many thanks for the help I got in this forum.
I am back to data evaluation now and since time is not so pressing any more, I would like to take one
step back and maybe do everything right from the beginning. And once again, help would be greatly
appreciated.
The starting point is a number of files that are named something like this
"-19d76V_800mK_08-21-B+00800.dat"
"-19d76V_800mK_08-21-B+00900.dat"
etc.
This contains some voltage, temperature, date and, most importantly in this case, a magnetic field information.
The file itself then contains about 38 lines starting with the character #. One of those lines contains for instance
the wave names. The actual data starts in, say, line 38 and is saved again in what I referred to as gnuplot format
above. That is
-019.781E+0 0.004 0.04454944704 7.951061E-1 291.746913 +004.000E+0 0.00345184 -0.000777248 0.01 3.122207E+3 2.655852E-4
-019.781E+0 0.003985 0.04317095718 7.951061E-1 291.957619 +003.985E+0 0.00334503 -0.000676159 0.01 3.122207E+3 2.655852E-4
-019.781E+0 0.00397 0.04654329498 7.951061E-1 292.165466 +003.970E+0 0.00360633 -0.000598434 0.01 3.122207E+3 2.655852E-4
-019.781E+0 0.003955 0.05044516596 7.951061E-1 292.37239 +003.955E+0 0.00390866 -0.000755314 0.01 3.122207E+3 2.655852E-4
...
-019.781E+0 -0.003965 0.00265740993 7.951061E-1 403.818239 -003.965E+0 0.000205905 0.000131221 0.0005 3.122207E+3 2.655852E-4
-019.781E+0 -0.00398 0.00293011371 7.951061E-1 404.025506 -003.980E+0 0.000227035 0.000159503 0.0005 3.122207E+3 2.655852E-4
-019.781E+0 -0.003995 0.00357360687 7.951061E-1 404.238158 -003.995E+0 0.000276895 0.000221879 0.0005 3.122207E+3 2.655852E-4
-019.780E+0 0.004 0.05638386186 7.917412E-1 407.215005 +004.000E+0 0.00436881 -0.00141621 0.01 3.127052E+3 5.272175E-4
-019.780E+0 0.003985 0.0643164057 7.917412E-1 407.409343 +003.985E+0 0.00498345 -0.000787739 0.01 3.127052E+3 5.272175E-4
-019.780E+0 0.00397 0.06605187552 7.917412E-1 407.622087 +003.970E+0 0.00511792 -0.00103761 0.01 3.127052E+3 5.272175E-4
and so on.
You see that in the first column we have what we here call "master" parameter. This is only set every other linetrace. The second
column then contains the "slave" parameter (God, I hope this is politically correct, NO OFFENSE MEANT!!!) that runs from 4mV to -4mV.
The third column finally contains the measured data. All other columns contain only monitoring parameters and can be ignored.
So, I would like to somehow automate the procedure of
1.) loading the appropriate columns as waves,
2.) turning the gnuplot format into an image format with the correct scaling and the correct wave name (taken from the filename if possible), and
3.) finally displaying all matrix waves as separate images.
For 1.) I read a bit of the DisplayHelpTopic "Loading Waves Using Igor Procedures" but the generic examples were not doing all the tricks, especially not the sensible autonaming.
For 2.) I tried the procedure that "741" posted. It basically did work but it also interpretes the empty lines that are between each step of the "master" value which it shouldn't.
For 3.) I am still completely lost.
I attached one of the files as an example to this post. Maybe it helps.
February 3, 2012 at 05:33 am - Permalink
WaveTransform zapNaNs, yWave
WaveTransform zapNaNs, zWave
To automatically determine where the data starts in the file, you will need to open the file using the Open operation, and read line-by-line using FReadLine. You will need to look for the first line containing data. This is identified by the absence of a # symbol at the start of the line.
Here is an example of a header-parsing function: http://www.igorexchange.com/node/2509
Search for ReadHeaderInfo.
Your header parser would start like this:
String pathName // Name of symbolic path. Ignored if filePath is full path.
String filePath // File name, relative path or full path
Variable& dataStartLine // Output: Zero-based line number of start of data
The calling function would pass the dataStartLine to the LoadWave /L flag.
To name the wave based on the file name, use the LoadWave /B flag. This will give you an unwieldy "liberal" name. Read about liberal names by executing this:
DisplayHelpTopic "Programming With Liberal Names"
February 3, 2012 at 07:40 pm - Permalink
It's basically just a copy and paste collection from forum posts and examples from the help files.
Now, it does in principle everything I would like it to do but probably not in an ideal and most certainly not in a flexible way.
If anyone found the time to briefly look through the code and point out the most obvious ways of improvement I would be very grateful.
One concrete thing I would like to know is how I would generate the image displays at slightly varying position so that they appear stacked rather exactly on top of each other.
Function/S DoLoadMultipleFiles()
Variable refNum, lineNumber
String message = "Select one or more files"
String outputPaths
String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
DFREF saveDFR = GetDataFolderDFR() // Get reference to current data folder
fileFilters += "All Files:.*;"
Open /D /R /MULT=1 /F=fileFilters /M=message refNum
outputPaths = S_fileName
if (strlen(outputPaths) == 0)
Print "Cancelled"
else
Variable numFilesSelected = ItemsInList(outputPaths, "\r")
Variable i, pre, field
String filename, fields
String pretext, name
for(i=0; i<numFilesSelected; i+=1)
String path = StringFromList(i, outputPaths, "\r")
Printf "%d: %s\r", i, path
filename = path[Strlen(path)-15,Strlen(path)]
print filename
sscanf filename, "%d%*[-B]%d%*[.dat]", pre, field
sprintf fields, "B%dmT", field
NewDataFolder/S $fields
ReadHeaderInfo("",path,lineNumber)
// Add commands here to load the actual waves. An example command
// is included below but you will need to modify it depending on how
// the data you are loading is organized.
//LoadWave/J/M/D/A=wave/K=0 path
LoadWave/J/D/W/A/K=0/L={lineNumber-4,lineNumber,0,0,3} path
GnuPlotToMatrix(measVgate, Vbiasdc, GacxCalc)
Display as fields;AppendImage M_Matrix;DelayUpdate
ModifyImage M_Matrix ctab= {-0.03,0.1,BlueHot,0}
SetDataFolder saveDFR
endfor
endif
return outputPaths // Will be empty if user canceled
End Function
Function GnuPlotToMatrix(xWave, yWave, zWave)
wave xWave, yWave, zWave
WaveTransform zapNaNs, xWave
WaveTransform zapNaNs, yWave
WaveTransform zapNaNs, zWave
// assumptions: the x values are monotonic
// equivalent: the data is row-major (if x dimension = rows)
variable nPoints = DimSize(zWave, 0)
// get the wave scaling and the dimensions of the image
variable dx, dy = yWave[1] - yWave[0]
variable i, ySize = 1
for (i = 1; i < nPoints; i+=1)
if (xWave[i] == xWave[0])
ySize += 1
else
dx = xWave[i] - xWave[0]
break
endif
endfor
variable xSize = nPoints / ySize
// make a copy of the z-data
Duplicate /O zWave, M_Matrix
// redimension the data to a 2D matrix (image)
// there's a catch: Igor stores its data in column-major
// format, but yours is row-major. So we solve that by doing
// a transpose
Redimension /N=(ySize, xSize) M_Matrix
MatrixOP /O M_Matrix = M_Matrix^t
// finally set the wave scaling
SetScale /P x, xWave[0], dx, M_Matrix
SetScale /P y, yWave[0], dy, M_Matrix
End Function
Function ReadHeaderInfo(pathName, filePath, lineNumber)
String pathName // Name of symbolic path. Ignored if filePath is full path.
String filePath // File name, relative path or full path
//Variable dataStartLine // Output: Zero-based line number of start of data
Variable& lineNumber
Variable refNum
String line
String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
fileFilters += "All Files:.*;"
lineNumber = 0
Open/R/F=fileFilters/P=$pathName refNum as filePath
if (refNum == 0) // File not opened - bad pathName or fileName
return -1 // Failure
endif
do
FReadLine refNum, line
if (strlen(line) == 0)
Print "File ended without finding anything"
return -1 // Failure
endif
String temp = line[0]
if (CmpStr(temp,"#") == 0)
lineNumber += 1
else
break
endif
if (lineNumber > 100)
Print "Searched 100 lines without finding the header width and height information."
return -1
endif
while(1)
return 0
End Function
Maybe this will finally do the trick and teach me some programming skills.
Thanks,
Markus
February 6, 2012 at 08:49 am - Permalink
You need to use the /W flag with the Display command. Create local variable like this outside your main for loop:
Use these values in the Display /W flag.
Then add some offset to them at the end of the for loop:
I don't see any major issues with your procedures but do have some minor comments: Here is a slightly modified version. Search for ***HR*** for my changes and comments.
#pragma rtGlobals=1 // Use modern global access method.
Function/S DoLoadMultipleFiles()
Variable refNum, lineNumber
String message = "Select one or more files"
String outputPaths
String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
DFREF saveDFR = GetDataFolderDFR() // Get reference to current data folder
fileFilters += "All Files:.*;"
Open /D /R /MULT=1 /F=fileFilters /M=message refNum
outputPaths = S_fileName
if (strlen(outputPaths) == 0)
Print "Cancelled"
else
Variable numFilesSelected = ItemsInList(outputPaths, "\r")
Variable i, pre, field
String filename, fields
String pretext, name
for(i=0; i<numFilesSelected; i+=1)
String path = StringFromList(i, outputPaths, "\r")
Printf "%d: %s\r", i, path
// ***HR*** Use ParseFilePath so this will still work if the file name length changes
//filename = path[Strlen(path)-15,Strlen(path)]
filename = ParseFilePath(0, path, ":", 1, 0) // Get last element (i.e., file name)
print filename
sscanf filename, "%d%*[-B]%d%*[.dat]", pre, field
sprintf fields, "B%dmT", field
// ***HR*** Test return value of ReadHeaderInfo
// ReadHeaderInfo("",path,lineNumber)
if (ReadHeaderInfo("",path,lineNumber) != 0)
Printf "Skipping %s because of error in ReadHeaderInfo\r", filename
continue // Continue from "for" statement
endif
NewDataFolder/S $fields
// Add commands here to load the actual waves. An example command
// is included below but you will need to modify it depending on how
// the data you are loading is organized.
//LoadWave/J/M/D/A=wave/K=0 path
LoadWave/J/D/W/A/K=0/L={lineNumber-4,lineNumber,0,0,3} path
GnuPlotToMatrix(measVgate, Vbiasdc, GacxCalc)
Display as fields;AppendImage M_Matrix;DelayUpdate
ModifyImage M_Matrix ctab= {-0.03,0.1,BlueHot,0}
SetDataFolder saveDFR
endfor
endif
return outputPaths // Will be empty if user canceled
// ***HR*** Change "End Function" to "End". Some day Igor might complain about the extraneous "Function"
// End Function
End
Function GnuPlotToMatrix(xWave, yWave, zWave)
wave xWave, yWave, zWave
WaveTransform zapNaNs, xWave
WaveTransform zapNaNs, yWave
WaveTransform zapNaNs, zWave
// assumptions: the x values are monotonic
// equivalent: the data is row-major (if x dimension = rows)
variable nPoints = DimSize(zWave, 0)
// get the wave scaling and the dimensions of the image
variable dx, dy = yWave[1] - yWave[0]
variable i, ySize = 1
for (i = 1; i < nPoints; i+=1)
if (xWave[i] == xWave[0])
ySize += 1
else
dx = xWave[i] - xWave[0]
break
endif
endfor
variable xSize = nPoints / ySize
// make a copy of the z-data
Duplicate /O zWave, M_Matrix
// redimension the data to a 2D matrix (image)
// there's a catch: Igor stores its data in column-major
// format, but yours is row-major. So we solve that by doing
// a transpose
Redimension /N=(ySize, xSize) M_Matrix
MatrixOP /O M_Matrix = M_Matrix^t
// finally set the wave scaling
SetScale /P x, xWave[0], dx, M_Matrix
SetScale /P y, yWave[0], dy, M_Matrix
// ***HR*** Change "End Function" to "End". Some day Igor might complain about the extraneous "Function"
// End Function
End
Function ReadHeaderInfo(pathName, filePath, lineNumber)
String pathName // Name of symbolic path. Ignored if filePath is full path.
String filePath // File name, relative path or full path
//Variable dataStartLine // Output: Zero-based line number of start of data
Variable& lineNumber
// ***HR*** Put a blank line after parameter declarations for better readability
Variable refNum
String line
String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
fileFilters += "All Files:.*;"
lineNumber = 0
Open/R/F=fileFilters/P=$pathName refNum as filePath
if (refNum == 0) // File not opened - bad pathName or fileName
return -1 // Failure
endif
do
FReadLine refNum, line
if (strlen(line) == 0)
Print "File ended without finding anything"
return -1 // Failure
endif
String temp = line[0]
if (CmpStr(temp,"#") == 0)
lineNumber += 1
else
break
endif
if (lineNumber > 100)
Print "Searched 100 lines without finding the header width and height information."
return -1
endif
while(1)
return 0
// ***HR*** Change "End Function" to "End". Some day Igor might complain about the extraneous "Function"
// End Function
End
February 6, 2012 at 09:51 am - Permalink
Thanks, as always this did the trick! With many graphs the graph browser proved to be a very useful tool!
Thank you for looking through the code and for your comments and additions. It works perfectly!
February 7, 2012 at 02:11 am - Permalink