XPS data from Phy Multipak

hegedus
Posts: 311
Joined: 2009-03-21
Location: United States

Hi,

Has anyone have experience with .spe file from a Phy Multipak? I have a data file (attached) that I am trying to extract the data from. There should be 6 spectra that are defined in the header. Simpler files with only 1 data range I can sort of get, though there is a bit of garbage at the beginning of the wave after export.

edit: The exported CSV file is also attached.

My goal is to extract the data directly, because the export software does not support batch processing.

The multiple ranges a bit of a stump for me. Anyone with a magic .ipf that they can share?

Andy

AttachmentSize
C0ELR081_033.spe_.zip4.04 KB
C0ELR081_HR_50_30.csv11.27 KB

[ last edited April 14, 2014 - 14:56 ]
ChrLie
Posts: 266
Joined: 2009-01-06
Location: Switzerland

Here is one way, starting from the csv file:

function FileLoader(Key, base, skip)		
	string key					// keyword after which a spectrum begins
	string base					// base name for spectra		
	variable skip				// line to skip after keyword
 
	variable refNum
	Open/D/R/F="*.csv"/M="Select SPE-CSV File" refNum
	string filePath = S_fileName
	Open/R refNum as filepath
 
	Make/FREE/O/T/N=0 w
 
	variable i =0		
	do
		string line
		FReadLine refNum, line
 
		if( strlen(line) == 0 )							// end of file
			break										// exit do-while loop
		endif
 
		line = RemoveEnding(line, "\r")
 
		InsertPoints i, 1, w
 
		w[i] = line
		i+=1			
 
	while( 1 )
	Close refNum
 
 
	i=0
	variable  filenum = 0
 
	for(i=0; i<numpnts(w); i+=1)
 
		if(cmpstr(w[i], key) == 0)
			i+=skip
			Make/O/N=(numpnts(w)) $base + num2str(filenum)
			wave ww = $base + num2str(filenum)
			ww = NaN
		endif
 
		string ThisLine = w[i]
		variable strStart = strsearch(ThisLine, ",", 0)+1
		ww[i] = str2num(ThisLine[strStart, inf])
 
		if(strlen(w[i]) == 0)
			filenum +=1
			WaveTransform zapNaNs ww
		endif	
	endfor
end

try:

FileLoader("Area5", "data", 4)

The entire file is first loaded as text wave using FReadLine (maybe LoadWave would be easier), then I use a keyword (here "Area5") to figure out where a spectrum begins.
You need to add a bit of code to either extract x-data or to set the wave scaling.


[ last edited April 15, 2014 - 00:42 ]
hegedus
Posts: 311
Joined: 2009-03-21
Location: United States

Hi ChrLie,

Thank you very much for your code. I used it as a starting point and produced this using some patterns in the file. I would really love to be able to do this off of the .spe file. It has a text header with a lot of information needed including the number of spectra and the ranges to set the scaling. The actual data is in binary and I have not been successful in parsing it into the individual spectra.

Function ImportXPS()
 
	//get file
	variable refNum
	Open/D/R/F="*.csv"/M="Select SPE-CSV File" refNum
	string filePath = S_fileName
	Open/R refNum as filepath
 
	Make/O/T/N=0 w
	Make/O/N=0 SpectraStart
 
	variable i,j =0		
	do
		string line
		FReadLine refNum, line
 
		if( strlen(line) == 0 )							// end of file
			break										// exit do-while loop
		endif
 
		line = RemoveEnding(line, "\r")
 
		InsertPoints i, 1, w
		//check to see if it is the start of a new spectra and note the line number
		if(stringmatch(line,"*Area*")==1)
			InsertPoints j, 1, SpectraStart
 			SpectraStart[j] = i
 			j +=1
		endif
		w[i] = line
		i+=1			
 
	while( 1 )
	Close refNum
 
	//make individual spectra
	variable SpectraIndex,index,offset
 
 
	For(SpectraIndex=0;SpectraIndex<numpnts(SpectraStart);SpectraIndex +=1)
 
		//Read in data
		Make/O/N=(0,2) DataIn
		offset = spectrastart[spectraindex]+4
		index=0
		Do
			if(strlen(w[(offset+index)])==0)
				break
			endif
			insertpoints index,1, Datain
			DataIn[index][0] = str2num(stringfromlist(0,w[(offset+index)],","))
			DataIn[index][1] = str2num(stringfromlist(1,w[(offset+index)],","))
			index +=1
//			print index
		while(1)
		//Make Spectra with Name
		Make/O/N=(dimsize(datain,0)) $w[spectraStart[spectraIndex]+2]
		Wave DataOut = $w[spectraStart[spectraIndex]+2]
		Dataout[]=Datain[p][1]
		//Set wave Scaling
		SetScale/I x, datain[0][0], datain[dimsize(datain,0)-1],  dataout
	EndFor
End


thomas_braun
Posts: 569
Joined: 2009-10-07
Location: Germany

Is the PHY Multipak format standardized?
In [1] I've found a file format specification for SPE files, maybe these are the same formats.

Thomas

[1]: ftp://ftp.princetoninstruments.com/Public/Manuals/Princeton%20Instrument...


hegedus
Posts: 311
Joined: 2009-03-21
Location: United States

Hi Thomas,
Same extension different files. Thanks for the idea.

Andy


jtigor
Posts: 401
Joined: 2007-09-04
Location: United States

Andy,

Using a hex file editor, it isn't too difficult to figure out how to load the spectra. See the code below. The purpose of some elements of the file aren't clear (to me); however, the procedure loads these values into waves. The values are interpreted in ways that seem to make sense (i.e. as integers or floats) but I may have guessed incorrectly. Hopefully, they will make some sense to you. Also it wasn't clear how to get the wavelength information, but, maybe you already know how to do this. Oh, yes, this procedure will likely need modification if you want to load files with spectra from more than one position.

The procedure is embedded here and as a ipf file. Also attached is an experiment file with the results for your example file.

Hope this helps.

Jeff

#pragma rtGlobals=3		// Use modern global access method and strict wave access.
 
Function ImportXPS_bin()
//Read .spe files from Phy Multipak XPS 
 	Variable vFileRef
 	Variable vIndex = 0
 	Variable vLoc
 	Variable vNumSpectra
 	Variable vPointsInSpectrum
 	Variable vReadVal
 	String sFilePath
 	String sLine
 	String sSpectrumWaveName
 
//Get name/path to file to open
	Open/D/R/F="Multipak Files (*.spe):.spe;"/M="Select SPE-CSV File" vFileRef
	sFilePath = S_fileName
	if( StrLen( S_fileName ) == 0 )
		return -1	//user cancelled file selection
	endif
 
	Open/R vFileRef as sFilePath
 
//Text Header 
	Make/O/T/N=0 wHeader
 
//Read the File Header one line at at time; read terminates on LF/CR combinations, per the help file
//In this case the terminator is LF
	Do
		FReadLine vFileRef, sLine
 		FStatus vFileRef
 		if( V_filePos > V_logEOF )
 			print "Attempting to read past end of file"
 			return -1	//attempting to read past end of file
		endif
 
		sLine = RemoveEnding( sLine, "\r" )
 
		InsertPoints vIndex, 1, wHeader
		wHeader[vIndex] = sLine		
	//get number of spectra in file from header containing 
		if( stringmatch( sLine,"NoSpectralReg:*" ) == 1 )
			vLoc = strsearch( sLine, ":", 0 )
			vNumSpectra = str2num( sLine[vLoc + 1, inf ] )
			print "Number of Spectra in file: ", vNumSpectra
		endif
 
		//End of header is given by "EOFH" string; quit when this is found
		if( stringmatch( sLine,"EOFH" ) == 1 )
			print "End of header found"
			FStatus vFileRef
			print "Current file position: ", V_filePos
			break
		endif
		vIndex += 1			
	While( 1 )
 
//Binary headers
	//brief (16 bytes) header; seems to be 4 4 byte unsigned integer values, second valued may give number of spectra
	//otherwise the purpose of this part is not clear
	Make/O/N=4 wBinHeader_1
	//each spectrum gets 24 unsigned 4 byte integers; fifth value is number of points in corresponding spectrum
	Make/O/N=( 24 * vNumSpectra ) wSpectraHeader	 
	//read the binary headers
	FBinRead/B=0/F=3/U vFileRef, wBinHeader_1
	FStatus vFileRef
	print "Current file position after bin header read: ", V_filePos
	FBinRead/B=0/F=3/U vFileRef, wSpectraHeader
	FStatus vFileRef
	print "Current file position after spectra header read: ", V_filePos
 
//Mystery value at end of each spectrum...
	//4 byte floating value at end of spectrum; don't know what it is for; read it anyway
	Make/O/N=( vNumSpectra ) wMysteryBytes	
 
//load each spectrum into a separate wave
	for( vIndex = 0; vIndex < vNumSpectra; vIndex += 1 )
	//name for spectrum wave; name is "wSpectrum_" plus number corresponding to position in file
		sprintf sSpectrumWaveName, "wSpectrum_%02d", vIndex + 1
		print sSpectrumWaveName
	//get number of points to load for the spectrum
		vPointsInSpectrum = wSpectraHeader[ 5 + vIndex * 24 ]
	//make the wave for the current spectrum
		Make/O/N=( vPointsInSpectrum ) $sSpectrumWaveName
		Wave wSpectrum = $sSpectrumWaveName
	//read spectrum; format is 64 bit float		
		FBinRead/B=0/F=5 vFileRef, wSpectrum
//deal with 4 bytes at end of a spectrum; either read into wave just for this or skip
//code for skipping is commented out
	//read 4 bytes at end of spectrum, interpret at 32 bit float
		FBinRead/B=0/F=4 vFileRef, vReadVal
		wMysteryBytes[ vIndex ] = vReadVal
//	//next 4 bytes have unknown purpose, skip for now
//		FStatus vFileRef
//		FSetPos vFileRef, V_filePos + 4
	endfor
 
	Close vFileRef
End 

AttachmentSize
Phy_Multipak_Loader.pxp49.23 KB
Phy_Multipak_SPE_Loader.ipf3.41 KB

McIlhenny
Posts: 7
Joined: 2012-04-29
Location: Germany

How did you know how to interpret the *.spe binary file to get out the data?
I'm also working on a PHI machine and I'm also interested in directly loading the spe data.


jtigor
Posts: 401
Joined: 2007-09-04
Location: United States

McIlhenny wrote:
How did you know how to interpret the *.spe binary file to get out the data?
I'm also working on a PHI machine and I'm also interested in directly loading the spe data.

I used an application (a "hex editor", see the Wikipedia article.. http://en.wikipedia.org/wiki/Hex_editor) that permits viewing the contents of a file in binary form, bit by bit, if necessary. The text header in the file gave insights into interpreting the binary data and, certainly, having the text export version of the spe file was a big help. With those hints, you have to use the hex editor to examine the file for clues to its structure and make some guesses.

There was more information in the binary portion of the file that I did not understand. This was saved in a couple of other waves that, may make sense to someone with experience using the instrument. Hopefully, those mysterious portions were interpreted (as integer or floating point values) correctly.

Also, as I mentioned in my original post, it appears that spectra from different regions of the sample can be combined in the same data file. I don't think the code provided will handle this without some changes.

By the way, are you able to load your data with the Igor procedure that was posted?


[ last edited April 20, 2014 - 15:19 ]
hegedus
Posts: 311
Joined: 2009-03-21
Location: United States

Hi Jeff,

Thank you very much. I have never used a hex editor and this is certainly a good place to start. The spectra do make sense and match the ASCII values so your guesses were correct.

This great!

Thanks again.

Andy


realMAX
Posts: 3
Joined: 2016-04-24
Location: South Korea

Jeff,

Thanks for your positng.
Recently I strongly require data loader of the .SPE files.
The pxp or ipf file you uploaded gives an error "While sxecuting FBinRead, the following error occured: logical end-of file was reached unexpectedly uring read operation." when I execute 'importXPS_bin()".
I attach an example of SPE files.
Can you check it up? I am beginner of igor program.

JW

AttachmentSize
BP.118.zip6.01 KB

jtigor
Posts: 401
Joined: 2007-09-04
Location: United States

realMAX wrote:
Jeff,

Thanks for your positng.
Recently I strongly require data loader of the .SPE files.
The pxp or ipf file you uploaded gives an error "While sxecuting FBinRead, the following error occured: logical end-of file was reached unexpectedly uring read operation." when I execute 'importXPS_bin()".
I attach an example of SPE files.
Can you check it up? I am beginner of igor program.

JW

I will look into this.

Jeff


jjweimer
jjweimer's picture
Posts: 1327
Joined: 2007-08-14
Location: United States

I have started a project to generate a functioning PHI SPE file loader.

http://www.igorexchange.com/project/phispefileloader

Please contact me when you would like to join in the project.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH


jjweimer
jjweimer's picture
Posts: 1327
Joined: 2007-08-14
Location: United States

The 0.3beta version should load the spectra correctly.

FWIW, the mistake in the existing code was to read "mystery bytes" at the end of each spectrum.

Please test and confirm. I'll post a non-beta release based on feedback.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH


realMAX
Posts: 3
Joined: 2016-04-24
Location: South Korea

Thanks a lot....jjweimer
I tested your IGOR.6.30.x-1.0 on both Igor versions 6.37 and 7.02.
As you noted, the error message always pops up.

In addition, there are several issues as the following.
The spe file that I uploaded consists of four spectra. But your macro only loads one of them.
I need a single wave with proper parameters such as row number, start, step per each spectrum. Total 7 waves are not necessary at all.
As you compare the Igor display with real spectrum in the attached figure, spectral shape and scale change a lot. Even the background is removed. I do not understand what happens...

Longing for better version,
realMAX

AttachmentSize
BP118.jpg186.79 KB

jjweimer
jjweimer's picture
Posts: 1327
Joined: 2007-08-14
Location: United States

realMAX wrote:
Thanks a lot....jjweimer
I tested your IGOR.6.30.x-1.0 on both Igor versions 6.37 and 7.02.
As you noted, the error message always pops up.

Would you do me the favor and ...

* Log the error report to the Package (so that I can keep track of it there)
* Include a ZIP archive of your SPE file (so that I can troubleshoot on hand the data you have)

realMAX wrote:
The spe file that I uploaded consists of four spectra. But your macro only loads one of them.

I suspect this is because the SPE file that I tested only had one "region", whereas yours has more than one.

realMAX wrote:
I need a single wave with proper parameters such as row number, start, step per each spectrum. Total 7 waves are not necessary at all.

I prefer not to create wave formats that are different than what are currently used. The loader can always be run with a function that subsequently cleans up behind it or does other formatting/calculating steps, for example ...

Function MyLoadSPE()
 
    LoadPHISPE()
    setscale/I ...
    killwaves/Z ....
    return 0
end

As long as the data are all loaded correctly, decisions about what to do with the values and waves can be left to each user.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH


realMAX
Posts: 3
Joined: 2016-04-24
Location: South Korea

jjweimer, Thanks

The error message is as follows
"While executing Make, the following error occurred: Expected name"
The zip file is attached again, the same file above BP_118. This is composed of 4 spectra.

Most serious problem is that the imported wave is different from real spectrum as I reported yesturday.

realMAX

AttachmentSize
BP.118.zip6.01 KB

jjweimer
jjweimer's picture
Posts: 1327
Joined: 2007-08-14
Location: United States

OK. This took is a bit longer than I anticipated. I found the problem. Each "region" has five associated sub-regions.

SpectralRegDef: the XPS region definition
SpectralRegDef2: ??
SpectralRegDefBackgrond: ??
SpectralRegHero: ??
SpectralRegIR: ??

I will adjust the loader to skip past everything except the XPS regions.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH


Back to top