Loading Wave From Weird Time Column

Share icon

Fri, 11/20/2015 - 06:19 pm

Hello All,

I am a new user of Igor Pro (version 6.3.7.2) and I have run into a snag with what should be a simple problem.

I get comma separated values files from an instrument I use in the lab which, unfortunately, come with a time column which uses a time stamp instead of something like 0, 1, 2,...,n seconds. The format of the time column in the raw file is "mm/dd/yyyy hh:mm:ss.ff [AM/PM]". The goal, ultimately, is to have the format in seconds, starting from zero and incrementing as necessary. The time step will not always be the same across different data sets, they may be every second or ten milliseconds. It appears from examining data in the past that the time step is not even constant within an experiment such that I may set the instrument to acquire data every one second but the actual values are not one second apart. So that pretty much rules out the possibility of my being able to make a new wave which starts from zero and increments by the known time step forcing me to manipulate the time stamp data into a typical format (seconds for example).

I am attaching an example of what I am dealing with at the moment as a .csv in case my description is ambiguous.

I have played with the Load Waves dialog, trying various combinations of settings for the time/date format in the Tweaks dialog but I have had not luck. I either get a wave in which each entry is simply the year or I get an empty wave. If someone has a tip on what I can do with some Load Waves options, I'm all ears but I have had no luck with it. Additionally, if someone could tip me off about how I can write a procedure whereby the Time column is treated as a different data type then the rest of the columns without the user having to play with the dialog, this would help me greatly.

If the problem can be resolved here then the rest of the post becomes an interesting exercise at most.

As a work around, I have had a bit of luck by loading the "Time" column as a text wave, which gives me a TimeW containing the information as a string (example: "2015-11-14 19:49:48.8", note that the slashes are switched to "-" by IgorPro automatically). I have gotten so far as being able to parse the string using the following commands:

Make /T TimeW

TimeW[0] = "2015-11-14 19:49:48.8"

String expr = "([[:digit:]]+)-([[:digit:]]+)-([[:digit:]]+) ([[:digit:]]+):([[:digit:]]+):([[:digit:]]+).([[:digit:]]+)" //included the plus signs using the Igor Pro manual but have no idea what they really do

String one, two, three, four, five, six, seven

SplitString/E=(expr) TimeW[0], one, two, three, four, five, six, seven


This bit of code appears to work quite well and I now have the entry parsed such that the string variables [four, five, six, seven] contain the information I could theoretically use to massage into a proper time wave. The variables [one, two, three] are not really necessary to me, I just had them in this code as I was learning about the SplitString patterns functionality. I have concatenated the strings into another string variable to reproduce the hh:mm:ss.ff format. Now my problem is that I can't seem to find a way to cast this string into a double or time or something I can use to move forward. If anyone has a tip to do this easily, I'd be eternally grateful. Otherwise, a tip as to how I may cast the string "49" into a double would be helpful as I can probably work out some code to do the rest.

Thanks for any help you can offer. ToIgorExchange.csv
Yanick wrote:
Otherwise, a tip as to how I may cast the string "49" into a double would be helpful as I can probably work out some code to do the rest.


variable var = str2num("49")

Its counterpart is num2str.

November 21, 2015 at 05:41 am - Permalink

The file that you attached can be loaded if you tell Igor the date/time format that the file is using. You do this using the Date format popup menu in the Tweaks subdialog of the Load Waves dialog (Data->Load Waves->Load Waves).

This generates this command:
LoadWave/J/D/W/E=1/K=0/R={English,2,2,2,2,"Year-Month-DayOfMonth",40} ""

which correctly loads your sample file.

In order to show the fractional seconds in the table, you need to choose Table->Format->Show Fractional Seconds.

November 22, 2015 at 06:28 am - Permalink

Hi All,

Sorry for apparently falling off the face of the earth. Both suggestions were very useful and that particular set of flags for LoadWaves appears to work perfectly!

A quick follow up question is if I can change the format from date-time to decimal via the command line. I'm hoping to write a procedure to handle these data sets mostly automatically.

Thanks again.

November 26, 2015 at 09:13 am - Permalink

I'm not entirely sure what you want to do.
To change the layout of the column in the table use
ModifyTable format(datewave)=3.
Loading the data with the /R flag stores the date and time as seconds elapsed since 1/1/1904. Hence, there is nothing to do in order to convert it to seconds.
Maybe you want to subtract the first value as an offset to obtain a duration of the experiment:
Variable Ofs=datewave[0]
datewave-=Ofs


As a general hint: Igor often displays a "To Cmd Line" button. The produced output is a good starting point to "get stuff into procedures".
HJ

November 26, 2015 at 10:50 am - Permalink

HJDrescher wrote:
I'm not entirely sure what you want to do.
To change the layout of the column in the table use
ModifyTable format(datewave)=3.
Loading the data with the /R flag stores the date and time as seconds elapsed since 1/1/1904. Hence, there is nothing to do in order to convert it to seconds.
Maybe you want to subtract the first value as an offset to obtain a duration of the experiment:
Variable Ofs=datewave[0]
datewave-=Ofs


As a general hint: Igor often displays a "To Cmd Line" button. The produced output is a good starting point to "get stuff into procedures".
HJ


Yes, you are correct in that I wanted the time wave to be elapsed time of the experiment, starting with t=0. As I am still learning IgorPro I initially tried the following commands on the loaded wave (note that my plan was never to make a new wave but to do all of the manipulation of the original TimeW, this code was just so I can have access to TimeW without having to reload the data file):

Make /N = (numpnts(TimeW)) TimeG
TimeG = TimeW
TimeG = TimeG - TimeG[0]


TimeW was:
11/25/2015 19:21:29
11/25/2015 19:21:30
11/25/2015 19:21:31
11/25/2015 19:21:33
11/25/2015 19:21:34
11/25/2015 19:21:35
11/25/2015 19:21:36
11/25/2015 19:21:37
11/25/2015 19:21:38
11/25/2015 19:21:39
11/25/2015 19:21:40

TimeG gave me:
0.000
3531324160.000
3531324160.000
3531324160.000
3531324160.000
3531324160.000
3531324160.000
3531324160.000
3531324160.000
3531324160.000
3531324160.000

However, as I was asking a lab mate for help, I input the commands as follows:

Make /N=(numpnts(TimeW)) TimeF
TimeF = TimeW - TimeW[0]


giving me a TimeF:
0.000
1.900
2.900
4.000
5.000
6.000
7.000
8.000
9.100
10.200
11.200

This is exactly what I wanted the time to look like but it appears that I cannot re-assign a wave as I did the first time (wave = wave - wave[0]). It appears I need to make new waves to hold the output each time I want to manipulate entries.

Subtracting a constant from the original wave:
Variable var = TimeW[0]
TimeW = TimeW - var

Gives a TimeW of:
00:00:00
00:00:01
00:00:02
00:00:04
00:00:05
00:00:06
00:00:07
00:00:08
00:00:09
00:00:10
00:00:11

Just something I will need to keep in mind I guess.
Thank you all for your suggestions.

November 27, 2015 at 08:48 am - Permalink

Hi,

Make /N = (numpnts(TimeW)) TimeG
TimeG = TimeW
TimeG = TimeG - TimeG[0]


Immediately changes TimeG[0] to 0; from then on it subtracts 0 from each subsequent value of TimeG. Why all output values are the same after the initial value is 0 is somewhat mysterious unless the display doesn't show sufficient resolution.

As HJ showed (and you discovered) this problem can be avoided by first copying TimeG[0] to a variable and subtracting this variable from each entry in TimeG:

Variable Ofs=datewave[0]
datewave-=Ofs


This, of course, still leaves the original wave changed. If this is a problem then you will need to create a new wave to hold the output.

November 27, 2015 at 09:20 am - Permalink

Maybe you want to store only the offset of the wave in a variable (like in my example) and use it as an offset for other processes, e.g., graphing (see x-offset)?
A general remark to the behavior you found: It it quite common and should be that way. For each data point on the left the expression on the right is calculated. If the left side expression is also on the right side it influences the result. This is why you need the (temporary) variable.
Looking at your example data it seems that your data points are pseudo-equally spaced by 10 sec. I don't know your exact experiment and the necessary time resolution, but you might consider to abandon the time wave and use wave scaling instead (it has a lot of benefits!). ->
displayhelptopic "The Waveform Model of Data"
HJ

November 28, 2015 at 03:13 am - Permalink

jtigor wrote:

Make /N = (numpnts(TimeW)) TimeG
TimeG = TimeW
TimeG = TimeG - TimeG[0]


Immediately changes TimeG[0] to 0; from then on it subtracts 0 from each subsequent value of TimeG. Why all output values are the same after the initial value is 0 is somewhat mysterious unless the display doesn't show sufficient resolution.


The mystery lies in the fact that date/time waves require double precision (/D) waves ...
Make /D /N=(numpnts(TimeW)) TimeG

HJ

November 28, 2015 at 03:19 am - Permalink