Odd format for loading

Share icon

Mon, 07/13/2015 - 01:50 am

I've attached a smple file of data. I would like to make a function for loading the data. The big problem is that they only put the original date in the first line and then just have a time wave for the rest of the data lines. The date changes at midnight, but there is no date wave to specify this... Is there a way to create a function that will take that starting date and make it it's own wave where it jumps up a day every time midnight comes around? It is an annoying way that they post the data, but its what I have to work with and adding the date all the way down is tedious. 2-2-15.txt
The easiest way would be to just convert your time data to real date+time. First, you would need to write a parser which extracts the date information from the first line and then load the data columns. Add the date to the time data using date2secs (doing it the manual way in your case would be  mytimedata += date2secs(2015,2,2)). Then have a routine find the day crossing and shift by one day from there. I have written a short function which does just that. It should be not too much hassle to fully automate all these steps. You could even write a catch function which processes your data immediately upon loading (I am not sure if you are familiar with hook functions, so I leave it at that for now). Here the shift function using EdgeStats:
Function Shiftdate(timewave)
    wave timewave
    EdgeStats/Q/P timewave  // find an 'edge' = change of day in time data (assumes only one change)
    timewave[floor(V_EdgeLoc2)+1, ] += 24*60*60 // shift the time by one day from edge start
End

I also attach an Igor experiment where I adjusted you time data.
2-2-15 time shift.pxp

July 13, 2015 at 04:18 am - Permalink

I believe that the Unwrap operation will fix your time values. Assuming the times are loaded into wave0 using Igor's LoadWave/J and therefore are in Igor time format (seconds since midnight):
Unwrap 24*60*60, wave0


Then,
wave0 += Date2Secs(2014, 02, 02)  // Add date represented as seconds since 1904-01-01


To test this, I opened the file as an Igor notebook (File->Open->Notebook). I then copied the data from the file, excluding the first two lines, into the clipboard. I then used Data->Load Waves->Load Waves and selected Delimited Text (/J). I clicked the Make table, Load From Clipboard, Double-precision, and Auto Name checkboxes and unchecked the rest. I then clicked Do It.

I then executed the following commands:
Display wave0               // Shows sawtooth
Unwrap 24*60*60, wave0      // Straightens it to straight line


To determine if your date/time data is evenly-spaced, I executed the following commands:
Make/D/N=(numpnts(wave0)-1), dif = wave0[p+1] - wave0[p]
Display dif
ShowInfo
Cursor A, dif, 20


From this I see that your data is mostly every 10 seconds but occasionally has 11-second spacing. You could convert this to waveform data on 10-second intervals by interpolation with little loss of fidelity.

July 13, 2015 at 10:35 am - Permalink