sorting time series by month and time of day

Share icon

Fri, 09/02/2011 - 02:38 am

I need some help please programming a routine to sort my data. I have a multi-year data set with hourly averages of particle concentration, so basically I have a times series with a value every hour for ~5 years. The time format in igor is currently dd/mm/yyyy hh:mm.

I would like to create a program where the data is split up according to its time of day (so all the values that happened at noon would be in one wave and all the values at 13:00 would be in another, etc). But I don't know how to use only the time part of my date and time wave to pull data out. You see, I was thinking of some sort of "if then" loop to parse out the values into waves, but I can't figure out how to refer to only the time part of my date and time wave (e.g. if value was measured at 1:00 put value in wave1, if value was measured at 2:00 put in wave2). No clue if that is the best way, just the only way I thought of it! So I have searched through the programs on the site for help with parsing out data using the time from my date and time wave and have found where I can parse the date and time (e.g. ParseDateStringUSNumeric(str)), but those are for strings and my date and time wave is not a string. So I am not certain what to do.

Also, I would like to alternatively sort these data by the month (e.g., all values that happened in January in one wave and then all the values that happened in February in another etc). But again, I just can't figure out how to use parts of my date and time wave to put values into different waves depending upon the month of the measurement.

Does that make sense? Does anyone have any ideas on how to do this? How to refer to only parts of a date and time wave, or alternatively how to just extract parts from a date and time wave? I am just not very good at programming, so am at a loss! Any help would be greatly appreciated.

Thanks


Here are a couple points in my time series if it's helpful to see what I am talking about

26/07/2006 08:00:00 68.5
26/07/2006 09:00:00 74
26/07/2006 10:00:00 85.9
26/07/2006 11:00:00 129.1
26/07/2006 12:00:00 99.6
26/07/2006 13:00:00 102.6
26/07/2006 14:00:00 77.8
Within Igor the master clock is the number of seconds since 1/1/1904 (beginning of time). You need to convert the string to seconds and then sort. The first part is convert the date to secs and there is a built in function for that

date2secs(year, month, day )

The second part converting the time of day to seconds does not have a built in function, but is very straight forward - parse the time code by hours and minutes.

Example TimeStamp = 26/07/2006 08:00:00 68.5

So in general take the string, Timestamp
parse item 0 using " "(space) as delimiter [StringFromList(0, TimeStamp, " ")] and apply date2secs(year, month, day )
parse item 1 using " "(space) as delimiter [StringFromList(1, TimeStamp, " ")]and then parse that resulting string using ":" as delimiter
[StringFromList(0, SubString, ":")]multiply and sum as appropriate
parse item 2 using " "(space) as delimiter [StringFromList(0, TimeStamp, " ")] and add to the result after converting to a num str2num



result is the total number of seconds since beginning of time.

September 2, 2011 at 08:00 am - Permalink

You can determine the time of day from a date/time value using the mod function:
// Get time of day from date/time value
timeOfDay = mod(dateTimeValue, 24*60*60)


You can determine the month from a date/time value by generating a date string using Secs2Date and then parsing the string using sscanf:
String dateStr = Secs2Date(wIn[point], -2)      // e.g., 2011-09-01
Variable year, month, day
sscanf dateStr, "%d-%d-%d", year, month, day


I have used these ideas to create the start of a solution to your challenge. See the attached Igor experiment. Open the experiment and then read the comments for the procedures. Then look at the commands that I executed (after creating the sample data manually).
Identify Points Based on DateTime.pxp

September 3, 2011 at 03:54 am - Permalink

Thank you both so much for the responses, they were very helpful. And thank you hrodstein for the sample code. That made so much sense to see it written out (as a non-programmer, I struggle sometimes with putting all this info into practice). I was able to follow your program very easily and changed it to fit my data and it worked! Thank you for your help, it made my day.

September 5, 2011 at 04:53 am - Permalink

Pls find attached a function I am trying to modify to help sort a wave acquired for two weeks. I want to be able to sort day or night data based on the time. I tried modifying your igor you posted but no avail. Pls help.
my time wave is let sat's time_all and my data is no2_all.

macro TimeofdayFilter(wIn, startTOD, endTOD, wOutName)
string wIn
Variable startTOD // In seconds, e.g., 12*60*60 for 12:00
Variable endTOD // In seconds, e.g., 13*60*60 for 13:00
prompt wIn, "Wave to be sorted", popup WaveList("*",";","")
String wOutName
IdentifyPointsBasedOnTimeOfDay(wIn, startTOD, endTOD, wOutName)
end macro


Function IdentifyPointsBasedOnTimeOfDay(wIn, startTOD, endTOD, wOutName)
Wave wIn
Variable startTOD // In seconds, e.g., 12*60*60 for 12:00
Variable endTOD // In seconds, e.g., 13*60*60 for 13:00
String wOutName // Name for output wave

Variable numInputPoints = numpnts(wIn)
Make/O/N=(numInputPoints) $wOutName
Wave wOut = $wOutName

Variable numOutputPoints = 0

Variable point
for(point=0; point Variable timeOfDay = mod(wIn[point], 24*60*60)
if (timeOfDay >= startTOD)
if (timeOfDay < endTOD)
wOut[numOutputPoints] = point
numOutputPoints += 1
endif
endif
endfor

Redimension/N=(numOutputPoints) wOut
return numOutputPoints
End

September 11, 2012 at 01:58 pm - Permalink