Loading a .csv file
Karen
Thu, 08/11/2011 - 09:22 am
I am having trouble loading the attached .csv file with comma delimited headers. I have created the .csv file from a .na (nasa ames) file and comma delimited the wave name text but igor isn't pick it up as wave names. There are 189 lines in the header and I have told igor that the column headers are in row 188 (factoring in that iogr starts from row 0) and the first row of data in in row 189, but I either get 1 long string or wave0 etc. Any suggestions on which might be missing from my comma delimited header?
Cheers,
karen
The attached is major overkill, but should do what you want (with some changes ;-). Using FindFirstDataLine() gives some flexibility in the size of the header following the last all numeric header line. You can also use it to concatenate several flights (files for everyone else) by changing the MODE on the first line of the first function. Choices are to read a single CSV, concatenate all CSVs in a directory, or concatenate all CSVs in a directory tree.
GrepString seems to be acting up on my computer, so this only works for headers with capitals in.
I'm now going to be lambasted for taking several hundred lines of code to do what a built in function could have done in 5. Just watch - I bet you.
August 12, 2011 at 03:10 am - Permalink
Igor is not happy with the name that starts "1,3-butadiene_pptV". It looks to Igor (simpleton that he/she/it is:) like it's supposed to be the number 1 followed by a comma delimiter. Igor says "the user is giving me data when I expect a name so I will through in the towel."
A solution is to change "1,3-butadiene_pptV" to "1-3-butadiene_pptV".
Your column names are very long and may be cumbersome to work with so I have come up with another solution. This solution will work only if you have a number of files with the exact same columns. The solution is to tell Igor the wave names directly, using the LoadWave /B flag, rather than asking Igor to read the names from the file.
I have included a function that does this below.
Some of the names were longer than Igor's 31 character limit for wave names so I shortened them. If I were you, I would shorten and simplify some of the other names too to get wave names that are easier to work with. For example, I would change "WindVelocityv(North)(ms-1)'" to "WindNorth".
My function also converts your date column, which is in YYYYMMDD format, into Igor date/time format (seconds since 1904-01-01).
String columnInfoStr = ""
columnInfoStr += "N='StartTime_UTC';"
columnInfoStr += "N='StopTime_UTC';"
columnInfoStr += "N='MidTime_UTC';"
columnInfoStr += "N='Date_(YYYYMMDD)',T=4;" // Force to double-precision
columnInfoStr += "N='RVSMairspeed(ms-1)';"
columnInfoStr += "N='Deicedtemperature(degK)';"
columnInfoStr += "N='Staticpressure(hPa)';"
columnInfoStr += "N='RVSMpressurealtitude(m)';"
columnInfoStr += "N='Cabinpressure(hPa)';"
columnInfoStr += "N='Cabintemperatureatcoreconsoles';" // Was 'Cabintemperatureatcoreconsoles(degC)'
columnInfoStr += "N='DewPoint(K)';"
columnInfoStr += "N='XR5GPSLatitude';"
columnInfoStr += "N='XR5GPSLongitude';"
columnInfoStr += "N='XR5GPSAltitude';"
columnInfoStr += "N='Solarzenithangle';"
columnInfoStr += "N='WindVelocityv(North)(ms-1)';"
columnInfoStr += "N='WindVelocityu(East)(ms-1)';"
columnInfoStr += "N='WindVelocityw(vertical)(ms-1)';"
columnInfoStr += "N='TECO49O3(ppb)';"
columnInfoStr += "N='TECO42NO(ppb)';"
columnInfoStr += "N='TECO42NO2(ppb)';"
columnInfoStr += "N='TECO42NOx(ppb)';"
columnInfoStr += "N='10sfastNOxNO(ppt)';"
columnInfoStr += "N='10sfastNOxNO2(ppt)';"
columnInfoStr += "N='1sfastNOxNO(ppt)';"
columnInfoStr += "N='1sfastNOxNO2(ppt)';"
columnInfoStr += "N='Ethane_pptV';"
columnInfoStr += "N='Propane_pptV';"
columnInfoStr += "N='Iso-butane_pptV';"
columnInfoStr += "N='n-butane_pptV';"
columnInfoStr += "N='Cyclopentane_pptV';"
columnInfoStr += "N='Iso-pentane_pptV';"
columnInfoStr += "N='n-pentane_pptV';"
columnInfoStr += "N='2+3-methylpentane_pptV';"
columnInfoStr += "N='n-hexane_pptV';"
columnInfoStr += "N='n-heptane_pptV';"
columnInfoStr += "N='n-octane_pptV';"
columnInfoStr += "N='Ethene_pptV';"
columnInfoStr += "N='Propene_pptV';"
columnInfoStr += "N='acetylene_pptV';"
columnInfoStr += "N='Trans-2-butene_pptV';"
columnInfoStr += "N='1-butene_pptV';"
columnInfoStr += "N='iso-butene_pptV';"
columnInfoStr += "N='cis-2-butene_pptV';"
columnInfoStr += "N='1,3,butadiene_pptV';" // Fixed error here after original post to IgorExchange
columnInfoStr += "N='Propyne_pptV';"
columnInfoStr += "N='trans-2-pentene_pptV';"
columnInfoStr += "N='1-pentene_pptV';"
columnInfoStr += "N='isoprene_pptV';"
columnInfoStr += "N='PAN_ppbv';"
columnInfoStr += "N='OH_(10^6molecule_cm-3)';"
columnInfoStr += "N='HO2_(10^8moleculecm-3)';"
columnInfoStr += "N='CO(ppbv)';"
columnInfoStr += "N='BlueTSI3563nephelometer';" // Was UncorrectedbluetotalscatteringcoefficientfromTSI3563nephelometer.(m-1)
columnInfoStr += "N='GreenTSI3563nephelometer';" // Was UncorrectedgreentotalscatteringcoefficientfromTSI3563nephelometer.(m-1)
columnInfoStr += "N='RedTSI3563nephelometer';" // Was UncorrectedredtotalscatteringcoefficientfromTSI3563nephelometer.(m-1)
columnInfoStr += "N='COFromAEROAL5002';" // Was MolefractionofCarbonMonoxideinairfromtheAEROAL5002instrument(ppb)
columnInfoStr += "N='LIFNO2ppb';"
columnInfoStr += "N='LIFRO2NO2ppb';"
columnInfoStr += "N='LIFRONO2ppb';"
columnInfoStr += "N='NO3ppt';"
columnInfoStr += "N='N2O5+NO3ppt';"
columnInfoStr += "N='Sulphate';"
columnInfoStr += "N='Ammonium';"
columnInfoStr += "N='Organics';"
columnInfoStr += "N='Nitrate';"
columnInfoStr += "N='Chloride';"
columnInfoStr += "N='HCHO(pptv)';"
columnInfoStr += "N='RO2+HO2_pptv';"
columnInfoStr += "N='mje_THETA_(K)';"
columnInfoStr += "N='mje_total_num_density_(#cm-3)';"
columnInfoStr += "N='mje_specific_humidity_(kg/kg)';"
columnInfoStr += "N='mje_water_mixing_ratio_(ppm)';"
columnInfoStr += "N='mje_relative_humidity_(%)';"
columnInfoStr += "N='mje_equivalent_potential_temp';" // Was mje_equivalent_potential_temperature_(K)'
columnInfoStr += "N='mje_average_wind_direction';" // Was mje_average_wind_direction_(degN)
columnInfoStr += "N='mje_average_wind_speed_(ms-1)';"
LoadWave/J/D/E=1/K=0/L={0,189,0,0,0}/B=columnInfoStr/Q
if (V_flag > 0)
// Correct date wave
Wave dw = 'Date_(YYYYMMDD)'
dw = YYYYMMDDToIgorDate(dw)
SetScale d, 0, 0, "dat", dw // Mark as date/time wave
ModifyTable format(dw) = 6 // Display as date
endif
End
Function YYYYMMDDToIgorDate(dt)
Variable dt // Date value expressed as YYYYMMDD
Variable year = trunc(dt / 10000)
dt -= year * 10000
Variable month = trunc(dt / 100)
dt -= 100 * month
Variable day = dt
Variable result = Date2Secs(year, month, day)
return result
End
August 11, 2011 at 06:15 pm - Permalink
August 12, 2011 at 02:01 am - Permalink