Summation of columns of a multi-dimensional wave

Share icon

Sat, 08/04/2012 - 01:59 pm

In the function below I would have liked to use a multi-dimensional wave ResidsSquared after defining this wave through the command Make/O/N=18 (Length-1, 18), so that I can separately approach the ResidsSquared wave for the 18 iterations of i. However, in the manual I read that WaveStats cannot beproperly used for multi-dimensional waves.

I would like to use the WaveStats command because I am interested in the 18 sums of the ResidsSquared wave. That is, the sums for i=0 up to i=17. Does anybody know a solution to my problem? Any suggestions are highly welcome.

Eduard

Function Local(Cumulative, Length) // Function to cacluate detrended fluctuation
    Wave Cumulative // Read series of integrated share price returns
    Variable Length // Number of points in share price series
   
    // The fluctuation is determined for eighteen different scales. The scales relate to an
    // integrated series of share price returns with 4,380 observations
    // The wave NumbSegments shows the number of segments in which the series with 4,380 observations
    // can be divided. The total series can be subdivided in 480non-overlapping segments with a length
    //(scale) of 8, 240 non-overlapping segments with a length of 16, etc.  
   
    Make/O/N=18 Scale= {8,16,12,32,40,48,64,80,96,120,128,160,192,240,256,320,384,480}
    Make/O/N=18 NumbSegments = {480,240,160,120,96,80,60,48,40,32,30,24,20,16,15,12,10,8}
   
    Make/O/N=18 SumSquaredResids // Create a wave with 18 points to store the sum of each of the
                             // squared residual series
   
    Variable i ,j  // Create local variables i and j
   
    for (i=0; i<18; i +=1)  // Repeat commands 18 times from i=1 to i=18
   
        Variable segmlength = Scale[i]  // Segment length varies from 8 when i=0 to 480, when  i=17
        Variable nsegments = NumbSegments[i]    // Number of segments varies from 480 when i=0  and
                                    // segment length is 8 to 8 when i=17 and segment length = 480
                               
   
        Make/O/N=(Length-1) Xwave = mod(p, segmlength) +1   // Create Xwave series; X runs from
                                                    // 1 up to the final point in a scale,
                                                    // for a scale / segment length of 8, the
                                                    // X wave runs from 1 up to 8
   
            for (j=0; j<nsegments; j += 1)  // Repeat CurveFit function for each of the segments
                                    // when i=0 CurveFit is repeated 480 times in accordance with
                                    // the number of segments; when i=17, Curve Fit is repeated
                                    // 8 times, for each of the segments (scales) with a length of
                                    // 480 observations
                                                                           
            // Determine local trend per segment (scale) as well as residuals against local trend. Square the
            // residuals
            // Create new wave ResidsSquaredd. Store squared residuals to this series.
            // Next determine sum of each of the ResidsSquared eighteen times (for i=1 to i=17) and store
            // the sums to the wave SumSquaredResids
       
            CurveFit/Q Line Cumulative[j*segmlength, (j+1)*segmlength -1] /X=Xwave /D /R /A=0
                Wave Res_Cumulative
                Make/O/N=(Length-1) ResidsSquared  
                ResidsSquared[j*segmlength, (j+1)*segmlength -1] = Res_Cumulative^2
                WaveStats/Q ResidsSquared
                SumSquaredResids[i] = V_Sum        
            endfor 
    endfor
End
Hello Eduard,

take a look at the documentation for the /R flag for WaveStats, I think it will do what you want.

In general I would use a 2D wave to hold data for multiple runs of the same experiments, rather than concatenating them in a single 1D wave. Your way works, of course, but I find 2D data easier conceptually and more importantly it no longer requires that you know or pass around the length of each segment (it simply becomes the number of rows or columns in the 2D wave).

WaveStats will treat your 2D (or higher) waves as 1D waves, which means that with such data you will typically want to use a loop and pass WaveStats single rows or columns at a time. For such cases you can take a look at the ImageTransform getCol or getRow functionality, or MatrixOP's col() or row() functions.

August 6, 2012 at 02:34 am - Permalink

Some other things that came to mind as I quickly glanced over your code:
CurveFit/Q Line Cumulative[j*segmlength, (j+1)*segmlength -1] /X=Xwave /D /R /A=0
Wave Res_Cumulative


You're hardcoding the knowledge that the wave that is fitted is called 'Cumulative', and that the residual will therefore be called 'Res_Cumulative'. However, if you were to pass in a wave called 'whatever' then the residual would be called 'Res_Whatever', and the wave statement would fail. I would recommend passing '/R=Res_Cumulative' (or whatever name) instead of simply '/R' to the CurveFit operation, so that the residual wave will always be called 'Res_Cumulative'. This makes your code more robust.

ResidsSquared[j*segmlength, (j+1)*segmlength -1] = Res_Cumulative^2


This won't work properly due to the way in which Igor does wave assignments. In the above statement, Igor will try to fill e.g. ResidsSquared[j*segmlength] with Res_Cumulative[j*segmlength]^2. This point won't exist for any but the starting iteration, and therefore you'll either get a constant value (the last point in Res_Cumulative) or a runtime error. Instead you should change this to:

ResidsSquared[j*segmlength, (j+1)*segmlength -1] = Res_Cumulative[p - j * segmlength]^2


This stuff is easy to overlook while programming, so I tend to always use the magic variables 'p, q, r, t' explicitly when writing wave assignments.

August 6, 2012 at 03:10 am - Permalink

Many thanks 741! Your suggestions are fine. I used the MatrixOp and changed the R flag in the Curve Fit function. Moreover, I changed the code so that the squared residuals form the Curve Fit are being stored column by column to subsequent columns in the matrix. Think this works.

Function LocalForward(Cumulative, Length)   // Forward function to cacluate detrended fluctuation
    Wave Cumulative // Read series of integrated share price returns
    Variable Length // Number of points in share price series
   
    //  The fluctuation is determined for eighteen different scales. The scales relate to an
    //  integrated series of share price returns with 4,380 observations
    //  The wave NumbSegments shows the number of segments in which the series with 4,380
    //  observations can be divided. The total series can be subdivided in 480non-overlapping
    //  segments with a length (scale) of 8, 240 non-overlapping segments with a length of 16, etc.  
   
    Make/O/N=18 Scale={8,16,12,32,40,48,64,80,96,120,128,160,192,240,256,320,384,480}
    Make/O/N=18 NumbSegments={480,240,160,120,96,80,60,48,40,32,30,24,20,16,15,12,10,8}
   
    Make/O/N=(Length-1,18) MatrixFWSquaredResids    // Create a matrix with squared residual values
   
    Variable i ,j   // Create local variables i and j
   
    for (i=0; i<18; i +=1)  // Commands between "for" and "endfor" are repeated
   
        Variable segmlength = Scale[i]  // Segment length varies from 8 when i=0 to 480, when i=17
        Variable nsegments = NumbSegments[i]    // Number of segments varies from 480 when i=0  
                                        // and segment length is 8 to 8 when i=17 and
                                        // segment length = 480
                               
   
        Make/O/N=(Length-1) Xwave = mod(p, segmlength) +1   // Create Xwave series; X runs from
                                                    // 1 up to the final point in a scale,
                                                    // for a scale / segment length of 8,
                                                    // the X wave runs from 1 up to 8
                                                               
        Make/O/N=(Length-1) FWResidsSquared
   
            for (j=0; j<nsegments; j += 1)  // Commands between "for" and "endfor" are repeated
                                    // CurveFit function is repeated for each of the
                                    // segments when i=0 CurveFit is repeated 480 times in
                                    // accordance with the number of segments; when i=17,
                                    // Curve Fit is repeated 8 times, for each of the segments
                                    // (scales) with a length of 480 observations
                                                                           
            //  Determine local trend per segment (scale) as well as residuals against local trend.
            //  Square the residuals
            //  Create new wave ResidsSquared. Store squared residuals to this series.
            //  Next determine sum of each of the ResidsSquared eighteen times (for i=1 to i=17) and
            //  store the sums to the wave SumSquaredResids
       
            CurveFit/Q Line Cumulative[j*segmlength, (j+1)*segmlength -1] /X=Xwave /R=Res_Cumulative
                Wave Res_Cumulative
                FWResidsSquared[j*segmlength, (j+1)* segmlength -1] = Res_Cumulative^2
               
            endfor 
           
            MatrixFWSquaredResids[][i] = FWResidsSquared[p]
            Make/O/N=(18,1) FWSummedSqValues
            MatrixOp/O FWSummedSqValues = SumCols(MatrixFWSquaredResids)
           
    endfor
End


I do, however, have a further question related to the use of matrices. The above function performs calculations progressing from the oldest to the most recent observation of the Cumulative wave. I have written a similar function for which the same calculations are done in the opposite direction: from the most recent observation of the Cumulative wave down to the oldest observation. I wish to sum the outcomes of these two - forward and backward - functions, but I run into problems in trying to do so. In fact I try to change two 2D waves ('FWSummedSqValues' and 'BWSummedSqValues') with one row and 18 columns to two 1D waves with one column and 18 rows so that I can combine them in a calculation with the wave 'NumbSegments', a 1D wave with one column and 18 rows. Any help is highly welcome. Below are the attempts from my side.

Function FluctMagn(FWSummedSqValues, BWSummedSqValues, NumbSegments, Length)
                                                             
    Wave  FWSummedSqValues, BWSummedSqValues, NumbSegments  // Read waves
    Variable Length // Number of points in share price series
   
    Make/O/N=18 ForwardFluct
    Make/O/N=18 BackwardFluct
    ForwardFluct = FWSummedSqValues[q]
    BackwardFluct = BWSummedSqValues[q]
   
    Make/O/N=18 Fluctuation = Sqrt(ForwardFluct+BackwardFluct)/(NumbSegments*2)
                                     // Determine standard deviation of fluctuation for the 18 scales      
End

August 7, 2012 at 01:24 am - Permalink

It appears that the indexing into the source wave was faulty. Try the modified code below.

Function FluctMagn(FWSummedSqValues, BWSummedSqValues, NumbSegments, Length)
                                                             
    Wave  FWSummedSqValues, BWSummedSqValues, NumbSegments  // Read waves
    Variable Length // Number of points in share price series
   
    Make/O/N=18 ForwardFluct
    Make/O/N=18 BackwardFluct
    ForwardFluct = FWSummedSqValues[0][p]  //get values from first row, use p from left side of assignment to index into source wave
    BackwardFluct = BWSummedSqValues[0][p] //get values from first row, use p from left side of assignment to index into source wave
   
    Make/O/N=18 Fluctuation = Sqrt(ForwardFluct+BackwardFluct)/(NumbSegments*2)
                                     // Determine standard deviation of fluctuation for the 18 scales      
End

August 7, 2012 at 05:40 am - Permalink

Thanks for your reply. I implemented your suggestion, but I still experience a problem. No error is shown at the compiling stage. However, after calling the top function, which also includes a number of other functions, a problem turns up in the Function FluctMagn. The Debuggers shows an error at the line Make/O/N=18 Forwardfluct. The Debugger describes the problem as "WAVE error: "WAVE reference to "FWSummedSqValues" failed". As a further clarification of the problem. after creating a new table for the waves Forwardfluct and Backwardfluct it appears that seventeen rows show a zero, while the first row is empty; it has no value in it, neither a zero nor another value.

Hope the above gives any idea of what might be the problem. Thanks in advance on any suggestion to solve this problem.

Eduard

August 7, 2012 at 06:28 am - Permalink

Eduard Kas wrote:
... The Debuggers shows an error at the line Make/O/N=18 Forwardfluct. The Debugger describes the problem as "WAVE error: "WAVE reference to "FWSummedSqValues" failed". As a further clarification of the problem. after creating a new table for the waves Forwardfluct and Backwardfluct it appears that seventeen rows show a zero, while the first row is empty; it has no value in it, neither a zero nor another value.


Based on ForwardFluct being (18,1) in dimension, I think the indexing is to be as below

Function FluctMagn(FWSummedSqValues, BWSummedSqValues, NumbSegments, Length)
    Wave  FWSummedSqValues, BWSummedSqValues, NumbSegments  // Read waves
    Variable Length // Number of points in share price series
 
    Make/O/N=18 ForwardFluct, BackwardFluct, Fluctuation
    ForwardFluct = FWSummedSqValues[p][0]  //get values from first row, use p from left side of assignment to index into source wave
    BackwardFluct = BWSummedSqValues[p][0] //get values from first row, use p from left side of assignment to index into source wave
    Fluctuation = Sqrt(ForwardFluct+BackwardFluct)/(NumbSegments*2)  // Determine standard deviation of fluctuation for the 18 scales
    return 0
End


If I might also suggest a brief modification to your other function as follows:

Function MakeScaleWaves()

    SetDataFolder root
    NewDataFolder/O/S Globals
 
    Make/O/N=18 Scale={8,16,12,32,40,48,64,80,96,120,128,160,192,240,256,320,384,480}
    Make/O/N=18 NumbSegments={480,240,160,120,96,80,60,48,40,32,30,24,20,16,15,12,10,8}
 
    SetDataFolder root
   
    return 0
end

Function LocalForward(Cumulative, Length)   // Forward function to cacluate detrended fluctuation
    Wave Cumulative // Read series of integrated share price returns
    Variable Length // Number of points in share price series
 
    //  The fluctuation is determined for eighteen different scales. The scales relate to an
    //  integrated series of share price returns with 4,380 observations
    //  The wave NumbSegments shows the number of segments in which the series with 4,380
    //  observations can be divided. The total series can be subdivided in 480non-overlapping
    //  segments with a length (scale) of 8, 240 non-overlapping segments with a length of 16, etc.  
 
    DFREF gl=root:Globals
    if (DataFolderRefStatus(gl)==0)
        MakeScaleWaves()
    endif

    wave Scale = gl:Scale
    wave NumbSegments = gl:NumbSegments
        ...


--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAHuntsville

August 7, 2012 at 07:21 am - Permalink

jjweimer wrote:
Eduard Kas wrote:
... The Debuggers shows an error at the line Make/O/N=18 Forwardfluct. The Debugger describes the problem as "WAVE error: "WAVE reference to "FWSummedSqValues" failed". As a further clarification of the problem. after creating a new table for the waves Forwardfluct and Backwardfluct it appears that seventeen rows show a zero, while the first row is empty; it has no value in it, neither a zero nor another value.


Based on ForwardFluct being (18,1) in dimension, I think the indexing is to be as below



ForwardFluct is (18,1) or (18,0) in Igor-speak, but, I think, FWSummedSqValues and BWSummedSqValues are (1,18), thus

ForwardFluct = FWSummedSqValues[0][p]  //reference 0th row, cols based on p

Unless I'm being brain dead... always possible.

About the wave reference failure, are you certain that you are passing a valid reference and that the wave exists at run time? I tried to duplicate the problem and only generated an error about attempting to operate on a null or missing wave, not an invalid wave reference.***

Hope this helps.

***Note: If wave checking is turned on in the debugger, the wave reference failed error will be given.

August 7, 2012 at 09:02 am - Permalink

jtigor wrote:
jjweimer wrote:
Eduard Kas wrote:
... The Debuggers shows an error at the line Make/O/N=18 Forwardfluct. The Debugger describes the problem as "WAVE error: "WAVE reference to "FWSummedSqValues" failed". As a further clarification of the problem. after creating a new table for the waves Forwardfluct and Backwardfluct it appears that seventeen rows show a zero, while the first row is empty; it has no value in it, neither a zero nor another value.


Based on ForwardFluct being (18,1) in dimension, I think the indexing is to be as below



ForwardFluct is (18,1) or (18,0) in Igor-speak, but, I think, FWSummedSqValues and BWSummedSqValues are (1,18), thus


Oops. I meant to say, the starting function shows

Make/O/N=(18,1) FWSummedSqValues

So it seems to me, the indexing has to be

ForwardFluct = FWSummedSqValues[p][0]

Unless I have also missed something obvious too.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAHuntsville

August 7, 2012 at 08:35 am - Permalink

JJ,

You are correct. The code states...
Make/O/N=(18,1) FWSummedSqValues


I was paying attention to the description...

Eduard Kas wrote:
In fact I try to change two 2D waves ('FWSummedSqValues' and 'BWSummedSqValues') with one row and 18 columns to two 1D waves with one column and 18 rows so that I can combine them in a calculation with the wave 'NumbSegments', a 1D wave with one column and 18 rows.

August 7, 2012 at 08:52 am - Permalink

Many thanks for your contributions. I implemented the Function FluctMagn in exactly the way proposed by J.J. Weimer. Yet, there is still the Debugger error "WAVE error: "WAVE reference to "FWSummedSqValue" failed. What also has remained the same is that the first row out of 18 of the 'BackwardFluct' and 'ForwardFluct' waves are empty. The problem may relate to earlier code at the end of the Function LocalForward and similar lines of code at the end of Function LocalBackward. Here are the lines I refer to:

MatrixFWSquaredResids[][i] = FWResidsSquared[p]
    Make/O/N=(18,1) FWSummedSqValues
    MatrixOp/O FWSummedSqValues = SumCols(MatrixFWSquaredResids)


MatrixBWSquaredResids[][i] = BWResidsSquared[p]
    Make/O/N=(18,1) BWSummedSqValues
    MatrixOp/O BWSummedSqValues = SumCols(MatrixBWSquaredResids)


In the second lines I attempted to define 'FWSummedSqValues' and 'BWSummedSqValues' as waves with 18 rows and 1 column. However, the waves that appeared are of a 1 row and 18 columns nature.

Eduard

August 7, 2012 at 12:59 pm - Permalink

Eduard Kas wrote:
... Here are the lines I refer to:

MatrixFWSquaredResids[][i] = FWResidsSquared[p]
    Make/O/N=(18,1) FWSummedSqValues
    MatrixOp/O FWSummedSqValues = SumCols(MatrixFWSquaredResids)


MatrixBWSquaredResids[][i] = BWResidsSquared[p]
    Make/O/N=(18,1) BWSummedSqValues
    MatrixOp/O BWSummedSqValues = SumCols(MatrixBWSquaredResids)


In the second lines I attempted to define 'FWSummedSqValues' and 'BWSummedSqValues' as waves with 18 rows and 1 column. However, the waves that appeared are of a 1 row and 18 columns nature.

Eduard


From the Help ...

sumCols(w)  Returns an 1xm matrix containing the sums of the m columns in the nxm input wave w :
   
sumRows(w)  Returns an nx1 matrix containing the sums of the n rows in the nxm input wave w :


So, perhaps you want sumRows rather than sumCols. I have to admit though that someone with a better understanding of MatrixOp could probably help you better than me.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAHuntsville

August 7, 2012 at 01:48 pm - Permalink

Thanks again. In the meantime I solved the problem of the Waves FWSummedSqValues and BWSummedSqValues showing rows instead of the desired columns. The most simple solution appeared to just change the dimension of the waves, as follows:

MatrixFWSquaredResids[][i] = FWResidsSquared[p]
Make/O/N=(18,1) FWSummedSqValues
MatrixOp/O FWSummedSqValues = SumCols(MatrixFWSquaredResids)
Redimension/N=18 FWSummedSqValues


However, to my surprise there remained a problem in the next function when I tried to add the outcomes of two functions - FWSummedSqValues and BWSummedSqValues - in a third function, which I simplified along the following lines:

Function FluctMagn(FWSummedSqValues, BWSummedSqValues , NumbSegments, Length)
Wave  FWSummedSqValues, BWSummedSqValues , NumbSegments // Read waves
Variable Length // Number of points in share price series
   
Make/N=18 SumSqValues = FWSummedSqValues+BWSummedSqValues
Make/N=18 Fluctuation =  Sqrt((SumSqValues)/(NumbSegments*2)) // Determine standard deviation of fluctuation for the 18 scales  
End


Debugger continues to show: WAVE error: "WAVE reference to "FWSummedSqValues" failed." However both waves do have vales. This time at the line Make/N=18 SumSqValues = FWSummedSqValues+BWSummedSqValues.

Other information that might be relevant: In the Debugger the Stack FluctMagn is selected and six local and global variables are shown. Four of these - BWSummedSqValues, Fluctuation, FWSummedSqValues and SumSqValues - are shown with a null denotation.

Hope this gives an idea of what might cause the problem.

August 8, 2012 at 01:21 am - Permalink

Eduard Kas wrote:
Debugger continues to show: WAVE error: "WAVE reference to "FWSummedSqValues" failed." However both waves do have vales. This time at the line Make/N=18 SumSqValues = FWSummedSqValues+BWSummedSqValues.

Other information that might be relevant: In the Debugger the Stack FluctMagn is selected and six local and global variables are shown. Four of these - BWSummedSqValues, Fluctuation, FWSummedSqValues and SumSqValues - are shown with a null denotation.

Hope this gives an idea of what might cause the problem.


It sounds like the reference you pass to FluctMagn is not valid, given both the error message and the fact that the debugger shows them as null. Do you change directories in your code? If you get the wave reference using

Wave wave1 = wave2

this will fail if the wave doesn't exist in the current directory. You could try

Wave wave1 = pathtowave:wave2

That is give the full path to the wave you want to reference.

You could also try setting a break point somewhere before the call to FluctMagn and step through your code line by line. Often this helps make clear the source of problems. Look to see if your wave references are valid before the call is made.

August 8, 2012 at 06:04 am - Permalink

I think the best way to solve this would be to upload an actual experiment that contains the necessary data and code, as well as instructions on how to trigger the error. That way we can reproduce this on our systems and figure out what it happening, because it's rather hard to diagnose from the information provided here.

August 8, 2012 at 07:46 am - Permalink

I really appreciate your contributions. As far as I know I made no changes to the directory. Attached you find the experiment. Showing the experiment might indeed be far more insightful. For this I copied the procedure to a new Igor file and I simulated one wave with 3841 observations. All the other waves are generated by the procedure.

Apart form the problem with the FWSummedSqValues and BWSummedSqValues waves the residuals of the two CurveFits in the LocalForward and LocalBackward behave in a strange way. Although the Y-values of the Curve Fits (Cumulative and New Cumulative, respectively) are different, the FWSummedSqValues and BWSummedSqValues waves are identical. (This should not be the case, as NewCumulative is Cumulative in reverse order.)

Instead of the R/ flags in the CurveFits I earlier used the R/Res_Cumulative and R_Res_NewCumulative denotations. However, when I use these denotations, Debugger shows an error at the first curve fit.

Hope that you can discover what went wrong in the procedure.


HurstEstimation.pxp

August 8, 2012 at 12:56 pm - Permalink

This is very helpful to me! Many thanks 741. I will look a bit further at it later today, but a quick test already gives me the idea that it works. I simulated a random noise pattern, which should deliver an end result (a Hurst estimate) of 0.5 and the result came in at 0.50018, so exactly 0.5 in rounded terms.

A quick look at the changes you made learns me that I missed on some required initialisations, while the loops were placed at the wrong place.

Eduard

August 9, 2012 at 01:52 am - Permalink

Eduard Kas wrote:
This is very helpful to me! ...


Of course, now we all want to know how we can apply your procedures to enhance our own stock option holdings. :-)

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAHuntsville

August 9, 2012 at 08:02 am - Permalink