Matrix Profile and Weakly Labelled Data - Update 1

This is the first post in a short series detailing my recent work following on from my previous post. This post will be about some problems I have had and how I partially solved them.

The main problem was simply the speed at which the code (available from the companion website) seems to run. The first stage Matrix Profile code runs in a few seconds, the second, individual evaluation stage in no more than a few minutes, but the third stage, greedy search, which uses Golden Section Search over the pattern candidates, can take many, many hours. My approach to this was simply to optimise the code to the best of my ability. My optimisations, all in the compute_f_meas.m function, are shown in the following code boxes. This while loop

i = 1;
while true
    
 if i >= length(anno_st)
    break;
 endif
       
first_part = anno_st(1:i);
second_part = anno_st(i+1:end);
bad_st = abs(second_part - anno_st(i)) < sub_len;
second_part = second_part(~bad_st);
anno_st = [first_part; second_part;];
i = i + 1;
      
endwhile

is replaced by this .oct compiled version of the same while loop

#include 
#include 

DEFUN_DLD ( stds_f_meas_while_loop_replace, args, nargout,
"-*- texinfo -*-\n\
@deftypefn {Function File} {} stds_f_meas_while_loop_replace (@var{input_vector,sublen})\n\
This function takes an input vector and a scalar sublen\n\
length. The function sets to zero those elements in the\n\
input vector that are closer to the preceeding value than\n\
sublen. This function replaces a time consuming .m while loop\n\
in the stds compute_f_meas.m function.\n\
@end deftypefn" )

{
octave_value_list retval_list ;
int nargin = args.length () ;

// check the input arguments
if ( nargin != 2 ) // there must be a vector and a scalar sublen
   {
   error ("Invalid arguments. Inputs are a column vector and a scalar value sublen.") ;
   return retval_list ;
   }

if ( args(0).length () < 2 )
   {
   error ("Invalid 1st argument length. Input is a column vector of length > 1.") ;
   return retval_list ;
   }
   
if ( args(1).length () > 1 )
   {
   error ("Invalid 2nd argument length. Input is a scalar value for sublen.") ;
   return retval_list ;
   }
// end of input checking  
  
ColumnVector input = args(0).column_vector_value () ;
double sublen = args(1).double_value () ;
double last_iter ;

// initialise last_iter value
last_iter = input( 0 ) ;
     
for ( octave_idx_type ii ( 1 ) ; ii < args(0).length () ; ii++ )
    {
    
      if ( input( ii ) - last_iter >= sublen )
      {
        last_iter = input( ii ) ;
      }
      else
      {
        input( ii ) = 0.0 ;
      }
      
    } // end for loop
   
retval_list( 0 ) = input ;

return retval_list ;

} // end of function

and called thus

anno_st = stds_f_meas_while_loop_replace( anno_st , sub_len ) ;
anno_st( anno_st == 0 ) = [] ;

This for loop

is_tp = false(length(anno_st), 1);
for i = 1:length(anno_st)
    if anno_ed(i) > length(label)
        anno_ed(i) = length(label);
    end
    if sum(label(anno_st(i):anno_ed(i))) > 0.8*sub_len
        is_tp(i) = true;
    end
end
tp_pre = sum(is_tp);

is replaced by use of cellslices.m and cellfun.m thus

label_length = length( label ) ;
anno_ed( anno_ed > label_length ) = label_length ;
cell_slices = cellslices( label , anno_st , anno_ed ) ;
cell_sums = cellfun( @sum , cell_slices ) ;
tp_pre = sum( cell_sums > 0.8 * sub_len ) ;

and a further for loop

is_tp = false(length(pos_st), 1);
for i = 1:length(pos_st)
    if sum(anno(pos_st(i):pos_ed(i))) > 0.8*sub_len
        is_tp(i) = true;
    end
end
tp_rec = sum(is_tp);

is replaced by

cell_slices = cellslices( anno , pos_st , pos_ed ) ;
cell_sums = cellfun( @sum , cell_slices ) ;
tp_rec = sum( cell_sums > 0.8 * sub_len ) ;

Although the above measurably improves running times, overall the code of the third stage is still sluggish. I have found that the best way to deal with this, on the advice of the original paper's author, is to limit the number of patterns to search for, the "pat_max" variable, to the minimum possible to achieve a satisfactory result. What I mean by this is that if  pat_max = 5 and the result returned also has 5 identified patterns, incrementally increase pat_max until such time that the number of identified patterns is less than pat_max. This does, by necessity, mean running the whole routine a few times, but it is still quicker this way than drastically over estimating pat_max, i.e. choosing a value of say 50 to finally identify maybe only 5/6 patterns.

More in due course.

"Matrix profile: Using Weakly Labeled Time Series to Predict Outcomes" Paper

Back in May of this year I posted about how I had intended to use Matrix Profile (MP) to somehow cluster the "initial balance" of Market Profile charts with a view to getting a heads up on immediately following price action. Since then, my thinking has evolved due to my learning about the paper "Matrix profile: Using Weakly Labeled Time Series to Predict Outcomes" and its companion website. This very much seems to accomplish the same end I had envisaged with my clustering of initial balances, so I am going to try and use this approach instead.

As a preliminary, I have decided to "weakly label" my time series data using the simple code loop shown below.

for ii = 1 : numel( ix )
  
y_values = train_data( ix( ii ) + 1 : ix( ii ) + 19 , 1 ) ;
london_session_ret = y_values( end ) - y_values( 1 ) ;

[ max_y , max_ix ] = max( y_values ) ;
max_long_ex = max_y - y_values( 1 ) ;

[ min_y , min_ix ] = min( y_values ) ;
max_short_ex = min_y - y_values( 1 ) ;

if ( london_session_ret > 0 && ( max_long_ex / ( -1 * max_short_ex ) ) >= 3 && max_ix > min_ix )
 labels( ix( ii ) - 11 : ix( ii ) , 1 ) = 1 ; 
elseif ( london_session_ret < 0 && ( max_short_ex / max_long_ex ) <= -3 && max_ix < min_ix )
  labels( ix( ii ) - 11 : ix( ii ) , 1 ) = -1 ;
endif
 
endfor

What this essentially does (for the long side) is ensure that price is higher at the end of y_values than at the beginning and there is a reward/risk opportunity of at least 3:1 for at least 1 trade during the period covered by the time range of y_values (either the London a.m. session or the combined New York a.m./London p.m. session) following a 7a.m. to 8.50a.m. (local time) formation of an opening Market profile/initial balance and the maximum adverse excursion occurs before the maximum favourable excursion. A typical chart on the long side looks like this.

This would have the "weak" label for a long trade, and the label would be applied to the Market Profile data that immediately precedes this price action. On the other side, a short labelled chart typically looks like this.

As can be seen, trading "against the label" offers few opportunities for profitable entries/exits. My hope is that a "dictionary" of long/short biased Market Profile patterns can be discovered using the ideas/code in the links above. For completeness, the following chart is typical of price action which does not meet the looped code bias for either long or short.

It is easy to envisage trading this type of price action by fading moves that go outside the "value area" of a Market Profile chart.

More in due course.