Matrix Profile and Weakly Labelled Data - Update 1

This is the first post in a short series detailing my recent work following on from my previous post. This post will be about some problems I have had and how I partially solved them.

The main problem was simply the speed at which the code (available from the companion website) seems to run. The first stage Matrix Profile code runs in a few seconds, the second, individual evaluation stage in no more than a few minutes, but the third stage, greedy search, which uses Golden Section Search over the pattern candidates, can take many, many hours. My approach to this was simply to optimise the code to the best of my ability. My optimisations, all in the compute_f_meas.m function, are shown in the following code boxes. This while loop

i = 1;
while true
    
 if i >= length(anno_st)
    break;
 endif
       
first_part = anno_st(1:i);
second_part = anno_st(i+1:end);
bad_st = abs(second_part - anno_st(i)) < sub_len;
second_part = second_part(~bad_st);
anno_st = [first_part; second_part;];
i = i + 1;
      
endwhile

is replaced by this .oct compiled version of the same while loop

#include 
#include 

DEFUN_DLD ( stds_f_meas_while_loop_replace, args, nargout,
"-*- texinfo -*-\n\
@deftypefn {Function File} {} stds_f_meas_while_loop_replace (@var{input_vector,sublen})\n\
This function takes an input vector and a scalar sublen\n\
length. The function sets to zero those elements in the\n\
input vector that are closer to the preceeding value than\n\
sublen. This function replaces a time consuming .m while loop\n\
in the stds compute_f_meas.m function.\n\
@end deftypefn" )

{
octave_value_list retval_list ;
int nargin = args.length () ;

// check the input arguments
if ( nargin != 2 ) // there must be a vector and a scalar sublen
   {
   error ("Invalid arguments. Inputs are a column vector and a scalar value sublen.") ;
   return retval_list ;
   }

if ( args(0).length () < 2 )
   {
   error ("Invalid 1st argument length. Input is a column vector of length > 1.") ;
   return retval_list ;
   }
   
if ( args(1).length () > 1 )
   {
   error ("Invalid 2nd argument length. Input is a scalar value for sublen.") ;
   return retval_list ;
   }
// end of input checking  
  
ColumnVector input = args(0).column_vector_value () ;
double sublen = args(1).double_value () ;
double last_iter ;

// initialise last_iter value
last_iter = input( 0 ) ;
     
for ( octave_idx_type ii ( 1 ) ; ii < args(0).length () ; ii++ )
    {
    
      if ( input( ii ) - last_iter >= sublen )
      {
        last_iter = input( ii ) ;
      }
      else
      {
        input( ii ) = 0.0 ;
      }
      
    } // end for loop
   
retval_list( 0 ) = input ;

return retval_list ;

} // end of function

and called thus

anno_st = stds_f_meas_while_loop_replace( anno_st , sub_len ) ;
anno_st( anno_st == 0 ) = [] ;

This for loop

is_tp = false(length(anno_st), 1);
for i = 1:length(anno_st)
    if anno_ed(i) > length(label)
        anno_ed(i) = length(label);
    end
    if sum(label(anno_st(i):anno_ed(i))) > 0.8*sub_len
        is_tp(i) = true;
    end
end
tp_pre = sum(is_tp);

is replaced by use of cellslices.m and cellfun.m thus

label_length = length( label ) ;
anno_ed( anno_ed > label_length ) = label_length ;
cell_slices = cellslices( label , anno_st , anno_ed ) ;
cell_sums = cellfun( @sum , cell_slices ) ;
tp_pre = sum( cell_sums > 0.8 * sub_len ) ;

and a further for loop

is_tp = false(length(pos_st), 1);
for i = 1:length(pos_st)
    if sum(anno(pos_st(i):pos_ed(i))) > 0.8*sub_len
        is_tp(i) = true;
    end
end
tp_rec = sum(is_tp);

is replaced by

cell_slices = cellslices( anno , pos_st , pos_ed ) ;
cell_sums = cellfun( @sum , cell_slices ) ;
tp_rec = sum( cell_sums > 0.8 * sub_len ) ;

Although the above measurably improves running times, overall the code of the third stage is still sluggish. I have found that the best way to deal with this, on the advice of the original paper's author, is to limit the number of patterns to search for, the "pat_max" variable, to the minimum possible to achieve a satisfactory result. What I mean by this is that if  pat_max = 5 and the result returned also has 5 identified patterns, incrementally increase pat_max until such time that the number of identified patterns is less than pat_max. This does, by necessity, mean running the whole routine a few times, but it is still quicker this way than drastically over estimating pat_max, i.e. choosing a value of say 50 to finally identify maybe only 5/6 patterns.

More in due course.

"Matrix profile: Using Weakly Labeled Time Series to Predict Outcomes" Paper

Back in May of this year I posted about how I had intended to use Matrix Profile (MP) to somehow cluster the "initial balance" of Market Profile charts with a view to getting a heads up on immediately following price action. Since then, my thinking has evolved due to my learning about the paper "Matrix profile: Using Weakly Labeled Time Series to Predict Outcomes" and its companion website. This very much seems to accomplish the same end I had envisaged with my clustering of initial balances, so I am going to try and use this approach instead.

As a preliminary, I have decided to "weakly label" my time series data using the simple code loop shown below.

for ii = 1 : numel( ix )
  
y_values = train_data( ix( ii ) + 1 : ix( ii ) + 19 , 1 ) ;
london_session_ret = y_values( end ) - y_values( 1 ) ;

[ max_y , max_ix ] = max( y_values ) ;
max_long_ex = max_y - y_values( 1 ) ;

[ min_y , min_ix ] = min( y_values ) ;
max_short_ex = min_y - y_values( 1 ) ;

if ( london_session_ret > 0 && ( max_long_ex / ( -1 * max_short_ex ) ) >= 3 && max_ix > min_ix )
 labels( ix( ii ) - 11 : ix( ii ) , 1 ) = 1 ; 
elseif ( london_session_ret < 0 && ( max_short_ex / max_long_ex ) <= -3 && max_ix < min_ix )
  labels( ix( ii ) - 11 : ix( ii ) , 1 ) = -1 ;
endif
 
endfor

What this essentially does (for the long side) is ensure that price is higher at the end of y_values than at the beginning and there is a reward/risk opportunity of at least 3:1 for at least 1 trade during the period covered by the time range of y_values (either the London a.m. session or the combined New York a.m./London p.m. session) following a 7a.m. to 8.50a.m. (local time) formation of an opening Market profile/initial balance and the maximum adverse excursion occurs before the maximum favourable excursion. A typical chart on the long side looks like this.

This would have the "weak" label for a long trade, and the label would be applied to the Market Profile data that immediately precedes this price action. On the other side, a short labelled chart typically looks like this.

As can be seen, trading "against the label" offers few opportunities for profitable entries/exits. My hope is that a "dictionary" of long/short biased Market Profile patterns can be discovered using the ideas/code in the links above. For completeness, the following chart is typical of price action which does not meet the looped code bias for either long or short.

It is easy to envisage trading this type of price action by fading moves that go outside the "value area" of a Market Profile chart.

More in due course.

Another Iterative Improvement of my Volume/Market Profile Charts

Below is a screenshot of this new chart version, of today's (Friday's) price action at a 10 minute bar scale:

Just by looking at the chart it might not be obvious to readers what has changed, so the changes are detailed below.

The first change is in how the volume profile (the horizontal histogram on the left) is calculated. The "old" version of the chart calculates the profile by assuming the "model" that tick volume for each 10 minute bar is normally distributed across the high/low range of the bar, and then the profile histogram is the accumulation of these individual, 10 minute, normally distributed "mini profiles." A more complete description of this is given in my Market Profile Chart in Octave blog post, with code.

The new approach is more data centric rather than model based. Every 10 minutes, instead of downloading the 10 minute OHLC and tick volume, the last 10 minutes worth of 5 second OHLC and tick volume is downloaded. The whole tick volume of each 5 second period is assigned to a price level equivalent to the Typical price (rounded to the nearest pip) of said 5 second period, and the volume profile is then the accumulation of these volume ticks per price level. I think this is a much more accurate reflection of the price levels at which tick volume actually occurred compared to the old, model based charts. This second screenshot is of the old chart over the exact same price data as the first, improved version of the chart.

It can be seen that the two volume profile histograms of the respective charts differ from each other in terms of their overall shape and the number and price levels of peaks (Points of Control) and troughs (Low Volume Nodes).

The second change in the new chart is in how the background heatmap is plotted. The heatmap is a different presentation of the volume profile whereby higher volume price levels are shown by the brighter yellow colours. The old chart only displays the heatmap associated with the latest calculated volume profile histogram, which is projected back in time. This is, of course, a form of lookahead bias when plotting past prices over the latest heatmap. The new chart solves this by plotting a "rolling" version of the heatmap which reflects the volume profile that was in force at the time each 10 minute OHLC candle formed. It is easy to see how the Points of Control and Low Volume Nodes price levels ebb and flow throughout the trading day.

The third change, which naturally followed on from the downloading of 5 second data, is in the plotting of the candlesticks. Rather than having a normal, open to close candlestick body, the candlesticks show the "mini volume profiles" of the tick volume within each bar, plotted via Octave's patch function. The white candlestick wicks indicate the usual high/low range, and the open and close levels are shown by grey and black dots respectively. This is more clearly seen in the zoomed in screenshot below.

I wanted to plot these types of bars because recently I have watched some trading webcasts, which talked about "P", "b" and "D" shaped bar profiles at "areas of interest." The upshot of these webcasts is that, in general, a "P" bar is bullish, a "b" is bearish and a "D" is "in balance" when they intersect an "area of interest" such as Point of Control, Low Volume Node, support and resistance etc. This is supposed to be indicative of future price direction over the immediate short term. With this new version of chart, I shall be in a position to investigate these claims for myself.

Market Profile Low Volume Node Chart

As a diversion to my recent work with Matrix Profile I have recently completed work on a new chart type in Octave, namely a Market Profile Low Volume Node (LVN) chart, two slightly different versions of which are shown below.

This first one is derived from a TPO chart, whilst the next

is derived from a Volume profile chart.

The horizontal lines are drawn at levels which are considered to be "lows" in the underlying, but not shown, TPO/Volume profiles. The yellow lines are "stronger lows" than the green lines, and the blue lines are extensions of the previous day's "strong lows" in force at the end of that day's trading.

The point of all this, according to online guru theory, is that price is expected to be "rejected" at LVNs by either bouncing, a la support or resistance, or by price powering through the LVN level, usually on increased volume. The charts show the rolling development of the LVNs as the underlying profiles change throughout the day, hence lines can appear and disappear and change colour. As this is a new avenue of investigation for me I feel it is too soon to make a comment on these lines' efficacy, but it does seem uncanny how price very often seems to react to these levels.

More in due course.

Update on Recent Matrix Profile Work

Since my previous post, on Matrix Profile (MP), I have been doing a lot of online reading about MP and going back to various source papers and code that are available at the UCR Matrix Profile page. I have been doing this because, despite my initial enthusiasm, the R tsmp package didn't turn out to be suitable for what I wanted to do, or perhaps more correctly I couldn't hack it to get the sort of results I wanted, hence my need to go to "first principles" and code from the UCR page.

Readers may recall that my motivation was to look for time series motifs that form "initial balance (IB)" set ups of Market Profile charts. The rationale for this is that different IBs are precursors to specific market tendencies which may provide a clue or an edge in subsequent market action. A typical scenario from the literature on Market Profile might be "an Open Test Drive can often indicate one of the day's extremes." If this is actually true, one could go long/short with a high confidence stop at the identified extreme. Below is a screenshot of some typical IB profiles:

where each letter typically represents a 30 minute period of market action. The problem is that Market Profile charts, to me at least, are inherently visual and therefore do not easily lend themselves to an algorithmic treatment, which makes it difficult to back test in a robust fashion. This is why I have been trying to use MP.

The first challenge I faced was how to preprocess price action data such as OHLC and volume such that I could use MP. In the end I resorted to using the mid-price, the high-low range and (tick) volume as proxies for market direction, market volatility and market participation. Because IBs occur over market opens, I felt it was important to use the volatility and participation proxies as these are important markers for the sentiment of subsequent price action. This choice necessitated using a multivariate form of MP, and I used the basic MP STAMP code that is available at Matrix Profile VI: Meaningful Multidimensional Motif Discovery, with some slight tweaks for my use case.

Having the above tools in hand, what should they be used for? I decided that Cluster analysis is what is needed, i.e. cluster using the motifs that MP could discover. For this purpose, I used the approach outlined in section 3.9 of the paper "The Swiss Army Knife of Time Series Data Mining." The reasoning behind this choice is that if, for example, an "Open Test Drive IB" is a real thing, it should occur frequently enough that time series sub-sequences of it can be clustered or associated with an "Open Test Drive IB" motif. If all such prototype motifs can be identified and all IBs can be assigned to one of them, subsequent price action can be investigated to check the anecdotal claims, such as quoted above.

My Octave code implementation of the linked Swiss Army Knife routine is shown in the code box below.

data = dlmread( '/path/to/mv_data' ) ;
skip_loc = dlmread( '/path/to/skip_loc' ) ;
skip_loc_copy = find( skip_loc ) ; skip_loc_copy2 = skip_loc_copy ; skip_loc_copy3 = skip_loc_copy ;
sub_len = 9 ;
data_len = size( data , 1 ) ;
data_to_use = [ (data(:,2).+data(:,3))./2 , data(:,2).-data(:,3) , data(:,5) ] ;

must_dim = [] ;
exc_dim = [] ;
[ pro_mul , pro_idx , data_freq , data_mu , data_sig ] = multivariate_stamp( data_to_use, sub_len, must_dim, exc_dim, skip_loc ) ;
original_single_MP = pro_mul( : , 1 ) ; ## just mid price
original_single_MP2 = original_single_MP .+ pro_mul( : , 2 ) ; ## mid price and hi-lo range
original_single_MP3 = original_single_MP2 .+ pro_mul( : , 3 ) ; ## mid price, hi-lo range and volume

## Swiss Army Knife Clustering
RelMP = original_single_MP ; RelMP2 = original_single_MP2 ; RelMP3 = original_single_MP3 ;
DissMP = inf( length( RelMP ) , 1 ) ; DissMP2 = DissMP ; DissMP3 = DissMP ; 
minValStore = [] ; minIdxStore = [] ; minValStore2 = [] ; minIdxStore2 = [] ; minValStore3 = [] ; minIdxStore3 = [] ;
## set up a recording matrix 
all_dist_pro = zeros( size( RelMP , 1 ) , size( data_to_use , 2 ) ) ;

for ii = 1 : 500
## reset recording matrix for this ii loop  
all_dist_pro( : , : ) = 0 ;

## just mid price
[ minVal , minIdx ] = min( RelMP ) ;
minValStore = [ minValStore ; minVal ] ; minIdxStore = [ minIdxStore ; minIdx ] ;
DissmissRange = data_to_use( minIdx : minIdx + sub_len - 1 , : ) ;
[ dist_pro , ~ ] = multivariate_mass (data_freq(:,1), DissmissRange(:,1), data_len, sub_len, data_mu(:,1), data_sig(:,1), data_mu(minIdx,1), data_sig(minIdx,1) ) ;
all_dist_pro( : , 1 ) = real( dist_pro ) ;
JMP = all_dist_pro( : , 1 ) ;
DissMP = min( DissMP , JMP ) ; ## dismiss all motifs discovered so far
RelMP = original_single_MP ./ DissMP ;
skip_loc_copy = unique( [ skip_loc_copy ; ( minIdx : 1 : minIdx + sub_len - 1 )' ] ) ;
RelMP( skip_loc_copy ) = 1 ;

## mid price and hi-lo range
[ minVal , minIdx ] = min( RelMP2 ) ;
minValStore2 = [ minValStore2 ; minVal ] ; minIdxStore2 = [ minIdxStore2 ; minIdx ] ;
DissmissRange = data_to_use( minIdx : minIdx + sub_len - 1 , : ) ;
[ dist_pro , ~ ] = multivariate_mass (data_freq(:,1), DissmissRange(:,1), data_len, sub_len, data_mu(:,1), data_sig(:,1), data_mu(minIdx,1), data_sig(minIdx,1) ) ;
all_dist_pro( : , 2 ) = real( dist_pro ) ;
[ dist_pro , ~ ] = multivariate_mass (data_freq(:,2), DissmissRange(:,2), data_len, sub_len, data_mu(:,2), data_sig(:,2), data_mu(minIdx,2), data_sig(minIdx,2) ) ;
all_dist_pro( : , 2 ) = all_dist_pro( : , 2 ) .+ real( dist_pro ) ;
JMP2 = all_dist_pro( : , 2 ) ;
DissMP2 = min( DissMP2 , JMP2 ) ; ## dismiss all motifs discovered so far
RelMP2 = original_single_MP2 ./ DissMP2 ;
skip_loc_copy2 = unique( [ skip_loc_copy2 ; ( minIdx : 1 : minIdx + sub_len - 1 )' ] ) ;
RelMP2( skip_loc_copy2 ) = 1 ;

## mid price, hi-lo range and volume
[ minVal , minIdx ] = min( RelMP3 ) ;
minValStore3 = [ minValStore3 ; minVal ] ; minIdxStore3 = [ minIdxStore3 ; minIdx ] ;
DissmissRange = data_to_use( minIdx : minIdx + sub_len - 1 , : ) ;
[ dist_pro , ~ ] = multivariate_mass (data_freq(:,1), DissmissRange(:,1), data_len, sub_len, data_mu(:,1), data_sig(:,1), data_mu(minIdx,1), data_sig(minIdx,1) ) ;
all_dist_pro( : , 3 ) = real( dist_pro ) ;
[ dist_pro , ~ ] = multivariate_mass (data_freq(:,2), DissmissRange(:,2), data_len, sub_len, data_mu(:,2), data_sig(:,2), data_mu(minIdx,2), data_sig(minIdx,2) ) ;
all_dist_pro( : , 3 ) = all_dist_pro( : , 3 ) .+ real( dist_pro ) ;
[ dist_pro , ~ ] = multivariate_mass (data_freq(:,3), DissmissRange(:,3), data_len, sub_len, data_mu(:,3), data_sig(:,3), data_mu(minIdx,3), data_sig(minIdx,3) ) ;
all_dist_pro( : , 3 ) = all_dist_pro( : , 3 ) .+ real( dist_pro ) ;
JMP3 = all_dist_pro( : , 3 ) ;
DissMP3 = min( DissMP3 , JMP3 ) ; ## dismiss all motifs discovered so far
RelMP3 = original_single_MP3 ./ DissMP3 ;
skip_loc_copy3 = unique( [ skip_loc_copy3 ; ( minIdx : 1 : minIdx + sub_len - 1 )' ] ) ;
RelMP3( skip_loc_copy3 ) = 1 ;

endfor ## end ii loop

There are a few things to note about this code:

• the use of a skip_loc vector 
• a sub_len value of 9
  
• 3 different calculations for DissMP and RelMP vectors

i) The skip_loc vector is a vector of time series indices (Idx) for which the MP and possible cluster motifs should not be calculated to avoid identifying motifs from data sequences that do not occur in the underlying data due to the way I concatenated it during pre-processing, i.e. 7am to 9am, 7am to 9am, ... etc.

ii) sub_len value of 9 means 9 x 10 minute OHLC bars, to match the 30 minute A, B and C of the above IB screenshot.

iii)  3 different calculations because different combinations of the underlying data are used. 

This last part probably needs more explanation. A multivariate RelMP is created by adding together individual dist_pros (distance profiles), and the cluster motif identification is achieved by finding minimums in the RelMP; however, a minimum in a multivariate RelMP is generally a different minimum to the minimums of the individual, univariate RelMPs. What my code does is use a univariate RelMP of the mid price, and 2 multivariate RelMPs of mid price plus high-low range and mid price, high-low range and volume. This gives 3 sets of minValues and minValueIdxs, one for each set of data. The idea is to run the ii loop for, e.g. 500 iterations, and to then identify possible "robust" IB cluster motifs by using the Octave intersect function to get the minIdx that are common to all 3 sets of Idx data. 

By way of example, setting the ii loop iteration to just 100 results in only one intersect Idx value on some EUR_USD forex data, the plot of which is shown below:

Comparing this with the IB screenshot above, I would say this represents a typical "Open Auction" process with prices rotating upwards/downwards with no real conviction either way, with a possible long breakout on the last bar or alternatively, a last upwards test before a price plunge.

My intent is to use the above methodology to get a set of candidate IB motifs upon which a clustering algorithm can be based. This clustering algorithm will be the subject of my next post.

Market/Volume Profile and Matrix Profile

A quick preview of what I am currently working on: using Matrix Profile to search for time series motifs, using the R tsmp package. The exact motifs I'm looking for are the various "initial balance" set ups of Market Profile charts. 

To do so, I'm concentrating the investigation around both the London and New York opening times, with a custom annotation vector (av). Below is a simple R function to set up this custom av, which is produced separately in Octave and then loaded into R.

mp_adjusted_by_custom_av <- function( mp_object , custom_av ){
  ## https://stackoverflow.com/questions/66726578/custom-annotation-vector-with-tsmp-r-package
  mp_object$av <- custom_av
  class( mp_object ) <- tsmp:::update_class( class( mp_object ) , "AnnotationVector" )
  mp_adjusted_by_custom_av <- tsmp::av_apply( mp_object )
  return( mp_adjusted_by_custom_av )
}

This animated GIF shows plots of short, exemplar adjusted market profile objects highlighting the London only, New York only and combined results of the relevant annotation vectors.

This is currently a work in progress and so I shall report results in due course.

A Forex Pair Snapshot Chart

After yesterday's Heatmap Plot of Forex Temporal Clustering post I thought I would consolidate all the chart types I have recently created into one easy, snapshot overview type of chart. Below is a typical example of such a chart, this being today's 10 minute EUR_USD forex pair chart up to a few hours after the London session close (the red vertical line).

The top left chart is a Market/Volume Profile Chart with added rolling Value Area upper and lower bounds (the cyan, red and white lines) and also rolling Volume Weighted Average Price with upper and lower standard deviation lines (magenta).

The bottom left chart is the turning point heatmap chart as described in yesterday's post.

The two rightmost charts are also Market/Volume Profile charts, but of my Currency Strength Candlestick Charts based on my Currency Strength Indicator. The upper one is the base currency, i.e. EUR, and the lower is the quote currency. 

The following charts are the same day's charts for:

GBP_USD,

USD_CHF

and finally USD_JPY

The regularity of the turning points is easily seen in the lower lefthand charts although, of course, this is to be expected as they all share the USD as a common currency. However, there are also subtle differences to be seen in the "shadows" of the lighter areas.

For the nearest future my self-assigned task will be to observe the forex pairs, in real time, through the prism of the above style of chart and do some mental paper trading, and perhaps some really small size, discretionary live trading, in additional to my normal routine of research and development.

Heatmap Plot of Forex Temporal Clustering of Turning Points

Following up on my previous post, below is the chart of the temporal turning points that I have come up with.

This particular example happens to be 10 minute candlesticks over the last two days of the GBP_USD forex pair.

The details I have given about various turning points over the course of my last few posts have been based on identifying the "ix" centre value of turning point clusters. However, for plotting purposes I felt that just displaying these ix values wouldn't be very illuminating. Instead, I have taken the approach of displaying a sort of distribution of turning points per cluster. I would refer readers to my temporal clustering part 3 post wherein there is a coloured histogram of the R output of the clustering algorithm used. What I have done for the heatmap background of the above chart is normalise each separate, coloured histogram by the maximum value within the cluster and then plotted these normalised cluster values using Octave's pcolor function. An extra step taken was to raise the values to the power four just to increase the contrast within and between the sequential histogram backgrounds.

Each normalised histogram has a single value of one, which is shown by the bright yellow vertical lines, one per cluster. This represents the time of day at which, within the cluster window, the greatest number of turns occured in the historical lookback period. The darker green lines show other times within the cluster at which other turns occured.

The hypothesis behind this is that there are certain times of the day when price is more likely to change direction, a turning point, than at other times. Such times are market opens, closes etc. and the above chart is a convenient visual representation of these times. The lighter the backgound, the greater the probability that such a turn will occur, based upon the historical record of such turn timings.

Enjoy!
 

Temporal Clustering Times on Forex Majors Pairs

In the following code box there are the results from the temporal clustering routine of my last few posts on the four forex majors pairs of EUR_USD, GBP_USD, USD_CHF and USD_JPY.

###### EUR_USD 10 minute bars #######
## In the following order
## Both Delta turning point filter and "normal" TPF combined ##
## Delta turning point filter only ##
## "Normal" turning point filter only

###################### Monday ##############################################
K_opt == 8, ix values == 13  38    63    89     112    135    162    186    ## averaged over all 15 n_bars 1 to 15 inclusive
                         00  4:10  8:20  12:40  16:30  20:20  00:50  4:50
                         
K_opt == 8, ix values == 13 39 64 89 112 135 161 186                        ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 5, ix_values == 21 60 97 134 175                                   ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )
K == 6,     ix values == 21 59 94 125 158 184

K_opt == 11, ix values == 9 26 43 60 78 95 113 132 151 169 185              ## averaged over all 15 n_bars 1 to 15 inclusive

K_opt == 8, ix values == 13  36  61  86 111 136 161 186                     ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 8, ix values == 13  34  61  87 110 137 164 187                     ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

K_opt == 8, ix values == 13  38  63  88 112 137 162 186                     ## averaged over all 15 n_bars 1 to 15 inclusive

K_opt == 10, ix values == 10  31  52  72  91 112 131 150 169 188            ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 8, ix values == 12  35  62  88 112 137 164 187                     ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Tuesday #############################################
K_opt == 6, ix values == 131   169    206   244    283    322               ## averaged over all 15 n_bars 1 to 15 inclusive
                         19:40 02:00  8:10  14:30  21:00  03:30
                         
K_opt == 6, ix values == 131 170 207 245 284 323                            ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 7, ix values == 131 168 206 243 274 305 330                        ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

K_opt == 11, ix values == 124 143 164 184 205 226 247 268 289 310 331       ## averaged over all 15 n_bars 1 to 15 inclusive

K_opt == 11, ix values == 124 144 164 185 204 225 246 267 288 309 332       ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour ) 

K_opt == 7, ix values = 133 169 206 241 273 304 329                         ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

K_opt == 9, ix values == 127 152 175 202 228 253 278 305 330                ## averaged over all 15 n_bars 1 to 15 inclusive

K_opt == 9, ix values == 127 152 177 202 228 253 278 304 329                ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 7, ix values == 132 168 205 242 273 304 329                        ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Wednesday ###########################################
K_opt == 6, ix values == 275    312    351    389    426    465             ## averaged over all 15 n_bars 1 to 15 inclusive
                         19:40  01:50  08:20  14:40  20:50  03:20
                         
K_opt == 6, ix values == 275 313 352 391 428 466                            ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 6, ix values == 274 312 350 389 424 463                            ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

K_opt == 9, ix values == 272 299 322 347 372 397 422 449 474                ## averaged over all 15 n_bars 1 to 15 inclusive

K_opt == 11, ix values == 268 288 308 329 348 369 390 411 432 453 476       ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 6, ix values == 275 312 351 388 424 463                            ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

K_opt == 9, ix values == 272 297 322 348 373 398 423 449 474                ## averaged over all 15 n_bars 1 to 15 inclusive 

K_opt == 9, ix values == 271 297 322 348 373 398 423 448 473                ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 6, ix values == 276 311 350 389 426 465                            ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

####################### Thursday ###########################################
K_opt == 6, ix values == 420    457    495    532    570    609             ## averaged over all 15 n_bars 1 to 15 inclusive
                         19:50  02:00  08:20  14:30  20:50  03:20
                         
K_opt == 6, ix values == 420 457 494 531 570 610                            ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 6, ix values == 420 457 495 532 568 607                            ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

K_opt == 9, ix values == 416 443 466 492 518 543 568 593 618                ## averaged over all 15 n_bars 1 to 15 inclusive 

K_opt == 10, ix values == 414 437 460 483 506 527 550 573 596 619           ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 9, ix values == 416 443 466 493 520 543 568 595 618                ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

K_opt == 9, ix values == 415 440 465 492 518 543 568 593 618                ## averaged over all 15 n_bars 1 to 15 inclusive 

K_opt == 9, ix values ==  415 440 465 492 518 543 568 593 618               ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 7, ix values == 420 457 494 529 561 592 617                        ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

####################### Friday #############################################
K_opt == 5, ix values == 564    599    635    670     703                   ## averaged over all 15 n_bars 1 to 15 inclusive
                         19:50  01:40  07:40  13:30   19:00
                         
K_opt == 6, ix values == 563 596 627 654 680 707                            ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour ) 
K == 5,     ix values == 564 599 635 668 703

K_opt == 5, ix values == 564 601 639 674 705                                ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

K_opt == 9, ix values == 556 575 595 614 633 652 672 691 711                ## averaged over all 15 n_bars 1 to 15 inclusive

K_opt == 11, ix values == 554 570 587 602 619 634 651 667 682 698 713       ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 9, ix values == 556 575 595 614 633 652 671 691 711                ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 9, ix values == 556 575 596 613 634 652 672 691 711                ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

K_opt == 9, ix values == 556 575 594 613 633 652 672 691 710                ## averaged over all 15 n_bars 1 to 15 inclusive 

K_opt == 9, ix values == 556 575 594 613 634 653 672 691 710                ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt == 5, ix values == 564 600 637 674 705                                ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

############################################################################

###### GBP_USD 10 minute bars #######
## In the following order
## Both Delta turning point filter and "normal" TPF combined ##

###################### Monday ##############################################
K_opt = 8, ix_values = 13    36    61    86     111    136    162    186    ## averaged over all 15 n_bars 1 to 15 inclusive
                       0:00  3:50  8:00  12:10  16:20  20:30  0:50   4:50

K_opt = 9, ix_values = 12  34  56  78  99 120 141 164 187                   ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour ) 

K_opt = 8, ix_values = 12  35  61  86 110 136 163 186                       ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Tuesday #############################################
K_opt = 12, ix_values = 124    143    162   180   199   216   235   254   274   293   312   332     ## averaged over all 15 n_bars 1 to 15 inclusive
                        18:30  21:40  0:50  3:50  7:00  9:50  13:00 16:10 19:30 22:40 1:50  5:10

K_opt = 11, ix_values = 124 143 164 185 206 227 248 269 290 311 332         ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )  

K_opt = 9, ix_values = 128 154 177 205 230 254 279 307 330                  ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Wednesday ###########################################
K_opt = 11, ix_values = 269   290   311  331  352  373   394   415   434   455   476   ## averaged over all 15 n_bars 1 to 15 inclusive
                        18:40 22:10 1:40 5:00 8:30 12:00 15:30 19:00 22:10 1:40  5:10

K_opt = 11, ix_values = 269 289 310 330 351 372 393 413 434 455 476         ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour ) 

K_opt = 8, ix_values = 275 310 341 367 394 422 451 475                      ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Thursday ############################################
K_opt = 9, ix_values = 415   440   465  492  517   542   568   594   618    ## averaged over all 15 n_bars 1 to 15 inclusive
                       19:00 23:10 3:20 7:50 12:00 16:10 20:30 0:50  4:50

K_opt = 9, ix_values = 415 440 465 491 517 542 568 593 618                  ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )  

K_opt = 9, ix_values = 416 441 464 492 519 542 569 596 619                  ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Friday ##############################################
K_opt = 9, ix_values = 557   576   595  614  633  652   671   690   711     ## averaged over all 15 n_bars 1 to 15 inclusive
                       18:40 21:50 1:00 4:10 7:20 10:30 13:40 16:50 20:20

K_opt = 9, ix_values = 557 576 595 614 633 652 671 691 711                  ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour ) 

K_opt = 8, ix_values = 557 576 599 621 642 665 686 709                      ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

############################################################################

###### USD_CHF 10 minute bars #######
## In the following order
## Both Delta turning point filter and "normal" TPF combined ##

###################### Monday ##############################################
K_opt = 11, ix_values = 8      25    42   61   79    96    113   131   150   169  188   ## averaged over all 15 n_bars 1 to 15 inclusive
                        23:10  2:00  4:50 8:00 11:00 13:50 16:40 19:40 22:50 2:00 5:10

K_opt = 11, ix_values =  9  26  43  60  79  96 114 133 151 170 189          ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour ) 

K_opt = 7, ix_values =  13  38  66  99 127 157 184                          ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Tuesday #############################################
K_opt = 9, ix_values = 127   152   177  202  228   253   279   306  330     ## averaged over all 15 n_bars 1 to 15 inclusive
                       19:00 23:10 3:20 7:30 11:50 16:00 20:20 0:50 4:50

K_opt = 11, ix_values = 124 144 165 185 204 225 246 267 288 309 331         ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )  

K_opt = 7, ix_values = 133 170 205 240 270 301 328                          ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Wednesday ###########################################
K_opt = 10, ix_values = 270   293   316  342  365   388   411   432   454  475  ## averaged over all 15 n_bars 1 to 15 inclusive
                        18:50 22:40 2:30 6:50 10:40 14:30 18:20 21:50 1:30 5:00

K_opt = 12, ix_values = 268 287 308 327 346 365 384 401 420 439 458 477     ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )  

K_opt = 7, ix_values = 276 313 349 383 414 444 471                          ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Thursday ############################################
K_opt = 11, ix_values = 413   432   452  471  491  512   533   554   575   598  619  ## averaged over all 15 n_bars 1 to 15 inclusive
                        18:40 21:50 1:10 4:20 7:40 11:10 14:40 18:10 21:40 1:30 5:00

K_opt = 12, ix_values = 412 431 450 469 488 507 526 545 563 582 601 621     ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour ) 

K_opt = 9, ix_values = 415 440 463 491 518 543 570 597 619                  ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Friday ##############################################
K_opt = 9, ix_values = 557   576   596  615  634  653   672   691   710     ## averaged over all 15 n_bars 1 to 15 inclusive
                       18:40 21:50 1:10 4:20 7:30 10:40 13:50 17:00 20:10

K_opt = 9, ix_values = 556 575 595 614 633 652 671 690 710                  ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour ) 

K_opt = 7, ix_values = 558 579 602 629 652 677 705                          ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )                

############################################################################

###### USD_JPY 10 minute bars #######
## In the following order
## Both Delta turning point filter and "normal" TPF combined ##

###################### Monday ##############################################
K_opt = 12, ix_values = 8     24   41   58   73    90    107   124   141   158  173  190  ## averaged over all 15 n_bars 1 to 15 inclusive
                        23:10 1:50 4:40 7:30 10:00 12:50 15:40 18:30 21:20 0:10 2:40 5:30

K_opt = 12, ix_values = 8  24  41  56  73  90 107 124 141 158 173 190       ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt = 5, ix_values = 20  60  99 136 175                                   ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Tuesday #############################################
K_opt = 9, ix_values = 128   154   179  204  229   254   279   306  331     ## averaged over all 15 n_bars 1 to 15 inclusive
                       19:10 23:30 3:40 7:50 12:00 16:10 20:20 0:50 5:00

K_opt = 9, ix_values = 128 153 178 203 228 254 279 305 330                  ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt = 7, ix_values = 133 168 205 240 271 302 329                          ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Wednesday ###########################################
K_opt = 11, ix_values = 269   289   310  331  352  373   394   414   433   454  476  ## averaged over all 15 n_bars 1 to 15 inclusive
                        18:40 22:00 1:30 5:00 8:30 12:00 15:30 18:50 22:00 1:30 5:10

K_opt = 9, ix_values = 272 297 322 348 374 399 424 449 474                  ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt = 10, ix_values = 269 288 309 331 352 376 398 423 450 475             ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Thursday ############################################
K_opt = 9, ix_values = 416   442   467  492  518   543   568   593  618     ## averaged over all 15 n_bars 1 to 15 inclusive
                       19:10 23:30 3:40 7:50 12:10 16:20 20:30 0:40 4:50

K_opt = 12, ix_values = 412 431 450 469 488 507 526 545 564 583 602 621     ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt = 7, ix_values = 420 455 492 527 560 591 618                          ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

###################### Friday ##############################################
K_opt = 7, 8 or 9
ix_values 7 = 561   588   613       638        663   686    709             ## averaged over all 15 n_bars 1 to 15 inclusive
ix_values 8 = 557   578   599  622  643        666   687    710
ix_values 9 = 557   576   596  616  635  653   672   691    711             ## timings are for this bottom row
              18:40 21:50 1:10 4:30 7:40 10:40 13:50 17:00  20:20

K_opt = 8, ix_values = 558 579 600 621 644 665 687 709                      ## averaged over n_bars 1 to 6 inclusive ( upto and include 1 hour )

K_opt = 6, ix_values = 563 594 621 646 676 705                              ## averaged over n_bars 7 to 15 inclusive ( over 1 hour )

############################################################################

This is based on 10 minute bars over the last year or so. Readers should read my last few previous posts for background.

The first set of results, EUR_USD, are what the charts of my previous posts were based on and include combined results of my "Delta Turning Point Filter" and "Normal Turning Point Filter" and the results for each filter separately. Since there doesn't appear to be significant differences between these, the other three pairs' results are the combined filter results only.

The K_opt variable is the optimal number of clusters (see my temporal-clustering-part-3 post for how "optimal" is decided) and the ix_values are also described in this post. For convenience the first set of ix_values per day have the relevant times anotated underneath and therefore it is a simple matter to count forwards/backwards in 10 minute increments to place times to the other ix_values. The variable n_bars is an input to the turning point filter functions and essentially indicates the lookback/lookforward period (n_bar == 2 would mean 2 x 10 minute periods) used for determining a local high/low according to each function's logic.

As to how to interpret this, a typical sequence of times per day might look like this:

18:40 22:00 1:30 5:00 8:30 12:00 15:30 18:50 22:00 1:30 5:10

where the highlighted times represent the BST times for the period covering the London session open to the New York session close for one day. The preceding and following times are the two "book-ending" Asian sessions. 

Close inspection of these results reveals some surprising regularities. In even just the above single example (an actual copy and paste of a code box example) there appear to be definite times per day at which a local high/low occurs. I hopefully will be able to incorporate this into some type of chart for a nice visual presentation of the data. 

More in due course. Enjoy.