Downloading FX Pairs via Oanda API to Calculate Currency Strength Indicator

In the past I have posted a series of blog posts about a Currency Strength Indicator (here, here, here and here). This blog post gives an Octave function to use Oanda's API to download all the 10 minute OHLC data required to calculate the above strength indicators on the 10 minute time frame.

## Copyright (C) 2020 dekalog
## 
## This program is free software: you can redistribute it and/or modify it
## under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 3 of the License, or
## (at your option) any later version.
## 
## This program is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
## GNU General Public License for more details.
## 
## You should have received a copy of the GNU General Public License
## along with this program.  If not, see
## .

## -*- texinfo -*- 
## @deftypefn {} {@var{retval} =} get_currency_index_10m_pairs()
##
## This function gets the date and time value of the last currency index update for
## 10 minute bars by reading the last line of the file at:
##
## "/home/path/to/file"
##
## and then downloads all the currencies required to calculate new values for
## new currency index calculations, via looped Oanda API calls. 
## 
##The RETVAL is a matrix of GMT dates in the form
## YYYY:MM:DD:HH:MM in the first 5 columns, followed by the 45 required
## currency candlestick close values.
##
## @seealso{}
## @end deftypefn

## Author: dekalog 
## Created: 2020-06-01

function retval = get_currency_index_10m_pairs()
 
## cell array of currency crosses to iterrate over to get the complete set 
## of currency crosses to create a currency index
iter_vec = {'AUD_CAD','AUD_CHF','AUD_HKD','AUD_JPY','AUD_NZD','AUD_SGD',...
'AUD_USD','CAD_CHF','CAD_HKD','CAD_JPY','CAD_SGD','CHF_HKD','CHF_JPY',...
'EUR_AUD','EUR_CAD','EUR_CHF','EUR_GBP','EUR_HKD','EUR_JPY','EUR_NZD',...
'EUR_SGD','EUR_USD','GBP_AUD','GBP_CAD','GBP_CHF','GBP_HKD','GBP_JPY',...
'GBP_NZD','GBP_SGD','GBP_USD','HKD_JPY','NZD_CAD','NZD_CHF','NZD_HKD',...
'NZD_JPY','NZD_SGD','NZD_USD','SGD_CHF','SGD_HKD','SGD_JPY','USD_CAD',...
'USD_CHF','USD_HKD','USD_JPY','USD_SGD'} ;

## read last line of current 10min_currency_indices
unix_command = [ "tail -1" , " " , "/home/path/to/file" ] ;
[ ~ , data ] = system( unix_command ) ;
data = strsplit( data , ',' ) ; ## gives a cell arrayfun of characters
## zero pad singular month representations, i.e. 1 to 01
if ( numel( data{ 2 } == 1 ) )
data{ 2 } = [ '0' , data{ 2 } ] ;
endif
## and also zero pad singular dates
if ( numel( data{ 3 } == 1 ) )
data{ 3 } = [ '0' , data{ 3 } ] ;
endif
## and also zero pad singular hours
if ( numel( data{ 4 } == 1 ) )
data{ 4 } = [ '0' , data{ 4 } ] ;
endif
## and also zero pad singular minutes
if ( numel( data{ 5 } == 1 ) )
data{ 5 } = [ '0' , data{ 5 } ] ;
endif
 
## set up the headers
Hquery = [ 'curl -s -H "Content-Type: application/json"' ] ; ## -s is silent mode for Curl for no paging to terminal
Hquery = [ Hquery , ' -H "Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"' ] ;
query_begin = [ Hquery , ' "https://api-fxtrade.oanda.com/v3/instruments/' ] ;

## get time from last line of data
query_time = [ data{1} , '-' , data{2} , '-' , data{3} , 'T' , data{4} , '%3A' , data{5} , '%3A00.000000000Z&granularity=M10"' ] ;

## initialise with AUD_CAD
## construct the API call for particular cross
query = [ query_begin , iter_vec{ 1 } , '/candles?includeFirst=true&price=M&from=' , query_time ] ;
## call to use external Unix systems/Curl and return result
[ ~ , ret_JSON ] = system( query , RETURN_OUTPUT = 'TRUE' ) ;
## convert the returned JSON object to Octave structure
S = load_json( ret_JSON ) ;
## parse the returned structure S
if ( strcmp( fieldnames( S( 1 ) ) , 'errorMessage' ) == 0 ) ## no errorMessage in S
end_ix = numel( S.candles ) ; ## how many candles?
if ( S.candles{ end }.complete == 0 ) end_ix = end_ix - 1 ; endif ## account for incomplete candles
## create retval
retval = zeros( end_ix , 50 ) ; ## 45 currencies plus YYYY:MM:DD:HH:MM columns
for ii = 1 : end_ix
 date_time = strsplit( S.candles{ ii }.time , { '-' , 'T' , ':' } ) ;
 retval( ii , 1 ) = str2double( date_time( 1 , 1 ) ) ; ## year
 retval( ii , 2 ) = str2double( date_time( 1 , 2 ) ) ; ## month
 retval( ii , 3 ) = str2double( date_time( 1 , 3 ) ) ; ## day
 retval( ii , 4 ) = str2double( date_time( 1 , 4 ) ) ; ## hour
 retval( ii , 5 ) = str2double( date_time( 1 , 5 ) ) ; ## min
 retval( ii , 6 ) = str2double( S.candles{ ii }.mid.c ) ; ## candle close price 
endfor ## end of ii loop
else
error( 'Initialisation with AUD_CAD has failed.' ) ; 
endif ## end of strcmp if

for ii = 2 : numel( iter_vec )
## construct the API call for particular cross
query = [ query_begin , iter_vec{ ii } , '/candles?includeFirst=true&price=M&from=' , query_time ] ;
## call to use external Unix systems/Curl and return result
[ ~ , ret_JSON ] = system( query , RETURN_OUTPUT = 'TRUE' ) ;
## convert the returned JSON object to Octave structure
S = load_json( ret_JSON ) ;
## parse the returned structure S
if ( strcmp( fieldnames( S( 1 ) ) , 'errorMessage' ) == 0 ) ## no errorMessage in S
end_ix = numel( S.candles ) ; ## how many candles?
if ( S.candles{ end }.complete == 0 ) end_ix = end_ix - 1 ; endif ## account for incomplete candles
temp_retval = zeros( end_ix , 6 ) ;
for jj = 1 : end_ix
 date_time = strsplit( S.candles{ jj }.time , { '-' , 'T' , ':' } ) ;
 temp_retval( jj , 1 ) = str2double( date_time( 1 , 1 ) ) ; ## year
 temp_retval( jj , 2 ) = str2double( date_time( 1 , 2 ) ) ; ## month
 temp_retval( jj , 3 ) = str2double( date_time( 1 , 3 ) ) ; ## day
 temp_retval( jj , 4 ) = str2double( date_time( 1 , 4 ) ) ; ## hour
 temp_retval( jj , 5 ) = str2double( date_time( 1 , 5 ) ) ; ## min
 temp_retval( jj , 6 ) = str2double( S.candles{ jj }.mid.c ) ; ## candle close price  
endfor ## end of jj loop

## checks dates and times allignment before writing to retval
date_time_diffs_1 = setdiff( retval( : , 1 : 5 ) , temp_retval( : , 1 : 5 ) , 'rows' ) ; 
date_time_diffs_2 = setdiff( temp_retval( : , 1 : 5 ) , retval( : , 1 : 5 ) , 'rows' ) ;

 if ( isempty( date_time_diffs_1 ) && isempty( date_time_diffs_2 ) ) 
 ## there are no differences between retval dates and temp_retval dates 
 retval( : , ii + 5 ) = temp_retval( : , 6 ) ;
 
 elseif ( ~isempty( date_time_diffs_1 ) || ~isempty( date_time_diffs_2 ) )
 ## implies a difference between the date_times of retval and temp_retval, so merge them
 
 dn_retval = datenum( [ retval(:,1) , retval(:,2) , retval(:,3) , retval(:,4) , retval(:,5) ] ) ;
 dn_temp_retval = datenum( [ temp_retval(:,1) , temp_retval(:,2) , temp_retval(:,3) , temp_retval(:,4) , temp_retval(:,5) ] ) ;
 new_dn = unique( [ dn_retval ; dn_temp_retval ] ) ; new_date_vec = datevec( new_dn ) ; new_date_vec( : , 6 ) = [] ; 
 new_retval = [ new_date_vec , zeros( size( new_date_vec , 1 ) , 45 ) ] ;
 [ TF , S_IDX ] = ismember( new_retval( : , 1 : 5 ) , retval( : , 1 : 5 ) , 'rows' ) ;
 TF_ix = find( TF ) ; new_retval( TF_ix , 6 : end ) = retval( : , 6 : end ) ;
 [ TF , S_IDX ] = ismember( new_retval( : , 1 : 5 ) , temp_retval( : , 1 : 5 ) , 'rows' ) ;
 TF_ix = find( TF ) ; new_retval( TF_ix , ii + 5 ) = temp_retval( : , 6 ) ; 
 retval = new_retval ; 
 clear new_retval new_dn dn_temp_retval dn_retval date_time_diffs_1 date_time_diffs_2 ;
 
 else 
 error( 'Mismatch between dates and times for writing to retval.' ) ; 
 endif ## TF == S_IDX check

endif ## end of strcmp if

endfor ## ii loop

endfunction

At the moment there are almost 50 separate API calls nested within a loop, which of course is non-vectorised and inefficient, and if I find out how to make a batch API call to do this I shall rewrite the function.

This function is called in a script which uses the output matrix "retval" to then calculate the various currency strengths as outlined in the above linked posts. The total running time for this script is approximately 40 seconds from first call to appending to the index file on disk. I wrote this function to leverage my new found Oanda API knowledge to avoid having to accumulate an ever growing set of files on disk containing the raw 10 minute data.

I hope readers find this useful.