Hacker’s Tools

For performing our financial hacking experiments (and for earning the financial fruits of our labor) we need some software machinery for research, testing, training, and trading financial algorithms. No existing software platform today is really up to all those tasks. So you have no choice but to put together your system from different software packages. Fortunately, two are normally sufficient. I’ll use Zorro and R for most articles on this blog, but will also occasionally look into other tools.

Choice of languages

There’s not as much choice as it seems at first glance. You can avoid a programming language entirely by using a visual ‘strategy builder’, ‘code wizard’ or spreadsheet program for defining your strategy. Unfortunately this works for rather simple systems only, and I never heard of such a system that produced any consistent trading profit. For real development and research tasks, there’s no stepping around ‘real programming’.

You’re also not totally free to select the programming language with the nicest or easiest syntax. One of the best compromises of simplicity and object orientation is probably Python. It also offers libraries with useful statistics and indicator functions. Consequently, many strategy developers start with programming their systems in Python. And eventually they wonder why they’re all the time waiting for their program to come up with results. And this even after having spent a lot of money for really fast computers. There’s another criterion that is more relevant for system development than syntax: execution speed.

Speed mostly depends on whether a computer language is compiled or interpreted. C, Pascal, or Java are compiled languages, meaning that the code runs directly on the processor (C, C++, Pascal) or on a ‘virtual machine’ (Java). Python, R, or Matlab is interpreted: The code won’t run by itself, but is executed by an interpreter software. Interpreted languages are much slower and need more CPU and memory resources than compiled languages. But they have the advantage of being interactive: you can enter commands directly at a console. Some languages, such as C#, are inbetween: They are compiled to a machine-independent interim code that is then, dependent on implementation, either interpreted or converted to machine code. C# is about 4 times slower than C, but still 30 times faster than Python.

Here’s a benchmark table of the same two test programs written in several languages: a sudoku solver and a loop with a 1000 x 1000 matrix multiplication (in seconds):

Language Sudoku Matrix
C, C++ 1.0 1.8
Java 1.7 2.6
Pascal 4
C# 3.8 9
JavaScript 18.1 16
Erlang 18 31
Python 119 121
Ruby 98 628
Matlab 621
R 1738

Why is execution speed so important for trading systems? Strategy development is a mostly empirical method; you’re all the time running tests with variants of your system. Assume that a C-written strategy needs 1 minute for a test run. The same strategy written in EasyLanguage would need about 30 minutes, in Python 2 hours, and in R more than 10 hours! If I had coded the trend experiment in R, I would today still be waiting for the results. You can see why trade platforms normally use a C variant or a proprietary compiled language for their strategies. High-frequency trading systems are either coded in C or directly in machine language.

Even compiled languages can have large speed differences due to different implementation of trading and analysis functions. The same benchmark script, a small RSI strategy from this page, runs with very different speeds on different trading platforms (10 years backtest, ticks resolution): 

  • Zorro: ~ 4 seconds (lite-C, a C variant)
  • MT4:  ~ 110 seconds (MQL4, another C variant)
  • MultiCharts: ~ 155 seconds (EasyLanguage, a C/Pascal mix)

However, the differences are not as bad as suggested by the benchmark table. There’s a trick for overcoming slow language speed. Even interpreted languages have function libraries that are often written in C/C++. A script that does not have to go step by step through historical data, but only calls library functions that process all data simultaneously, would run with comparable speed in all languages. Indeed some trading systems can be coded in this vector-based method (you’ll see an example below in R code). Unfortunately this works only with simple systems, and only for backtests. It can not be used for strategies that really trade.

Choice of tools

Zorro is a software for financial analysis and algo-trading – a sort of Swiss Knife tool since you can use it for all sorts of quick tests. It’s my software of choice for financial hacking because:

  • It’s free (unless you’re rich).
  • Scripts are in C, event driven and very fast. You can code a system or an idea in 5 minutes. 
  • Open architecture – you can add anything with DLL plugins.
  • Minimalistic – just a frontend to a programming language.
  • Can be automatized for experiments.
  • Very accurate, realistic trading simulation. 
  • Native portfolio support (multiple assets, algos, and time frames).
  • Has all basic data processing and statistics functions (most included with source code).
  • Is continuously developed and supported (new versions usually come out every 2..3 months).
  • Last but not least: I know it quite well, as I’ve written its tutorial…

Zorro

A strategy example in C, the classic SMA crossover:

function run()
{
  vars Close = series(priceClose());
  vars MA30 = series(SMA(Close,30));	
  vars MA100 = series(SMA(Close,100));
	
  Stop = 4*ATR(100);
  if(crossOver(MA30,MA100))
    enterLong();
  if(crossUnder(MA30,MA100))
    enterShort();
}

You can see that Zorro offers a very easy trading implementation. But here comes the drawback of the C language: You can not as easy drop in external libraries as in Python or R. Using a C/C++ based data analysis or machine learning package involves sometimes a lengthy implementation. Fortunately, Zorro can also call R functions for those purposes. 

R is a script interpreter for data analysis and charting. It is not a real language with consistent syntax, but more a conglomerate of operators, functions, and data structures that has grown over 20 years. It’s not very logically structured and harder to learn than a normal computer language, but offers some unique advantages. I’ll use it in this blog when it comes to complex analysis or machine learning tasks. It’s my tool of choice for financial hacking because:

  • It’s free. (“Software is like sex: it’s better when it’s free.”)
  • R scripts can be very short and effective (once you got used to the syntax).
  • It’s the global standard for data analysis and machine learning.
  • Open architecture – you can add modules for almost anything.
  • Minimalistic – just a console with a language interpreter. 
  • Has tons of “packages” for all imaginable mathematical and statistical tasks.
  • Is continuously developed and supported by the global scientific community (about 15 new packages usually come out every day).

r

This is the SMA crossover in R, for a vector-based backtest:

require(quantmod)
require(PerformanceAnalytics)

Data <- xts(read.zoo("EURUSD.csv", tz="UTC", format="%Y-%m-%d %H:%M", sep=",", header=TRUE))
Close <- Cl(Data)
MA30 <- SMA(Close,30)
MA100 <- SMA(Close,100)
 
Dir <- ifelse(MA30 > MA100,1,-1) # calculate trade direction
Dir.1 <- c(NA,Dir[-length(Dir)]) # shift by 1 for avoiding peeking bias
Return <- ROC(Close)*Dir.1 
charts.PerformanceSummary(na.omit(Return))

You can see that the vector-based code just consists of function calls. It runs almost as fast as the C equivalent. But it is difficult to read, it had to be rewritten for real trading, and details such as a stop loss had to be omitted since they would not work for a vector-based test. Thus, so good R is for interactive data analysis, so hopeless is it for writing trade strategies – although some R packages (for instance, quantstrat) even offer rudimentary optimization and test functions. They require an awkward coding style and do not simulate trading very realistically, but even then they are still too slow for serious tests. While R functions and packages alone can be pretty fast since they are mostly coded in C, R itself can not replace a serious backtest and trading platform. But Zorro and R complement each other perfectly.

More hacker’s tools

Aside from languages and platforms, you’ll often need auxiliary tools that may be small, simple, cheap, but all the more important since you’re using them all the time. For editing scripts I don’t use Zorro’s SED editor or the R console, but Notepad++. For interactive working with R I can recommend RStudio. Extremely helpful for strategy development is a file comparison tool: You often want to compare trade logs of different system variants and check which variant opened which trade a little earlier or later, and which consequences that had. For this I use Beyond Compare.

Aside from Zorro and R, there’s also a relatively new system development software that I plan to examine closer at some time in the future, TSSB for generating and testing bias-free trading systems with advanced machine learning algorithms. David Aronson and Timothy Masters were involved in its development, so it certainly won’t be as useless as most other “trade system generating” software. However, there’s again a limitation: TSSB can not trade or export, so you can not really use the ingenious systems that you developed with it. Maybe I’ll find a solution to combine TSSB with Zorro.

Links to the latest versions of Zorro and R are placed on the side bar. A R tutorial can be found here and a Zorro tutorial here. German readers can find here an extensive introduction into trading with Zorro.

9 thoughts on “Hacker’s Tools”

  1. Hi Johanne Christian.
    It is always interesting to read about your exploits. It seems, you have a real passion for the sience of algo trading and it’s contagious 🙂 Keep going!
    From the picture of your Zorro interface above, one can tell that it’s connected to IB. How did you do it?
    Thank for the great work and
    good luck!

  2. Yes, the IB plugin is in development and I have already a test version. It currently works for Forex, but not yet for stocks.

  3. Have you again looked at TSSB? I am biting my teeth out on it, but somehow not making enough progress.
    Thanks, Brian

  4. Not yet, but it’s on my todo list. Unfortunately the list is quite long at the moment, so it might take a while.

  5. Well, Zorro is not exactly free. It’s free unless you have more than 7000 US$ in your account.

  6. I am sceptical about this software. Nothing in this world comes free and this Zorro software looks too good to be true. If they want to give a software free then why don’t they open source it and host it in github.. Any way nice article

Leave a Reply

Your email address will not be published. Required fields are marked *