Hacking a HFT system

Compared with machine learning or signal processing algorithms of conventional trading strategies, High Frequency Trading systems can be surprisingly simple. They need not attempt to predict future prices. They know the future prices already. Or rather, they know the prices that lie in the future for other, slower market participants. Recently we got some contracts for simulating HFT systems in order to determine their potential profit and maximum latency. This article is about testing HFT systems the hacker’s way.

The HFT advantage is receiving price quotes earlier and getting orders filled faster than the majority of market participants. Its profit depends on the system’s latency, the delay between price quote and subsequent order execution at the exchange. Latency is the most relevant factor of a HFT system. It can be optimized in two ways: by minimizing the distance to the exchange, and by maximizing the speed of the trading system. The former is more important than the latter.

The location

Ideally, a HFT server is located directly at the exchange. And indeed most exchanges are dear selling server space in their cellars – the closer to the main network hub, the dearer the space. Electrical signals in a shielded wire travel with about 0.7 .. 0.9 times the speed of light (300 km/ms). 1 m closer to the signal source means a whopping 8 nanoseconds round-trip time advantage. How many trade opportunities disappear after 8 ns? I don’t know, but people are willing to pay for any saved nanosecond.

Unfortunately (or fortunately, from a cost perspective), the HFT system that we’ll examine here can not draw advantage from colocating at the exchange. For reasons that we’ll see soon, it must trade and receive quotes from the NYSE and the CME simultaneously. There’s a high speed cable and also a microwave radio link running between both cities. The theoretically ideal location of such a HFT system is Warren, Ohio, exactly halfway between New York City and Chicago. I don’t know whether there’s a high speed node in Warren, but if it is, the distance of 357 miles to both exchanges would be equivalent to about 4 ms round-trip latency.

Warren, Ohio – HFT Mecca of the world (image by Jack Pearce / Wikipedia Commons)

Without doubt, a server in this pleasant town would come at lower cost than a server in the cellar of the New York Stock Exchange. My today’s free tip for getting rich: Buy some garage space in Warren, tap into the high speed New York-Chicago cable or radio link, and rent out server racks!

The software

When you’ve already invested money for the optimal location and connection of the HFT system, you’ll also want a trading software that matches the required speed. Commercial trading platforms – even Zorro – are normally not fast enough. And you don’t know what they do behind your back. HFT systems are therefore normally not based on a trading platform, but directly coded. Not in R or Python, but in a fast language, usually one of the following:

  • C or C++. Good combination of high level and high speed. C is easy to read, but very effective and almost as fast as machine language.
  • Pentium Assembler. Write your algorithm in machine instructions. Faster than C when the algorithm mostly runs in loops that can be manually optimized. There are specialists for optimizing assembler code. Disadvantage: Any programmer has a hard time to understand assembler programs written by another programmer.
  • CUDA, HLSL, or GPU assembler. Run your algorithm on the shader pipeline of the PC’s video card. This makes sense when it is heavily based on vector and matrix operations.
  • VHDL. If any software would be too slow and trade success really depends on nanoseconds, the ultimate solution is coding the system in hardware. In VHDL you can define arithmetic units, digital filters, and sequencers on a FPGA (Field Programmable Gate Array) chip with a clock rate up to several 100 MHz. The chip can be directly connected to the network interface.

With exception of VHDL, especially programmers of 3D computer games are normally familar with those high-speed languages. But the standard language of HFT systems is plain C/C++. It is also used in this article.

The algorithm

Many HFT systems prey on traders by using fore-running methods. They catch your quote, buy the same asset some microseconds earlier, and sell it to you with a profit. Some exchanges prevent this in the interest of fair play, while other exchanges encourage this in the interest of earning more fees. For this article we won’t use fore-running methods, but simply exploit arbitrage between ES and SPY. We’re assuming that our server is located in Warren, Ohio, and has a high speed connection to Chicago and to New York.

ES is a Chicago traded S&P500 future, exposed to supply and demand. SPY is a New York traded ETF, issued by State Street in Boston and following the S&P500 index (minus State Street’s fees). 1 ES Point is equivalent to 10 SPY cents, so the ES price is about ten times the SPY price. Since both assets are based on the same index, we can expect that their prices are highly correlated. There have been some publications (1) that claim this correlation will break down at low time frames, causing one asset trailing the other. We’ll check if this is true. Any shortlived ES-SPY price difference that exceeds the bid-ask spreads and trading costs constitutes an arbitrage opportunity. Our arbitrage algorithm would work this way:

  • Determine the SPY-ES difference.
  • Determine its deviation from the mean. 
  • If the deviation exceeds the ask-bid spreads plus a threshold, open positions in ES and SPY in opposite directions.
  • If the deviation reverses its sign and exceeds the ask-bid spreads plus a smaller threshold, close the positions.

This is the barebone HFT algorithm in C. If you’ve never seen HFT code before, it might look a bit strange:

#define THRESHOLD  0.4  // Entry/Exit threshold 

// HFT arbitrage algorithm
// returns 0 for closing all positions
// returns 1 for opening ES long, SPY short
// returns 2 for opening ES short, SPY long
// returns -1 otherwise
int tradeHFT(double AskSPY,double BidSPY,double AskES,double BidES)
{
	double SpreadSPY = AskSPY-BidSPY, SpreadES = AskES-BidES;
	double Arbitrage = 0.5*(AskSPY+BidSPY-AskES-BidES);

	static double ArbMean = Arbitrage;	
	ArbMean = 0.999*ArbMean + 0.001*Arbitrage;
	static double Deviation = 0;
	Deviation = 0.75*Deviation + 0.25*(Arbitrage - ArbMean);

	static int Position = 0;	
	if(Position == 0) {
		if(Deviation > SpreadSPY+THRESHOLD)
			return Position = 1;
		if(-Deviation > SpreadES+THRESHOLD)
			return Position = 2;
	} else {
		if(Position == 1 && -Deviation > SpreadES+THRESHOLD/2)
			return Position = 0;
		if(Position == 2 && Deviation > SpreadSPY+THRESHOLD/2)
			return Position = 0;
	}
	return -1;	
}

The tradeHFT function is called from some framework – not shown here – that gets the price quotes and sends the trade orders. Parameters are the current best ask and bid prices of ES and SPY from the top of the order book (we assume here that the SPY price is multiplied with 10 so that both assets are on the same scale). The function returns a code that tells the framework whether to open or close opposite positions or to do nothing. The Arbitrage variable is the mid price difference of SPY and ES. Its mean (ArbMean) is filtered by a slow EMA, and the Deviation from the mean is also slightly filtered by a fast EMA to prevent reactions on outlier quotes. The Position variable constitutes a tiny state machine with the 3 states long, short, and nothing. The entry/exit Threshold is here set to 40 cents, equivalent to a bad SPY spread multiplied with 10. This is the only adjustable parameter of the system. If we wanted to really trade it, we had to optimize the threshold using several months’ ES and SPY data.

This minimalistic system would be quite easy to convert to Pentium assembler or even to a FPGA chip. But it is not really necessary: Even compiled with Zorro’s lite-C compiler, the tradeHFT function executes in just about 750 nanoseconds. A strongly optimizing C compiler, like Microsoft VC++,  gets the execution time down to 650 nanoseconds. Since the time span between two ES quotes is normally 10 microseconds or more, the speed of C is largely sufficient.

Our HFT experiment has to answer two questions. First, are those price differences big enough for arbitrage profit? And second, at which maximum latency will the system still work? 

The data

For backtesting a HFT system, normal price data that you can freely download from brokers won’t do. You have to buy high resolution order book or BBO data (Best Bid and Offer) that includes the exchange time stamps. Without knowing the exact time when the price quote was received at the exchange, it is not possible to determine the maximum latency.

Some companies are recording all quotes from all exchanges and are selling this data. Any one has its specific data format, so the first challenge is to convert this to lean data files that we then evaluate with our simulation software. We’re using this very simple target format for price quotes:

typedef struct T1    // single tick
{
	double time; // time stamp, OLE DATE format
	float fVal;  // positive = ask price, negative = bid price	
} T1;   

The company watching the Chicago Mercantile Exchange delivers its data in a specific CSV format with many additional fields, of which most we don’t need here (f.i. the trade volume or the quote arrival time). Every day’s quotes are stored in one CSV file. This is the Zorro script for pulling the ES December 2016 contract out of it and converting it to a dataset of T1 price quotes:

//////////////////////////////////////////////////////
// Convert price history from Nanotick BBO to .t1
//////////////////////////////////////////////////////

#define STARTDAY 20161004
#define ENDDAY   20161014

string InName = "History\\CME.%08d-%08d.E.BBO-C.310.ES.csv";  // name of a day file
string OutName = "History\\ES_201610.t1";
string Code = "ESZ";	// December contract symbol

string Format = "2,,%Y%m%d,%H:%M:%S,,,s,,,s,i,,";  // Nanotick csv format
void main()
{
	int N,Row,Record,Records;
	for(N = STARTDAY; N <= ENDDAY; N++)
	{
		string FileName = strf(InName,N,N+1);
		if(!file_date(FileName)) continue;
		Records = dataParse(1,Format,FileName);  // read BBO data
		printf("\n%d rows read",Records);
		dataNew(2,Records,2);  // create T1 dataset
		for(Record = 0,Row = 0; Record < Records; Record++)
		{
			if(!strstr(Code,dataStr(1,Record,1))) continue; // select only records with correct symbol
			T1* t1 = dataStr(2,Row,0);  // store record in T1 format
			float Price = 0.01 * dataInt(1,Record,3);  // price in cents
			if(Price < 1000) continue;  // no valid price
			string AskBid = dataStr(1,Record,2);
			if(AskBid[0] == 'B')  // negative price for Bid
				Price = -Price;
			t1->fVal = Price;
			t1->time = dataVar(1,Record,0) + 1./24.;  // add 1 hour Chicago-NY time difference
			Row++;
		}
		printf(", %d stored",Row);
		dataAppend(3,2,0,Row);  // append dataset
		if(!wait(0)) return;
	}
	dataSave(3,OutName);  // store complete dataset
}

We’re converting here 10 days in October 2016 for our backtest. The script first parses the CSV file into an intermediary binary dataset, which is then converted to the T1 target format. Since the time stamps are in Chicago local time, we have to add one hour to convert them to NY time.

The company watching the New York Stock exchange delivers data in a highly compressed specific format named “NxCore Tape”. We’re using the Zorro NxCore plugin for converting this to another T1 list:

//////////////////////////////////////////////////////
// Convert price history from Nanex .nx2 to .t1
//////////////////////////////////////////////////////

#define STARTDAY 20161004
#define ENDDAY 	 20161014
#define BUFFER	 10000

string InName = "History\\%8d.GS.nx2";  // name of a single day tape
string OutName = "History\\SPY_201610.t1";
string Code = "eSPY";

int Row,Rows;

typedef struct QUOTE {
	char	Name[24];
	var	Time,Price,Size;
} QUOTE;

int callback(QUOTE *Quote)
{
	if(!strstr(Quote->Name,Code)) return 1;
	T1* t1 = dataStr(1,Row,0);  // store record in T1 format
	t1->time = Quote->Time;
	t1->fVal = Quote->Price;
	Row++; Rows++;
	if(Row >= BUFFER)	{   // dataset full?
		Row = 0;
		dataAppend(2,1);    // append to dataset 2
	}
	return 1;
}


void main()
{
	dataNew(1,BUFFER,2); // create a small dataset
	login(1);            // open the NxCore plugin
	
	int N;
	for(N = STARTDAY; N <= ENDDAY; N++) {
		string FileName = strf(InName,N);
		if(!file_date(FileName)) continue;
		printf("\n%s..",FileName);
		Row = Rows = 0;  // initialize global variables
		brokerCommand(SET_HISTORY,FileName); // parse the tape
		dataAppend(2,1,0,Row);  // append the rest to dataset 2
		printf("\n%d rows stored",Rows);
		if(!wait(0)) return;  // abort when [Stop] was hit
	}
	dataSave(2,OutName); // store complete dataset
}

The callback function is called by any quote in the tape file. We don’t need most of the data, so we filter out only the SPY quotes (“eSPY”).

Verifying the inefficiency

With data from both sources, we can now compare the ES and SPY prices in high resolution. Here’s a typical 10 seconds sample from the price curves:

SPY (black) vs. ES (red), October 5, 2017, 10:01:25 – 10:01.35

The resolution is 1 ms. ES is plotted in $ units, SPY in 10 cents units, with a small offset so that the curves lie over each other. The prices shown are ask prices. We can see that ES moves in steps of 25 cents, SPY in steps of 1 cent. The prices are still well correlated even on a millisecond scale. ES appears to trail a tiny bit.

We can also see an arbitrage opportunity at the steep price step in the center at about 10:01:30. ES reacted a bit slower, but stronger on some event, probably a moderate price jump of one of the stocks of the S&P 500. This event also triggered a fast sequence of oscillating SPY quotes, most likely from other HFT systems, until the situation calmed down again a few 100 ms later (the scale is not linear since time periods with no quotes are skipped). For several milliseconds the ES-SPY difference exceeded the ask-bid spread of both assets (usually 25 cents for ES and 1..4 cents for SPY). We would here ideally sell ES and buy SPY immediately after the ES price step. So we have verified that the theoretized inefficiency does really exist, at least in this sample. 

The script for displaying high resolution charts:

#define ES_HISTORY	"ES_201610.t1"
#define SPY_HISTORY	"SPY_201610.t1"
#define TIMEFORMAT	"%Y%m%d %H:%M:%S"
#define FACTOR		10
#define OFFSET		3.575

void main()
{
	var StartTime = wdatef(TIMEFORMAT,"20161005 10:01:25"),
		EndTime = wdatef(TIMEFORMAT,"20161005 10:01:35");
	MaxBars = 10000;
	BarPeriod = 0.001/60.;	// 1 ms plot resolution
	Outlier = 1.002;  // filter out 0.2% outliers

	assetList("HFT.csv");
	dataLoad(1,ES_HISTORY,2);
	dataLoad(2,SPY_HISTORY,2);
	int RowES=0, RowSPY=0;
	
	while(Bar < MaxBars)
	{
		var TimeES = dataVar(1,RowES,0), 
			PriceES = dataVar(1,RowES,1), 
			TimeSPY = dataVar(2,RowSPY,0), 
			PriceSPY = dataVar(2,RowSPY,1);

		if(TimeES < TimeSPY) RowES++;
		else RowSPY++;

		if(min(TimeES,TimeSPY) < StartTime) continue;
		if(max(TimeES,TimeSPY) > EndTime) break;

		if(TimeES < TimeSPY) {
			asset("ES");
			priceQuote(TimeES,PriceES);
		} else {
			asset("SPY");
			priceQuote(TimeSPY,PriceSPY);
		}
		
		asset("ES");
		if(AssetBar > 0) plot("ES",AskPrice+OFFSET,LINE,RED);
		asset("SPY");
		if(AssetBar > 0) plot("SPY",FACTOR*AskPrice,LINE,BLACK);
	}
}

The script first reads the two historical data files that we’ve created above, and then parses them row by row. For keeping the ES and SPY quotes in sync, we always read the quote with the smaller timestamp from the datasets (the quotes are stored in ascending timestamp order). The priceQuote function checks the prices for outliers, stores the ask price in the AskPrice variable and the ask-bid difference in Spread, and increases the Bar count for plotting the price curves. A bar of the curve is equivalent to 1 ms. The AssetBar variable is the last bar with a price quote of that asset, and is used here to prevent plotting before the first quote arrived. 

Testing the system

For backtesting our HFT system, we only need to modify the script above a bit, and call the tradeHFT function in the loop:

#define LATENCY		4.0	// milliseconds

function main()
{
	var StartTime = wdatef(TIMEFORMAT,"20161005 09:30:00"),
		EndTime = wdatef(TIMEFORMAT,"20161005 15:30:00");
	MaxBars = 200000;
	BarPeriod = 0.1/60.;	// 100 ms bars 
	Outlier = 1.002;

	assetList("HFT.csv");
	dataLoad(1,ES_HISTORY,2);
	dataLoad(2,SPY_HISTORY,2);
	int RowES=0, RowSPY=0;
	
	EntryDelay = LATENCY/1000.;
	Hedge = 2;
	Fill = 8; // HFT fill mode;
	Slippage = 0;
	Lots = 100;
		
	while(Bar < MaxBars)
	{
		var TimeES = dataVar(1,RowES,0), 
			PriceES = dataVar(1,RowES,1),
			TimeSPY = dataVar(2,RowSPY,0),
			PriceSPY = dataVar(2,RowSPY,1);

		if(TimeES < TimeSPY) RowES++;
		else RowSPY++;

		if(min(TimeES,TimeSPY) < StartTime) continue;
		if(max(TimeES,TimeSPY) > EndTime) break;

		if(TimeES < TimeSPY) {
			asset("ES");
			priceQuote(TimeES,PriceES);
		} else {
			asset("SPY");
			priceQuote(TimeSPY,FACTOR*PriceSPY);
		}
		
		asset("ES");
		if(!AssetBar) continue;
		var AskES = AskPrice, BidES = AskPrice-Spread;
		asset("SPY");
		if(!AssetBar) continue;
		var AskSPY = AskPrice, BidSPY = AskPrice-Spread;

		int Order = tradeHFT(AskSPY,BidSPY,AskES,BidES);	
		switch(Order) {
			case 1: 
			asset("ES"); enterLong();
			asset("SPY"); enterShort();
			break;
		
			case 2: 
			asset("ES"); enterShort();
			asset("SPY"); enterLong();
			break;
		
			case 0:
			asset("ES"); exitLong(); exitShort();
			asset("SPY"); exitLong(); exitShort();
			break;
		}
	}
	printf("\nProfit %.2f at NY Time %s",
		Equity,strdate(TIMEFORMAT,dataVar(1,RowES,0)));
}

This script runs a HFT backtest of one trading day, from 9:30 until 15:30 New York time. We can see that it just calls the HFT function with the ES and SPY prices, then executes the returned code in the switch-case statement. It opens 100 units of each asset (equivalent to 2 ES and 1000 SPY contracts). The round-trip latency is set up with the EntryDelay variable. In HFT mode (Fill = 8), a trade is filled at the most recent price quote after the given delay. This ensures a realistic latency simulation when price quotes are entered with their exchange time stamps.

Here’s the HFT profit at the end of the day with different round-trip latency values:

LATENCY 0.5 ms 4.0 ms 6.0 ms 10 ms
Profit / day + $793 + $273 + $205 – $15

The ES-SPY HFT arbitrage system makes about $800 each day with an unrealistic small latency of 500 microseconds. Unfortunately, due to the 700 miles between the NYSE and the CME, you’d need a time machine for that (or some faster-than-light method of quantum teleportation). A HFT server in Warren, Ohio, at 4 ms latency would generate about $300 per day. A server slightly off the direct NY-Chicago line can still grind out $200 daily. But the system deteriorates quickly when located further away, as with a server in Nashville, Tennessee, at around 10 ms latency. This is a strong hint that some high-speed systems in the proximity of both exchanges are already busy with exploiting ES-SPY arbitrage.

Still, $300 per day result in a decent $75,000 annual income. However, what needed you to invest for that, aside from hardware and software? With SPY at $250, the 100 units translate to 100*$2500 + 100*10*$250 = half a million dollars trade volume. So you would get only 15% annual return on your investment. But you can improve this by adding more pairs of NY ETFs and their equivalent CME futures to the arbitrage strategy. And you can improve it further when you find a broker or similar service that can receive your orders directly at the exchanges. Due to the ETF/future hedging the position is almost without risk. So you could probably negotiate a large leverage. And also a flat monthly fee, since broker commissions were not included in the test.

Conclusion 

  • When systems react fast enough, profits can be achieved with very primitive methods, such as arbitrage between highly correlated assets.
  • Location has a large impact on HFT profits.
  • ES-SPY arbitrage cannot be traded by everyone from everywhere. You’re competing with people that are doing this already. Possibly in Warren, Ohio.

I’ve added the scripts and the asset list to the 2017 script archive in the “HFT” and “History” folders. Unfortunately I could not add the ES / SPY price history files, since I do not own the data. For reproducing the results, get BBO history from Nanex™ or Nanotick™ – their data can be read with the scripts above. You’ll also need Zorro S version 1.60 or above, which supports HFT fill mode.

References

(1) When Correlations Break Down (Jared Bernstein 2014)

(2) Financial Programming Greatly Accelerated (Cousin/Weston, Automated Trader 42/2017)

 


Addendum. I have been asked how the profit would be affected when the server is located in New York, with 0.25 ms signal delay to the NYSE and 4.8 ms signal delay to the CME. For simulating this, modify the script:

var TimeES = dataVar(1,RowES,0)+4.8/(1000*60*60*24),
  TimeSPY = dataVar(2,RowSPY,0)+0.25/(1000*60*60*24),
...
switch(Order) {
	case 1: 
		EntryDelay = 4.8/1000;
		asset("ES"); enterLong();
		EntryDelay = 0.25/1000;
		asset("SPY"); enterShort();
		break;
		
	case 2: 
		EntryDelay = 4.8/1000;
		asset("ES"); enterShort();
		EntryDelay = 0.25/1000;
		asset("SPY"); enterLong();
		break;
		
	case 0:
		EntryDelay = 4.8/1000;
		asset("ES"); exitLong(); exitShort();
		EntryDelay = 0.25/1000;
		asset("SPY"); exitLong(); exitShort();
		break;
}

The first line simulates the price quote arrivals with 4.8 and 0.25 ms delay, the other lines establish the different order delays for SPY and ES. Alternatively, you could artificially delay the incoming SPY quotes by another 4.55 ms, so that their time stamps again match the ES quotes. In both cases the system returns about $240 per day, almost as much as in Warren. However a similar system located in Aurora, close to Chicago (swap the delays in the script), would produce zero profit. The asymmetry is caused by the relatively long constant periods of ES, making the SPY latency more relevant for the money at the end of the day.

30 thoughts on “Hacking a HFT system”

  1. Very nice idea.

    I wonder if it would be possible to use this kind of system to trade the arbitrage between different Bitcoin marketplaces and the connected other digital currencies. So far I know they bid a higher range of spreads between the different marketplaces and the different currencies in exchange to each other or in an normal Currency

  2. Yes, and this could be much more promising since cryptocurrency trading is still at the pioneer stage and there are probably not many HFT systems trading bitcoins, if any at all. The problem is the infrastructure and reliability of bitcoin exchanges. You permanently hear that one or the other exchange is going down or does not pay out anymore.

  3. U cant arbitrage bitcoin, there are too hight fee on exchange, look at blackbird, could do like 2% month, but now not even 0.5%

  4. Hi,
    Great article on HFT. I have one questions though. Could you elaborate on why you use SpreadSPY/SpreadES to decide the magnitude of the deviation in order to enter or close a position?

    Thanks,
    Rich

  5. Because the price deviation must be greater than both ask-bid spreads for making profit. The ES spread is relatively constant, but SPY spread can be up to 6 cents, so it must be considered when opening a position. The spread is here split and checked on opening and on closing.

  6. Nice work jcl. By the way, is the zorro script able to get price data from 2 brokers for the arbitrage?

  7. Not this script, since it’s for backtesting only. But generally, broker arbitrage should be no problem, I would use two parallel running Zorros for that. But I have heard that broker arbitrage rarely works.

  8. This is a really nice article JCL. I enjoyed reading this immensely.

    I have a lot of experience is these methods and your article took me back to the early days of this trade.

    The main issue is the cost of the infrastructure to support this at a meaningful level. We used microwaves and lazers, which do not come cheap.

    Sub 8ns can make a difference. While CME has equidistant cabling there are some other tricks you can employ to gain edge.

    Great article.

  9. Btw there is a book called Flash Boys written by Michael L. Lewis which describes the most common ways of HFT. There was also said that you can’t just simply put your servers in the middle between two cities because the cable is not going straight – it goes zigzag through other cities so the best thing is to lay down your own cable.

  10. Hi,
    great tutorial there. Much appreciated 🙂

    How would I include the order fees for entering/exiting the position into the arbitrage calculations?

    Lets say I have a takerFee of 0.025%,
    can I just “increase” the spread like this for the system to work?
    (askSPY + (askSPY * takerFee)) – (bidES – (bidES * takerFee))

  11. I would not add it to the spread. Either sum the fees up in a separate Fee variable, or subtract them directly from the predefined “Equity” variable that contains the accumulated profit from the trades.

  12. you wouldn’t use floats in HFT code . Too slow . Use two integers in a struct to represent a price , fixed precision . Additionally everyone is using microwave now for index arb between NY4 and Aurora.

  13. HFT code today normally uses floats. Integer multiplication needs just about 1..4 clock cycles less than FPU multiplication, and since you have overhead for shifting etc, floats are often actually faster. Lite-C has a 22.10 fixed point variable type, but it gives no speed advantage on modern CPUs. You use integer only when you need it for special purposes, for instance realizing your system in hardware.

  14. JCL – we do not use floating point of any type (doubles, floats) in HFT since it is very hard to compare. Fixed point is usually the choice.

    Good piece. Some thoughts:

    (1) your SPY data was likely captured and timestamp’ed in NYC while the ES data was timestamp’ed in Aurora/Chicago.

    (2) most commercial firms that collect tick data for resale have really bad timestamps, which drift even milliseconds intraday.

    (3) Most companies (eg Reuters) capture all data in just one location (eg NYC) even if the exchange is in Chicago – and they use the cheapest network – that would be very favorable in a backtesting

    (4) ES futures have a discount in relation to spot SPY depending on the amount of days until expiration that you are not taking in account;

    (5) You are crossing the market and paying the spread every time you trade – you are not accounting for it; Every time you sell a stock or future you have to use the BID/lower price and vice versa.

    (5.a) You might mitigate the spread-crossing by using a market making simulator that will take in account market volume and queue position but then you will have to code also a more flexible position management. In this case the trading logic will be more complex too.

    (6) you are not accounting for fees and commissions.

    (7) VHDL for FPGA is still software and most RTL development is done in Verilog in my experience, not VHDL.

    (8) FPGAs nowadays have a roundtrip of 200 nanos, tick-to-trade, including trading logic.

    (9) you are not checking if the book spread is wide – this happens frequently and throws off any EMA.

  15. FP compare problems should play no role since you normally compare floats on above or below, not on equality. I can imagine the use of fixed point in some special cases, such as counting discrete price steps, or when your system runs in a FPGA.

    Timestamps from data collecting firms are not used in HFT testing. You’re using exchange timestamps. Expiration discount is not very relevant when your trades last milliseconds. Market spreads however are relevant, and are of course accounted for as you see in the scripts. If I would disregard market spreads, the script would make ten thousands of dollars every day, not a few hundred. Fees are not included as already mentioned. They normally depend on your infrastructure and not on the HFT algorithm. VHDL and Verilog are both languages for defining hardware components. We use VHDL, but that’s a matter of personal preference.

  16. Jcl,
    Thanks for all the info, it is very educational!
    I have just one question I can’t help but wonder about:
    As someone who can charge EUR170 for changing just one line of code sometimes, writing this extensive public blog must be worth a fortune in your time. Are you paid to do it? (Since you mentioned that you’d be ready to make youtube lectures for pay, I assume you’re paid for the blog). And more importantly, what is the agenda of the entity who pays you? What do they gain from it? …unless it`s a secret of course.

  17. Just as most bloggers, I’m doing this in my free time, so there’s no sinister entity that pays me. And although we theoretically indeed must charge EUR 170 for changing a single line of code, in reality this doesn’t happen. In such a case we advise the client how to change that line by himself.

  18. Oh I see, but in that case you might want to consider making a youtube video blog after all. The video medium is arguably more interactive and engaging. It would greatly enhance your altruistic effort of communicating your experience and ideas to the general public. I know I would watch your videos for sure.

  19. Hi JCL,
    Thanks for all your hard work!
    I am currently reading your book, halfway through and feel like I’m learning a lot.
    However, there is one statement at the beginning (or at the “begin” as you like to say :] ) that got me confused: You said there that “higher leverage does not increase risk, and in fact the higher the leverage the further we are from a margin call”.
    I just cant wrap my head around that statement. I always thought that leverage multiplies (amplifies) the volatility of an asset in relation to our margin. Let’s say a hypothetical un-leveraged price movement in some position of ours that goes against us claims 10% of our margin – then we’d only be wiped out by 10 consecutive movements against us (our stop loss would have time to react). However, leveraging e.g. 10x times would in the same situation make our whole margin disappear within that one price movement instead of ten. Which means risk is indeed amplified and we are much more exposed to the margin call danger from simple random fluctuations. Am I getting something wrong?

  20. I probably should have said: Higher leverage does not increase risk with the same trade volume.

    If you trade a volume X, you get a margin call when your loss exceeds your capital minus allocated margin, which is X/Leverage. You see that the higher the leverage, the bigger is the term (Capital-X/Leverage) and thus the bigger is the maximum loss that your account can survive.

    Higher leverage does in fact increase the risk when it tempts you to trade a higher volume.

  21. Ohha! So, we are actually always buying the same amount of asset, except when we leverage we have to commit less margin to back the transaction. Which in turn means we have less margin at risk! That make sense, thank you for the explanation.

  22. So I finished reading the “Black Book of Financial Hacking” and went over to study the Zorro manual – and what do I see? The book and the manual have 90% overlap!! And the manual actually goes far more in depth. Consensus for the reader: Don’t waste your money on the book, just read the manual.

    JCL, as the author of the book and the Zorro manual would you agree with this observation?
    Considering that the Zorro project is sponsored you already must have earned good money from working on the manual. And so, would you testify to be practicing a shameless cash grab by reselling a shortened version of the manual as your book? xD hahahaha

  23. I bought the book too. I think the book is very useful in explaining the fundamental concepts more than manual. I did not regret buying the book. I would still recommend it to beginners.

  24. Just to clarify, even though I wrote both the book and the manual, they are completely different. One is an introduction in algo trading and the other is a software manual that assumes that you know algo trading already. The only overlap is the programming course, but for obvious reasons a software manual can not provide as many details and examples as a book. The course in the manual has 7 pages and the course in the book maybe 100 pages.

  25. I dunno, from what I’ve seen even the motivating conversations between Alice and Bob for each exercise are exactly the same… But maybe that is just my impression. ^^

  26. jcl, off-topic, but I see that you will be presenting at Quant Expo. I will be unable to attend. Will you be recording the presentation by any chance?

Leave a Reply

Your email address will not be published. Required fields are marked *