Dear Brokers…

Whatever software we’re using for automated trading: We all need some broker connection for the algorithm to receive price quotes and place trades. Seemingly a simple task. And almost any broker supports it through a protocol such as FIX, through an automated platform such as MT4™, or through a specific broker API. But if you think you can quickly hook up your trading software to a broker API, you’re up for a bad surprise. Dear brokers – please read this post and try to make hacker’s and coder’s lifes a little easier!

A broker API allows software to trade, receive price quotes, and download price history. Those three functions are essential for an automated system. Good-to-have additional features are retrieving trade status, account status, and asset parameters. That’s six functions, or seven when you count login/logout. A broker API has often more than 100 functions. So you should assume that at least the 6 essential are more than covered. But sadly, it is not so. The debacle begins already with installing and starting up the API. 

API startup

Installing and starting a broker API can be so simple. The ideal case is a broker that provides no software at all, but just a protocol and a URL for a TCP/IP or UDP connection. You can then use well-established libraries such as curl or enet for connecting to the broker. In the second-best case you get an API DLL that you can redistribute to your clients. Your software then just calls the login function of that DLL, submits the authentification data and the connection is established. But for some unknown reason, many brokers feel obliged to provide connections to their APIs in much more complex and creative ways.

Broker A required any user to run an API installation package. After an installation dialogue with several Windows incompatibility warnings, the installer produced a bunch of DLLs. Then you can go through a lengthy procedure with many steps of building up all the internal tables and data structures that you need for connecting to your account. In a previous version, the installer also modified the Registry, so you could not step around it and distribute the DLLs directly to your clients together with your software. They had to go through the installation procedure themselves. This is fortunately not required anymore with the current version.

Broker B offers no direct connection at all. You have the choice of connecting either to their bloated Java trade platform, or to a small “Gateway” program. When you connect to the Java platform, your software must once per day log out, restart the platform, and log in again since the platform automatically shuts down every 24 hours. When you connect through the “Gateway”, you can not open their platform at the same time (not even from a different PC). So you can’t even check your positions. If you do, the Gateway connection breaks down and must be manually re-established. This is not a bug as you might think. It is intentionally implemented this way (according to the broker) as a service to their users.

Both broker A and broker B are major, world-wide operating Forex and stock brokers.  

Getting price quotes

Broker A keeps its asset prices in a “Quote Table” on his server. For getting a price quote you must first subscribe to that asset. For this you do not just call a function, you build up a “Request Factory” and generate a “Subscription Request”. Now you can enter the procedure of receiving a response on your request, which involves the creation of a “Response Listener”. Subscribing to an asset this way takes about 30 primary lines of C++ code – and this does not include the secondary code for sending the requests, generating listeners, and all the supplementary stuff. My clients can consider themselves fortunate that I don’t charge by code line. Getting the price itself then only requires looking up the asset in the server’s quote table.

Broker B makes it easy: You can not subscribe market quotes by API at all. You must do that manually on their website, and it is not free (except for Forex). For getting the price, a simple API call is sufficient. Or would be, if you knew the primary exchange, secondary exchange, asset type and sub-type, and other asset parameters. If any of them is not right, you get no price. And don’t think that these exchange and other parameters are documented somewhere. You’re supposed to find them out by trial and error, or to ask fellow programmers on the Internet.

Getting price history

Although any algorithm trades in real time, it still needs price history on startup for calculating initial values of its indicators and price-analyzing functions. With no access to price history, you had to wait a couple days before the first trade can be placed. Since this is not really practical, price history is an essential API function.

Broker A provides price history with no major problem. I can almost not believe it. Ok, you must again fire up the “Request Factory”, generate requests and listeners, but about 50 code lines enable you to download historical prices. The broker charges no fee for those prices (you can even download them with a demo account), and at least the recent data, from 2010 and above, is in acceptable quality. Eight out of ten points for the price history from broker A.

Broker B again got creative and enterprising. Their price history from demo accounts is worthless – you must indeed open a real account and deposit $10,000 when you want historical data. This sum, of course, is also required from a poor coder who only wants to implement a function for getting price history. Not that you get it then easily. Technically it’s simple – a few lines of code are enough – but you only have 60 requests. After that, the price history server is shut down for 10 minutes. 60 requests are enough for a single-asset system, but not for starting a portfolio system, and certainly not for backtests. Why this bizarre limit? Maybe broker B can’t afford a fast Internet connection for their price history server. Maybe they are using the CEO’s old PC, located in his living room, and the bandwidth goes down anytime when his kids play Tetris on it. Whatever the reason, that’s only one out of ten points for the price history from broker B.

Is there a broker with even zero points for the price history? Yes, there is: Broker C has no API function for retrieving historical prices at all. Apparently they had not expected that someone would really use their API. I had no choice but to advise my client to choose another broker for his algorithmic system.

Handling trades

We programmers tend to think binary. When we send a fill-or-kill order to the broker API, we assume that the position will be either opened (“yes”), or it will not (“no”). But from time to time, broker APIs produce a third result, like “maybe” or “I won’t tell”. A position that is maybe opened or maybe not can result in an orphaned trade. This is a trade opened by the broker, but not reported back to the trading software, and consequently not under automated control. This trade won’t be closed by a reversal or stop and can thus accumulate a large loss. It can blow the account. Orphaned trades are not a good thing.

Theoretically, they would be so easy to prevent. The API just needs to provide a simple order function with an identifier and a validity limit, like this:

openAtMarket(string Asset, int Amount, int ID, int SecondsValid);

ID = dear broker, please store this identifier together with the trade and use it for retrieving the trade status in case of doubt.
SecondsValid = dear broker, please disregard the order when after the given number of seconds your API still doesn’t know if it can fill it or not, or is unable to confirm it for whatever reason. 

That’s the ideal, but how’s the reality?

Broker A accepts a ton of parameters in his order functions, but no user-supplied trade ID and no SecondsValid limit. After sending an order and going through the usual “Request Factory” and “Response Listener” rigmarole, you’ll receive a confirmation and can retrieve the trade identifier for managing or closing it later. Or you’ll get notified that the order failed. Or neither of both. It happens that the API does not react at all on your order, maybe due to a server hickup, an Internet outage, or because it became confused by too many Factories and Listeners. Then you can only guess if your order made it to the server or not. If it did, you have an orphaned trade. But at least you can relatively easily identify and close it manually.

Broker B provides a simple order function, even with a user-supplied ID. And you get confirmation by a simple callback function that the order was filled or not. Usually not, since broker B needs again lots of information about the asset and the order routing, and won’t execute the order if something is missing or not fitting together. But at least you know the ID. This makes you think that you can later check the trade status. Think again: Broker B won’t reveal any information about your trades. At least not to you. In fact they do not store your trades at all. They only store net positions. So there is no way to identify orphans. Not even manually, since you can’t open the trade platform as long as an automated system is running.

Account parameters and trading costs 

Broker A provides a function where you can request leverage, pip size, pip cost, rollover, and any other parameters of the selected asset. Enough information to calculate margin and trading costs in advance, what is a good thing for an automated system – if there weren’t an important parameter missing. The API does not allow to retrieve the commission. You must enter it manually and store it per asset. Why this important parameter is not provided by the API remains broker A’s secret.

Broker B does not have this problem, as its API provides no asset and trading cost information at all – not even the leverage. For compensation, it supports a sort of “virtual order” that can be used to calculate profit, similar to Zorro’s “phantom trades”. Theoretically you could use that to calculate some of the asset parameters in a complex way from a series of virtual orders with different lot sizes. My clients didn’t pay me for that. For simulating a broker B account in a backtest, you must therefore enter all required asset parameters and trading costs manually in a spreadsheet – which is no easy task due to the complicated fee and margin structure of that broker.

---

There are APIs for almost all imaginable software tasks. And most are well structured nowadays and relatively easy to use. Why is it not so with most broker APIs? Dear brokers: If you think about providing API access to your customers, please look first into other broker’s solutions for learning how NOT to do it. And if really necessary, please add detailed documentation about which of the market or order parameters are really required and in which combination.

And if you then still are not sure how to implement your new API, just hire me. I had the dubious pleasure to implement dozens of APIs so far, and can show you how an API for algo trading should look like. And even though you’ll then have to hand over some shekels to my employer, you’ll make up for this by winning an excellent reputation. At least among the poor coders and hackers who have to implement your API.

---

Addendum (January 2017): For the sake of fairness it should be mentioned that broker B has meanwhile removed the bizarre 60 requests limit for historical data. One year of data is now available. Maybe they have read this blog. Or their CEO got a new PC for Christmas. Whatever the reason, it’s still not sufficient for backtests, but at least for live trading even large portfolio strategies.