Bye Yahoo, and thanks for all the fish

Just a quick post in the light of a very recent event. Users of financial functions of R, MatLab, Python, or Zorro got a bad surprise in the last days. Scripts and programs based on historical price data suddenly didn’t work anymore. And our favorite free historical price data provider, Yahoo, now responds on any access to their API in this way:

No, they won’t be right back. Their engineers went fishing and, from what I hear, won’t be working on the issue anytime soon. The Yahoo Finance API is dead. Without prior announcement, Yahoo has abandoned their only remaining service that was clearly ahead of the competition.

Many scripts that I’ve posted on this blog use Yahoo for downloading price data. So here’s a quick workaround. Before running the Yahoo based script, run this one:

void main()
{
	string MyAsset;
	while(MyAsset = loop("SPY","TNT","XIV","GLD","AAPL","YHOO")) // ... enter your assets here 
	{
		string URL = strf("https://finance.google.com/finance/historical?q=%s&startdate=01-Jan-2000&output=csv",MyAsset);
		string Content = http_transfer(URL,0);
		if(!Content) continue;
		file_write("History\\history.csv",Content,0);
		dataNew(1,0,7);
		if(!dataParse(1,"%d-%b-%y,f3,f1,f2,f4,f6","History\\history.csv"))
			continue;
		printf(" %s",MyAsset);
		dataSave(1,strf("History\\%s.t6",MyAsset));
	}
}

This script downloads the data with the Google API. When you afterwards start the script with the Yahoo download, it will detect that the data is already there, and not access the Yahoo API.

Why haven’t I used the Google API already in the first place? Their data quality is not as good (many gaps and outliers), and it’s not dividend adjusted. So the results will be a bit different than results with Yahoo data. And then there’s also this ominous message:

Like the Yahoo message, this one is also wrong, but in a more positive sense. The Google Finance API is still available and was so for years, but nobody knows how long it will remain. For the moment, the above script will do, but it’s only a temporary solution. Free stock data can alternatively be downloaded from Quandl with a similar script, but for ETFs they want 50 bucks per month (I hope they didn’t bribe Yahoo and Google into abandoning their APIs!).

I’ve added the above script to the 2017 repository, in order to make all the Yahoo accessing scripts work again, although with slightly different results. You’ll need the current Zorro version 1.58 or above. And if you know a free and reliable API for ETF EOD history, please post a comment!


Addendum (June 2017): Readers mentioned AlphaVantage (www.alphavantage.co) and Stooq (www.stooq.com) as free alternatives to Yahoo. And indeed, they provide also dividend and split adjusted data. From what I’ve seen, their data has better quality than Google’s. AlphaVantage also includes the unadjusted prices. Only problem that I’ve found is that some price histories don’t go further back than 5 years. Downloading price data from AlphaVantage, Stooq, and some other sources was implemented in the current Zorro version. A list of finance API access points for EOD data:

Google Finance https://finance.google.com/finance/historical?q=AAPL&
startdate=01-Jan-2010&output=csv
Quandl https://www.quandl.com/api/v3/datasets/WIKI/AAPL.csv?
start_date=2010-01-01&api_key=...
Stooq https://stooq.com/q/d/l/?s=AAPL.US&i=d
Alpha Vantage https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED
&symbol=AAPL&outputsize=full&apikey=...&datatype=csv
BarChart https://marketdata.websol.barchart.com/getHistory.csv?apikey=...&symbol=AAPL&
startDate=20100101&splits=true&dividends=true&volume=sum&nearby=1&jerq=true
IEX https://api.iextrading.com/1.0/stock/aapl/chart/5y?format=csv

 

107 thoughts on “Bye Yahoo, and thanks for all the fish”

  1. Google historical data API has shut down now too. I needed full historical EOD data (which Google provided before) but most API’s may only go back 10 or 20 years.

    The best solution for me was https://www.worldtradingdata.com.
    Most tickers go back to near their IPO (AAPL for example goes back all the way to 1980-12-12).

    Hope this can help some others too.

  2. I have found no “Free” alternative to what Yahoo was. AlphaVantage has gotten worse by the week. Some days/TOD it’s fine for most calls, others it’s not worth connecting. Response time does not perform has they indicate most of the time. Perhaps it’s growing pains, but would be a lot more reassuring if they were a lot more forthcoming about themselves.

    They all have much more limited datasets (markets/symbols/fields) when they do work.

  3. Thanks to your work on this blog, I found my own solution. I blogged about it here, but I’ve included 3 links just in case any go down.

    http://thistleknot.sytes.net/wordpress/sp500-download-via-cookie/

    I wrote my first script that deals with working with cookies. The sp500 is used as a based in portfolio management, but there are no direct csv links. They all redirect using some sort of cookie. I read about extracting a crumb from some other urls, but they were all in different languages. I’m on windows and I really didn’t want to branch out of batch. GNU tools are one thing, as that’s whats on linux. Anyways, lucky for me, other sripters think like me and they utilize gnu as well. So I figured how to compartmentalize the commands by making them into mini batch files, and I was able to extract the cookie and crumb and download the SP500 so I can use it as a index for portfolio management.

    https://github.com/thistleknot/alphaAdvantagePostGRESQL/blob/master/dlSP500.bat

    useful url’s:

    https://github.com/bradlucas/get-yahoo-quotes/blob/master/get-yahoo-quotes.sh

    http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/

    http://www.javaear.com/question/44030983.html

  4. Anyone knows how to use Stooq or IEX to download data for Indian stocks? Like INFY.NS PNB.(all symbols from yahoo) NS, NSEI NSEBANK etc?

  5. As you know, Alpha Vantage has adopted a commercial license scheme, quite expensive in my view. Their free version officially delivers only up to 5 quotes per minute, although my testing shows that api calls are often rejected even when they are placed less frequently. In general, feed acquisition is not stable and unexplained erratic intermittent failures occur.
    I have upgraded my Excel Deriscope addin to cope with these “anomalies” by placing feed requests on spaced 20 seconds intervals and filtering out any received bad values. All-in-all Deriscope allows you to have – let’s say – 30 stock tickers and have a constant stream of price quotes (received on 20 sec intervals) that will be no older than 10 minutes (30 tickers * 20 seconds = 600 seconds = 10 minutes).
    You may take a look at my youtube video https://youtu.be/loNLupE0qnQ or visit https://www.deriscope.com

  6. Dear everybody:
    I appreciate suggestions, preferable free, where I can mix data from stock information and from options information.
    I find stock and options but not a place where I can have them both
    I would like to have like closing price and daily volume for stocks and also have how much volume the stock received in the options market and open interest.
    I’d like to download historic prices for like 100 stocks
    Thank you
    Javier

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.