Bye Yahoo, and thanks for all the fish

Just a quick post in the light of a very recent event. Users of financial functions of R, MatLab, Python, or Zorro got a bad surprise in the last days. Scripts and programs based on historical price data suddenly didn’t work anymore. And our favorite free historical price data provider, Yahoo, now responds on any access to their API in this way:

No, they won’t be right back. Their engineers went fishing and, from what I hear, won’t be working on the issue anytime soon. The Yahoo Finance API is dead. Without prior announcement, Yahoo has abandoned their only remaining service that was clearly ahead of the competition.

Many scripts that I’ve posted on this blog use Yahoo for downloading price data. So here’s a quick workaround. Before running the Yahoo based script, run this one:

void main()
{
	string MyAsset;
	while(MyAsset = loop("SPY","TNT","XIV","GLD","AAPL","YHOO")) // ... enter your assets here 
	{
		string URL = strf("https://finance.google.com/finance/historical?q=%s&startdate=01-Jan-2000&output=csv",MyAsset);
		string Content = http_transfer(URL,0);
		if(!Content) continue;
		file_write("History\\history.csv",Content,0);
		dataNew(1,0,7);
		if(!dataParse(1,"%d-%b-%y,f3,f1,f2,f4,f6","History\\history.csv"))
			continue;
		printf(" %s",MyAsset);
		dataSave(1,strf("History\\%s.t6",MyAsset));
	}
}

This script downloads the data with the Google API. When you afterwards start the script with the Yahoo download, it will detect that the data is already there, and not access the Yahoo API.

Why haven’t I used the Google API already in the first place? Their data quality is not as good (many gaps and outliers), and it’s not dividend adjusted. So the results will be a bit different than results with Yahoo data. And then there’s also this ominous message:

Like the Yahoo message, this one is also wrong, but in a more positive sense. The Google Finance API is still available and was so for years, but nobody knows how long it will remain. For the moment, the above script will do, but it’s only a temporary solution. Free stock data can alternatively be downloaded from Quandl with a similar script, but for ETFs they want 50 bucks per month (I hope they didn’t bribe Yahoo and Google into abandoning their APIs!).

I’ve added the above script to the 2017 repository, in order to make all the Yahoo accessing scripts work again, although with slightly different results. You’ll need the current Zorro version 1.58 or above. And if you know a free and reliable API for ETF EOD history, please post a comment!


Addendum (June 2017): Readers mentioned AlphaVantage (www.alphavantage.co) and Stooq (www.stooq.com) as free alternatives to Yahoo. And indeed, they provide also dividend and split adjusted data. From what I’ve seen, their data has better quality than Google’s. AlphaVantage also includes the unadjusted prices. Only problem that I’ve found is that some price histories don’t go further back than 5 years. Downloading price data from AlphaVantage, Stooq, and some other sources was implemented in the current Zorro version. A list of finance API access points for EOD data:

Google Finance https://finance.google.com/finance/historical?q=AAPL&
startdate=01-Jan-2010&output=csv
Quandl https://www.quandl.com/api/v3/datasets/WIKI/AAPL.csv?
start_date=2010-01-01&api_key=...
Stooq https://stooq.com/q/d/l/?s=AAPL.US&i=d
Alpha Vantage https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED
&symbol=AAPL&outputsize=full&apikey=...&datatype=csv
BarChart https://marketdata.websol.barchart.com/getHistory.csv?apikey=...&symbol=AAPL&
startDate=20100101&splits=true&dividends=true&volume=sum&nearby=1&jerq=true
IEX https://api.iextrading.com/1.0/stock/aapl/chart/5y?format=csv

 

135 thoughts on “Bye Yahoo, and thanks for all the fish”

  1. Google historical data API has shut down now too. I needed full historical EOD data (which Google provided before) but most API’s may only go back 10 or 20 years.

    The best solution for me was https://www.worldtradingdata.com.
    Most tickers go back to near their IPO (AAPL for example goes back all the way to 1980-12-12).

    Hope this can help some others too.

  2. I have found no “Free” alternative to what Yahoo was. AlphaVantage has gotten worse by the week. Some days/TOD it’s fine for most calls, others it’s not worth connecting. Response time does not perform has they indicate most of the time. Perhaps it’s growing pains, but would be a lot more reassuring if they were a lot more forthcoming about themselves.

    They all have much more limited datasets (markets/symbols/fields) when they do work.

  3. Thanks to your work on this blog, I found my own solution. I blogged about it here, but I’ve included 3 links just in case any go down.

    http://thistleknot.sytes.net/wordpress/sp500-download-via-cookie/

    I wrote my first script that deals with working with cookies. The sp500 is used as a based in portfolio management, but there are no direct csv links. They all redirect using some sort of cookie. I read about extracting a crumb from some other urls, but they were all in different languages. I’m on windows and I really didn’t want to branch out of batch. GNU tools are one thing, as that’s whats on linux. Anyways, lucky for me, other sripters think like me and they utilize gnu as well. So I figured how to compartmentalize the commands by making them into mini batch files, and I was able to extract the cookie and crumb and download the SP500 so I can use it as a index for portfolio management.

    https://github.com/thistleknot/alphaAdvantagePostGRESQL/blob/master/dlSP500.bat

    useful url’s:

    https://github.com/bradlucas/get-yahoo-quotes/blob/master/get-yahoo-quotes.sh

    http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/

    http://www.javaear.com/question/44030983.html

  4. Anyone knows how to use Stooq or IEX to download data for Indian stocks? Like INFY.NS PNB.(all symbols from yahoo) NS, NSEI NSEBANK etc?

  5. As you know, Alpha Vantage has adopted a commercial license scheme, quite expensive in my view. Their free version officially delivers only up to 5 quotes per minute, although my testing shows that api calls are often rejected even when they are placed less frequently. In general, feed acquisition is not stable and unexplained erratic intermittent failures occur.
    I have upgraded my Excel Deriscope addin to cope with these “anomalies” by placing feed requests on spaced 20 seconds intervals and filtering out any received bad values. All-in-all Deriscope allows you to have – let’s say – 30 stock tickers and have a constant stream of price quotes (received on 20 sec intervals) that will be no older than 10 minutes (30 tickers * 20 seconds = 600 seconds = 10 minutes).
    You may take a look at my youtube video https://youtu.be/loNLupE0qnQ or visit https://www.deriscope.com

  6. Dear everybody:
    I appreciate suggestions, preferable free, where I can mix data from stock information and from options information.
    I find stock and options but not a place where I can have them both
    I would like to have like closing price and daily volume for stocks and also have how much volume the stock received in the options market and open interest.
    I’d like to download historic prices for like 100 stocks
    Thank you
    Javier

  7. I have been using Deriscope to get live stock prices in Excel.
    It works with various live feed providers, including Alpha Vantage.
    I like in particular Deriscope’s ability to fetch stock prices in the background at a rate that is allowed by Alpha Vantage free usage restriction of 5 api calls/minute. I use Deriscope’s default setting of 3 calls/minute so that no bottleneck ever occurs.
    My monitored portfolio consists of 18 non-US stocks, which means I get a full price update every 6 minutes. This is good enough for my purposes.
    With my US stocks I use the Deriscope-supported IEX live feed provider, who sends true real time prices with zero time delay.

  8. Disen,
    who said IEX does not provide intra-day open, close?
    It does and actually provide much more than that. A total of 20 fields, including the timestamps. Minute-by-minute!

    By the way, for an up-to-date live and historical feeds coverage by the free Deriscope Excel add-in, check out the link https://blog.deriscope.com/index.php/en/introduction-deriscope-5-excel-live-feeds

    The currently supported sources are Yahoo, Barchart, World Trading Data, Alpha Vantage, IEX and TrueFX

  9. My apologies for not finding this forum earlier, it would have made everybody’s life easier. Don’t be mad. Yahoo financial data is still available, in fact its much better than it ever was and it is accessible by everybody, you just need the right tools. Yahoo simply switched from CSV to JSON format, which is way more flexible for them. Over 150,000 stock symbols on 82 exchanges–that’s a lot of data.

    I have created some tools at http://www.signalsolver.com to help you with the transition. The first emulates the old Yahoo URL for getting historical stock price data and gets around the cookie issue. The second imports historical price data into Excel using the JSON interface, including one minute delayed prices. These are both free.

    The third is an Excel workbook with 17 worksheets of Yahoo financial data, imported via JSON. Over 800 types of financial data, including 15 minute delayed option prices, real time stock prices, balance sheets, fund information…just about anything you need. All using a single simple UDF. The data is free, the tool is a modest price. https://youtu.be/y5VADpqUkHU

    Forgive me for shamelessly promoting my own creations, but I thought this information may be add value to the discussion, and I don’t want people feeling badly about Yahoo when they are just keeping up with the technology. We need to do the same.

    Andrew MacLean
    Algorithm Science

  10. Hi, I have been with http://eodhistoricaldata.com/ for a few months now and I had extensive contact with their support people. This is the result:
    PROS:
    – They are cheap
    – They are kind of complete
    – Their history goes back sufficiently to do backtesting
    CONS:
    – They are amateurs. Their starter code is not theirs and it is full of bugs. You better know how to program.
    – Their help pages are not very helpful and they make a ton of assumptions which may or many not be true or may or may not be applicable or may or may not be understandable. Obviously these people never wrote a technical document, much less a User Requirement Specification / System Requirement / Design Specification, etc. (if you are into software engineering).
    – Some exchanges may “disappear” on you without notice
    – Their support is OK, but it will take you a few tries to communicate with them; their english is not great (this is a French company).
    – Their API is average, but, they do not have useful “bulk” download options. If you want to download entire exchanges from scratch, you will have to do it one day at the time.
    – They are not sure when EOD the data will be ready.
    – Data may not be available EOD for… no reason.

    BOTTOM LINE: if you want cheap data to develop systems and you need a lot of it, this is a good option. Pay for a month or two, download everything and then develop your systems. Once you start trading for real, get a reliable data provider.

  11. Stooq recently did the Yahoo and ceased delivering historical data on stooq.com. But stooq.pl seems to be still working. So if you got data from Stooq, change the URL to stooq.pl.

    I would not care, but Stooq seems to be today the only reliable free source for UK stocks and ETFs. Yahoo delivers them too, but unfortunately with totally wrong prices – their connection to the LSE seems to be seriously broken. This also affects data providers that just copy their data from Yahoo, such as AlphaVantage.

  12. Thanks for a marvelous posting! I really enjoyed reading it, you will be a great author.I will make certain to bookmark your
    blog and will eventually come back in the foreseeable future.
    I want to encourage you to continue your great job, have a nice weekend!

  13. https://github.com/JECSand/yahoofinancials

    Seems to be a great replacement for the old python yahoo-finance module. I started using yahoo financials a few weeks ago and it works great! Gets data relating to stock, etf, cryptocurrency, futures, and more. You are able to get financial statement data from this module as well. It’s all free and I’d recommend giving it a look.

  14. As you probably already know, IEX has changed part of its api to a different semi-commercial one.
    I would like to let you know that I have updated my Excel Addin Deriscope so that it still gets free real time IEX-traded US stock and ETFs last price, bid and offer quotes through the still functioning old api. It also gets historical data and company financial information through the new api, albeit this requires the entry in Excel of a previously IEX-acquired api code. As I explain at the link below, the api code can be obtained automatically free of charge if you register as a developer by IEX.
    Regarding the other comments about Yahoo delays wrt UK stocks, please keep in mind that Deriscope supports Yahoo, IEX, BarChart, World Trading Data, Alpha Vantage and True FX simultaneously. All actually provide a certain volume of data for free, whereas Yahoo, IEX and TrueFX place no restriction whatsoever. It is therefore likely that one of these providers delivers better timed data.
    https://blog.deriscope.com/index.php/en/introduction-deriscope-5-excel-live-feeds

  15. New kid on the block: http://finnhub.io/ . FREE real-time API with intraday data going back for years. You have 60 calls/minute with their free plan. Fundamental and alternative data are also available.

  16. Just for sharing the lastest information with Alpha Vantage, AlphaVantage does not support multiple symbol quotes any more. Hence, you have to loop over single quote access many times to achieve batch quote.

    For example, this request does not work any longer.

    https://www.alphavantage.co/query?function=BATCH_STOCK_QUOTES&apikey=xxx&symbols=MSFT,AAPL,FB

    Looping over single quote access for batch quote is just too much pain from your end because it is just drinking a lot of resource of your PC like hippo.

  17. Wield the power of Sentiment Analysis to better understand markets driven by psychology and behavioral finance. Alternative data provides the complement to fundamental StockMarket analysis, and now it’s available from us via API. Web/mobile coming soon: https://www.cityfalcon.com/blog/the-startup-journey/capital-emotions-track-sentiment-for-financial-content-globally/?utm_source=financial-hacker&utm_medium=referral&utm_campaign=ao_financial-hacker_sentiment

  18. Hey all,

    Investpy takes all data from investing.com. Even economic calendars etc. Its a python library and very simple to use.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.