The Market Meanness Index

This indicator can improve – sometimes even double – the profit expectancy of trend following systems. The Market Meanness Index tells whether the market is currently moving in or out of a “trending” regime. It can this way prevent losses by false signals of trend indicators. It is a purely statistical algorithm and not based on volatility, trends, or cycles of the price curve.

There are already several methods for differentiating trending and nontrending market regimes. Some of them are rumored to really work, at least occasionally. John Ehlers proposed the Hilbert Transform or a Cycle / Trend decomposition, Benoit Mandelbrot the Hurst Exponent. In comparison, the source code of the Market Meanness Index is relatively simple:

// Market Meanness Index
double MMI(double *Data,int Length)
{
  double m = Median(Data,Length);
  int i, nh=0, nl=0;
  for(i=1; i<Length; i++) {
    if(Data[i] > m && Data[i] > Data[i-1]) // mind Data order: Data[0] is newest!
      nl++;
    else if(Data[i] < m && Data[i] < Data[i-1])
      nh++;
  }
  return 100.*(nl+nh)/(Length-1);
}

This code is in C for Zorro, but there’s also a MQL4 MMI version that someone on Steve Hopwood’s forum has programmed (see blogroll). As the name suggests, the indicator measures the meanness of the market – its tendency to revert to the mean after pretending to start a trend. If that happens too often, all trend following systems will bite the dust.

The Three-Quarter Rule

A series of random numbers reverts to the mean – or more precisely, to the median – with a probability of 75%. So when you look at a random price sequence, if yesterday’s price was above the median, in 75% of all cases today’s price is lower than yesterday’s. And if the yesterday’s price was below the median, 75% chance is that today’s price is higher. The proof of the 75% is relatively simple and won’t require integral calculus. Consider a price curve with median M. By definition, half the prices are less than M and half are greater (for simplicity’s sake we’re ignoring the case when a price is exactly M). Now combine the prices of the curve to pairs each consisting of a price Py and the following price Pt. Thus each pair represents a price change from Py to Pt. We now got a lot of price changes that we divide into four sets:

  1. (Pt < M, Py < M)
  2. (Pt < M, Py > M)
  3. (Pt > M, Py < M)
  4. (Pt > M, Py > M)

These four sets have obviously the same number of elements – that is, 1/4 of all Py->Pt price changes – when Pt and Py are not correlated, i.e. completely independent of one another. The value of M and the shape of the price curve won’t matter for this. Now how many price pairs revert to the median? All pairs that fulfill this condition: (Py < M and Pt > Py) or (Py > M and Pt < Py) The condition in the first bracket is fulfilled for half the prices in set 1 (in the other half is Pt less than Py) and in the whole set 3 (because Pt is always higher than Py in set 3). So the first bracket is true for 1/2 * 1/4 + 1/4 = 3/8 of all price changes. Likewise,  the second bracket is true in half the set 4 and in the whole set 2, thus also for 3/8 of all price changes. 3/8 + 3/8 yields 6/8, i.e. 75%. This is the three-quarter rule for the differences of random numbers.

The MMI function just counts the number of data differences for which the conditition is met, and returns their percentage. The Data series may contain prices or price changes. Prices have always some serial correlation: If EUR / USD today is at 1.20, it will also be tomorrow around 1.20. That it will end up tomorrow at 70 cents or 2 dollars per EUR is rather unlikely. This serial correlation is also true for a price series calculated from random numbers, as not the prices themselves are random, but their changes. Thus, the MMI function should return a smaller percentage, such as 55%, when fed with prices.

 Unlike prices, price changes have not necessarily serial correlation. A one hundred percent efficient market has no correlation between the price change from yesterday to today and the price change from today to tomorrow. If the MMI function is fed with perfectly random price changes from a perfectly efficient market, it will return a value of about 75%. The less efficient and the more trending the market becomes, the more the MMI decreases. Thus a falling MMI is a indicator of an upcoming trend. A rising MMI hints that the market will get nastier, at least for trend trading systems.

Using the MMI in a trend strategy

One could assume that MMI predicts the price direction. A high MMI value indicates a high chance of mean reversion, so when prices were moving up in the last time and MMI is high, can we expect a soon price drop? Unfortunately it doesn’t work this way. The probability of mean reversion is not evenly distributed over the Length of the Data interval. For the early prices it is high (since the median is computed from future prices), but for the late prices, at the very time when MMI is calculated, it is down to just 50%. Predicting the next price with the MMI would work as well as flipping a coin.

Another mistake would be using the MMI for detecting a cyclic or mean-reverting market regime. True, the MMI will rise in such a situation, but it will also rise when the market becomes more random and more effective. A rising MMI alone is no promise of profit by cycle trading systems.

So the MMI won’t tell us the next price, and it won’t tell us if the market is mean reverting or just plain mean, but it can reveal information about the success chance of trend following. For this we’re making an assumption: Trend itself is trending. The market does not jump in and out of trend mode suddenly, but with some inertia. Thus, when we know that MMI is rising, we assume that the market is becoming more efficient, more random, more cyclic, more reversing or whatever, but in any case bad for trend trading. However when MMI is falling, chances are good that the next beginning trend will last longer than normal.

This way the MMI can be an excellent trend filter – in theory. But we all know that there’s often a large gap between theory and practice, especially in algorithmic trading. So I’m now going to test what the Market Meanness Index does to the collection of the 900 trend following systems that I’ve accumulated. For a first quick test, this was the equity curve of one of the systems, TrendEMA, without MMI (44% average annual return):

EMA strategy without MMI

This is the same system with MMI (55% average annual return):

EMA strategy with MMI

We can see that the profit has doubled, from $250 to $500. The profit factor climbed from 1.2 to 1.8, and the number of trades (green and red lines) is noticeable reduced. On the other hand, the equity curve started with a drawdown that wasn’t there with the original system. So MMI obviously does not improve all trades. And this was just a randomly selected system. If our assumption about trend trendiness is true, the indicator should have a significant effect also on the other 899 systems.

This experiment will be the topic of the next article, in about a week. As usually I’ll include all the source code for anyone to reproduce it. Will the MMI miserably fail? Or improve only a few systems, but worsen others? Or will it light up the way to the Holy Grail of trend strategies? Let the market be the judge.

Next: Testing the Market Meanness Index

45 thoughts on “The Market Meanness Index”

  1. There are some links in the page which does not work. As “next page” in some cases.
    There are some indicators which try to find trend more. Did you consider to do an analysis of all of them as you did with the trend indicators. I mean instead of just using MMI to the 900 systems, you could use another ones by using the same approach.

  2. I can certainly test other trend indicators, as this requires only a small change of the script. But I don’t know many that promise to really work. Which indicator do you have in mind?

  3. I dont have any really in mind. You mention above Hurst and Hilbert Transform and so on. I like the methodology you use to find out which trend indicator is actually better. I was wondering what would be the result if you use the same methodology to find out if the market is in trend mode by using different indicators and not just MMI.

  4. Yes, I’ll test more filters and also the more conventional crossover rather than peaks and valleys for trend systems. The MMI already produces pretty good results, but it can not harm to test as many algorithms as possible. However the next planned experiment is with a “deep learning” network.

  5. Well congratulations again for this amazing blog. Really inspiring as well.
    I wonder if you can keep aplying this kind of approach in the next steps of strategy development like one can lock the best filter found so far and use instead as a variable input different kind of money management functions for example.

  6. Um, isnt there lookahead bias here? the median is computed over the entire data set. Not sure it can be used at any point in the data set to make a decision, as is.

  7. Ah, you really looked in the code! If the median was computed over the entire data set, it would indeed be lookahead bias. But ‘Length’ in the code is only the length of the MMI period. You can not look ahead with Zorro, except when you set a certain flag, otherwise any access of future data produces a warning message.

  8. Hey! I’m enjoying the blog

    There seems to be an unnecessary restriction on your test for mean reversion vs trend. Why must the price be beyond the median before measuring which direction it goes? “Mean reversion” usually is referring to a more general ‘where prices really ought to be’ rather than the mathematical mean (or in your case, median). Wouldn’t a more appropriate trend-vs-mean-reversion system simply look at auto-correlation? If a price moves one direction and then back the other way, regardless of its relative position to the median, then it would be considered “reverting”. This doesn’t invalidate your approach, but insisting that noise is mean-reverting 75% of the time isn’t describing the reality accurately. Really you’re saying “noise moving away from the median tends to move back toward the median 75% of the time.” Or did I misunderstand?

    Thanks!

  9. I think you understood it correctly – your formulation is indeed more accurate. The MMI is not really intended for distinguishing between mean reversion and trend. For this f.i. the Hurst Exponent produces better results, at least according to my experiences. The MMI works however well for determining just the presence or absence of trend. As soon as it begins to return high values in the 75% area, it can be the begin of mean reversion or it can be just randomness – the MMI can not distinguish that. Both situations are likewise bad for trend following.

  10. Based on your post I have programmed and tested MMI on Quantopian. I choose different stocks (SPY, BAC, AAPL, NFLX, etc) daily close price changes and different MMI lengths (100,200,300,500) . The testing date range was from 01/01/2012 to 01/14/2016.

    The result was always around 75% There was no value under 70% and above 85% in any case.
    For example:
    SPY
    from 01/01/2012 to 01/14/2016
    MMI length 300
    Average: 76.092%
    Standard deviation: 1.716
    Max:80.27%
    Min:71.91%

    Is this a good result?
    It seems to me that MMI is not usable or just this small changes under 75% what I should follow?

    If I used prices instead of price changes, the result was always around 50%, the standard deviation was higher, but not so convincing.

  11. Yes, this is a good result when you tested daily returns, which are normally not significantly trending. Even a bit mean reverting before mid-2014. You should get such a curve:
    S&P500 MMI Day

    But as soon as you look into hour returns, you’ll see more trendiness:
    S&P500 MMI Hour

    The script:

    void run()
    {
    	StartDate = 20120101;
    	EndDate = 2016;
    	plot("MMI", MMI(series(priceClose(0)-priceClose(1)),300), NEW, RED);
    }
  12. Thank you for your quick answer.

    So all trend following systems based on daily close prices are doomed.
    In case of 1H or forex my hypothesis is that these continuous, connected data in time and so more trendiness. Between daily close prices there are a big gap in time after the market closes and next day opens.
    So maybe I should trade on shorter time scale or on forex.

  13. “So when you look at a random price sequence, if yesterday’s price was above the median, in 75% of all cases today’s price is lower than yesterday”

    In a random price sequence, the chance of going higher/lower at each point is *always* 50% – irrespective of where the mean lies or what happened yesterday.

  14. This is indeed a frequent misconception. The chance of going higher/lower at a certain point is very different to the chance of a price pair in a random sequence to revert to the median. If you do not believe the proof above, just run the script and look at the results. And if you do not believe the script either, roll the dice a hundred times, write down all the numbers, and count how often subsequent numbers revert to the median. 🙂

  15. “If you do not believe the proof above, just run the script and look at the results”

    So the snippet below is a Python transcription of the above run over a random walk sequence.
    The MMI is always 50% Why the difference?

    import random
    import numpy

    def random_walk(n):
    seq = [0]
    for i in range(n):
    if random.random() > 0.5:
    seq.append( seq[-1] + 1 )
    else:
    seq.append( seq[-1] – 1 )
    return seq

    def market_meanness_index(data):
    m = numpy.median(data)
    nh = nl = 0

    for i in range(1, len(data)):
    if data[i] > m and data[i] > data[i-1]:
    nl += 1
    elif data[i] < m and data[i] < data[i-1]:
    nh += 1

    return 100.0 * (nl+nh) / (len(data)-1)

    mmi = market_meanness_index(random_walk(10**6))
    print "{:.1f}%".format(mmi)

  16. Because your random walk is not a random sequence. A random walk has strong serial correlation. A correct random sequence would be:

    def random_walk(n):
    seq = [0]
    for i in range(n):
    seq.append(random.random())
    return seq

    The same goes for price data: prices are not random. They have serial correlation. But price changes are random, at least in a perfect efficient market.

  17. Hi, can you please explain the detail algorithm that how MMI helps in trend detection? Or some pseudo code. Though I found you check “falling” in the other blog, I don’t know how “falling” works.

    Thanks

  18. I think the algorithm is explained above under “The three-quarter rule”. If something is unclear, just ask. The MMI detects trend indirectly, by absence of mean reversion. “Falling” is a binary function that just determines if a data series is rising or falling.

  19. @Mathafarn and anyone who wants to have a look at MMI for Python, the python version shown is not the same as Zorros version.

    First of all np.median() isn’t the same as the median Zorro uses, second the for-loop is going the wrong way leading to different results.

    Here’s a fixed version:

    def market_meanness_index(data):
    data_sorted = sorted(data)

    if len(data) % 2:
    m = data_sorted[int(len(data) / 2)]
    else:
    m = (data_sorted[int(len(data_sorted) / 2)+1] + data_sorted[int(len(data_sorted) / 2)]) / 2.0

    nh = nl = 0

    for i in range(0, len(data)-1):
    if data[i] > m and data[i] > data[i+1]:
    nl += 1
    elif data[i] < m and data[i] < data[i+1]:
    nh += 1

    return 100.0 * (nl+nh) / (len(data)-1)

  20. Hi jcl,

    Great job!

    Several quick questions

    1. About the application of MMI, can we use percentile(MMI,25), if today’s MMI is lower than 25%, we say market is in trend, otherwise, not.
    2. Which ones are sound indicators for determining market regime?
    3. Some markets are easier to generate profit from than others, I call them more “tradable”, just like what you found in this experiment that SPY is the most noisy market while commodities are more trendy, any indicators on this?

    Thank you

    Jeff

  21. 1. Yes, a percentile threshold would certainly make sense.
    2. For instance the Hurst Exponent – a fellow blogger, Robot Wealth, has recently written an article about it.
    3. Theoretically, an indicator like Shannon Entropy could be used for determining more randomness or more tradeability in a market. This could be an interesting topic for research.

  22. The python example below tests a perfect trend and a perfect mean reversion data sequence. The MMI for both is 50%. Why is this? Thanks….

    import math
    import numpy

    def market_meanness_index(data):
    m = numpy.median(data)
    nh = nl = 0

    for i in range(1, len(data)):
    if data[i] > m and data[i] > data[i-1]:
    nl += 1
    elif data[i] < m and data[i] < data[i-1]:
    nh += 1

    return 100.0*(nl+nh)/(len(data)-1)

    # perfect trend: MMI 50%
    trend_data = numpy.arange(0.0, math.pi, 0.01)
    print market_meanness_index(trend_data)

    # perfect mean reversion: MMI 50%
    sin_data = map(lambda x: math.sin(x), trend_data)
    print market_meanness_index(sin_data)

  23. A flat trend will indeed produce about 50%, but a sine curve should give you a heavily fluctuating MMI in the 60% or 70% area, dependent on the period of the sine curve in relation to the MMI period.

    MMI Test

    function run()
    {
    MaxBars = 1500;
    LookBack = 100;
    asset(""); // dummy asset

    vars Data1 = series(genSine(5));
    plot("Sine",Data1,NEW,BLACK);
    plot("MMI_Sine",MMI(Data1,100),NEW,BLACK);

    vars Data2 = series(Bar*0.01);
    plot("Trend",Data2,NEW,BLACK);
    plot("MMI_Trend",MMI(Data2,100),NEW,BLACK);
    }

  24. Thanks for the excellent article. Would MMI be useful to prevent losses in am mean reverting strategy or which other indocator would you recommend? Thank you.

  25. No, the MMI will probably not work well for filtering mean reverting trades, since it makes no difference between “less trendy” and “more random”. For filtering mean reversion I would try to detect the dominant cycle – mean reversion is normally related to some short-term cyclic behavior – and check the amplitude of the dominant frequency component.

  26. Hi jcl! Do you happen to have the MetaTrader version of this? Every time I try to access the link you posted I get an error message.
    Also do you happen to know if anyone has coded this for Tradestation?

  27. It’s some time ago, but if I remember right someone posted a MT4 version on Steve Hopwood’s forum. There was also a Tradestation version, but I cannot remember the link anymore. It should be anyway relatively easy to port the code to Tradestation.

  28. I tried to translate the MMI into Easylanguage. But I always just get a steadily increasing line.
    nl and nh are always increasing values?

  29. I’m no Easylanguage expert, but AFAIK it has no local variables, anything is global. If so, then make sure that nl and nh are reset to zero at the begin of the function.

  30. If I reset nl and nh at each bar, I get a binary wave between 0 and 0.33.
    Do I have to loop the calculation? For how long?

  31. You should get about 0.75 when you test with random numbers. With 0.33, something is wrong. The indicator is not cumulative, so you need not loop it.

  32. if nl and nh get reset every bar and the indicator is not cumulative, the formula would always be
    100*1/299
    as nl or nh are always 1

    Could you maybe paraphrase the formula in pseudo-code, in case I misread the C-code?

  33. I don’t know if it helps, but anyway:

    count all price decreases above the median
    add all price increases below the median
    divide by the number of prices and return the percentage.

    Hacker’s first rule: Code only what you understand.

  34. >>jcl says:
    >>July 18, 2017 at 10:11
    >>It’s some time ago, but if I remember right someone posted a MT4 version on Steve Hopwood’s forum.
    >>There was also a Tradestation version, but I cannot remember the link anymore. It should be
    >>anyway relatively easy to port the code to Tradestation.

    The MT4 version was on this link, but it seems to be deleted:
    http://www.stevehopwoodforex.com/phpBB3/viewtopic.php?uid=17&f=27&t=3714&start=0
    Maybe someone still has the indicator for MT4 or Tradestation

  35. I have translated the indicator, but it looks quite different from your example
    http://imgur.com/a/WWD0O
    calculated on close of daily bars, length 300
    Seems like it’s high, when trending and low when mean reverting. Also highest value is 60%

  36. Then apparently the translation was still not a real success. You must get the same curve, and trending produces lower, not higher values.

  37. Yes. But make sure that you get the data order right. It’s a financial series, with the newest data at the begin. If the order is wrong, rising and falling is swapped.

  38. you mean it would be different if the counter of true conditions is increasing (from old to new data) from looking back from the current bar to the last 300 bars?

  39. Look at the code. Changing the data order is equivalent to swapping two of the four ‘>’ and ‘<' operators.

Leave a Reply

Your email address will not be published. Required fields are marked *