The financial industry generally places more emphasis on style than substance. Because of this, when their work is actually evaluated, results tend to be disappointing. Wall Street’s earnings forecasts? Overly optimistic. Performance of mutual fund managers? Quite embarrassing. You may be wondering: Do Morningstar ratings also belong in the same category?

You’re probably familiar with Morningstar and their one- to five-star mutual fund ratings. Many investors rely on Morningstar for stock and mutual fund research, and mutual fund companies love using Morningstar ratings in their marketing materials. But is there any value in a five-star Morningstar rating? (Disclosure: I use Morningstar software sold to investment advisors almost everyday.)

The Morningstar ratings for one of the funds J.D. owns.

Fortunately for us, researchers recently looked into these ratings and published their results. They compared Morningstar ratings to fund expense ratios as a predictor of future performance.

The expense ratio is the annual fee for investing in a fund. This fee is charged by the mutual fund manager, and it’s one of my favorite metrics. If you assume that mutual fund managers have no value — which I find to be a very good approximation — you would expect lower costs to predict better performance. And the report found just that:

Expense ratios are strong predictors of performance. In every asset class over every time period, the cheapest quintile produced higher total returns than the most expensive quintile.

What about Morningstar ratings? Five-star ratings predicted better performance than one-star ratings in 13 of 20 observations — a success rate of just 65%. That sounds pretty good on its own, but it’s still worse than a metric that anyone can look up in seconds.

Since Morningstar uses prior performance (after fees) to calculate its ratings, the ratings already include information about expense ratios indirectly. So what is Morningstar adding with its fancy algorithm? Let’s use a little high-school algebra to find out. (Geek Alert!)

And we just found out that:

Finally, using my graduate degree in math, I get this:

Yes, Morningstar’s algorithm is horrible. And that’s not all.

Morningstar reserves its five- and one-star ratings for the top and bottom 10% of funds. However, the researchers conducting this study divided expense ratios into quintiles — or, as normal people would say, 20% buckets. The expense ratios were handicapped by using 20% buckets instead of 10%, and still beat Morningstar ratings. Ouch!

Well, there’s one thing I forgot to tell you. People have performed this evaluation many times with similar results, so it isn’t news to serious students of investing. The interesting part of the report I quoted is the publisher: Morningstar. If you read its report [PDF], it sounds like a politician answering a tough question — uncomfortable. Independent thinkers can go directly to the results here [PDF].

Note: After writing this, I noticed that Morningstar clarified that ratings are indicators of past performance, and should not be used to predict future performance. If Morningstar were concerned about substance, it would tailor its ratings to how investors actually use them — as an indicator of a good investment. If it did that, most five-star rated funds would just be index funds. Unfortunately, Morningstar emphasizes style (and money), so it ends up with an imperfect rating system that benefits one of its biggest clients: mutual funds.

Another Note: Morningstar responded to a version of this article that was published on Mariposa Capital Management’s blog in August. You can read the comments here.