The investment principle that the stock market is most overvalued when optimism is at its highest and is most undervalued when pessimism is at its deepest turns out to be more of a curiosity than an iron-fisted law of economics. A quick look back at some of the market's most dramatic swings shows only mixed results for this rule, a rule that opens dangerous traps during initial cycle moves.

The principle is based on the availability of funds — that available investment funds are low when optimism is high because investors, by extension, are already fully invested. The mirror image holds when pessimism is high, that investors are out of the market implying that funds are available.

A rich source of data on market sentiment is available in the Conference Board's monthly consumer confidence report. Since June 1987, the Conference Board has asked consumers what they think of the stock market, whether they see gains ahead, loses or no change. These data are based on robust monthly samples of 4,000 consumers. The current mix is roughly balanced, showing pessimists at 31.8 percent, optimists at 28.6 percent, and the no-change camp at 39.6 percent.

But the data aren't always evenly balanced. The split widens dramatically when the outlook for the economy takes sudden turns such as it has over the past couple of years, a period that offers memorable but still questionable support for our principle of opposites.

It was in March last year that the Conference Board's data showed wide unity of opinion — that is nearly all negative. The pie chart above shows the black slice of the bears at 52.8 percent vs. 19.6 percent for the red slice of the bulls in what, at 33.2 percentage points, is the second greatest spread on record. The March reading of course would offer dramatically definitive proof for the principle of opposites if it weren't for the greatest spread on record, at 36.7 points which appears in July of the prior year, the month when oil spiked to $150. The rise in pessimism at this time was a very false buy signal for the stock market which would lose roughly 50 percent of its value through the second half of 2008 and into March of 2009. Premature signals prove to be the pitfall for the principle of opposites.

In all the years of the Conference Board's data, the 2008 and 2009 period proves to be by far the most extreme, both for optimism but especially for pessimism. The above graph tracks the red line of the bulls with the black line of the bears through the history of the series. The right of the graph shows the unmatched swelling in pessimism. But again, the beginning of this surge in pessimism is no buy signal at all, only the very end of the period.

The Conference Board's data are bumpy month-to-month, as illustrated in the above graph which compares the black line of the pessimist camp against the blue line of the S&P 500 index over the last three years. Big monthly spikes in sentiment, like the upward spike at the very left of the graph, don't necessarily point to a shift in trend which makes the graph difficult to gauge. The three-month average in the graph below is not only easier to read but underscores the importance of taking your time with this series, of not jumping to conclusions especially where your money is concerned.

The above chart offers a close-up view of the July 2008 peak in pessimism, at a record 49.3 on a three-month basis, followed by the second peak at 48.4 in March last year. Again, the first peak is not a contrarian signal at all, only a direct and leading signal for a great decline that was only beginning to unfold. It's interesting that pessimism didn't intensify during the stock market's collapse through the second-half of 2008, perhaps reflecting the possibility that consumers had already foreseen and discounted the collapse. The second high mark for pessimism, coming at the end of the collapse, proves in fact to be a very powerful buy signal. But during March last year there was little talk that the market had hit bottom and much more talk that the S&P was doomed for a 500 handle.

In the graph below we shift to the optimists for a look at the early 2000 period, one marked by the dotcom bubble followed by recession and then the invasion of Iraq. The graph, again using a three-month average, opens in the winter of 2000 with a spike to 45.9, a spike in optimism that accurately signaled a market top and, for the contrarian, a strong short. Optimism then fades with the market to a low at 28.9 in the spring of 2001. But this buy signal, like July 2008, is for dead men only as the market still had a long ways down to go. A subsequent rise in optimism to a 42.2 peak in the spring of 2002 correctly signaled a short. The short extends out a full year before the invasion of Iraq in the spring of 2003 sent confidence to deep lows, deep lows that correctly signaled, for the surviving contrarians, a cyclical shift higher for the market.

Let's close by taking a month-to-month look, in the graph above, at the great crash of 1987. Optimism collapsed along with the stock market to correctly signal in December that year a cyclical shift higher. But in March 1988, optimism suddenly soared, offering the contrarian a sell signal that however would last only a single month. But this chart is month to month — not the three-month average! This is a reminder that this series is jittery and is best looked at over a longer time frame. The chart below tracks the three-month average, pushing the buy signal out one month to January and smoothing out the false sell signal of March.

Bottom Line

Basing an outlook for the market on a contrarian view of sentiment can be dodgy. Correct signals do outnumber the wrong signals and can be profound in their accuracy. But initial shifts do not always signal an immediate contrary response in the market, a gray area that poses practical risks to the principle of opposites. Watch for the Conference Board's consumer confidence report on the last Tuesday of every month.