This piece in The Guardian from Daniel Kahneman’s new book, Thinking, Fast and Slow, offers probably the most succinct account of research that shows when you choose stocks, you usually choose wrong; it also goes on to point out that there is literally no evidence that stock-picking is a skill:
The subjective experience of traders is that they are making sensible educated guesses in a situation of great uncertainty. In highly efficient markets, however, educated guesses are no more accurate than blind guesses.
Our message to the executives was that, at least when it came to building portfolios, the firm was rewarding luck as if it were skill. This should have been shocking news to them, but it was not. There was no sign they disbelieved us. How could they? After all, we had analysed their own results, and they were sophisticated enough to see the implications, which we politely refrained from spelling out. We all went on calmly with our dinner, and I have no doubt that both our findings and their implications were quickly swept under the rug and that life in the firm went on as before. The illusion of skill is not only an individual aberration; it is deeply ingrained in their culture. Facts that challenge such basic assumptions – and thereby threaten people’s livelihood and self-esteem – are simply not absorbed. The mind does not digest them. This is particularly true of statistical studies of performance, which provide base-rate information that people generally ignore when it clashes with their personal impressions from experience.
And that’s really what confidence is about: subjective impressions. There is no feature of the world that causes confidence. You can be an doubt-plagued billionaire, and suffer terribly for it. For confidence (and, for a lot of people, happiness), it seems that we need to link up our actions to consequences; we need to believe that our work and skill led to the results that we care about, or else we’re going to feel really quite crappy about everything that matters to us.
The alternative, of course, is to not give a damn. To accept the world as a Dionysian game of dice. As Iggy Pop says, that can be quite fun, but it sure is counter-intuitive.