We have a strong need to make sense of our past. The thinking goes something like this: If we can make sense of our past, we could predict our future. This kind of thinking is especially true in trading. If we could merely see a pattern in market data from the past, we could use the past to forecast the future. It makes sense. Many disciplines do it. Insurance companies build regression models to estimate specific target groups that are likely to have an expensive auto accident.
In medicine, epidemiologists try to identify groups of people that are at risk of contracting a disease. Building regression models works in the long run if you have a large amount of data to use to build your models and there is relatively little “error” or small amounts of variation. Indeed, big time financial institutions often use economic data to forecast and trade long-term trends, and as long as they invest in the long term, and have huge amounts to invest, they can account for the variation in the markets and make a relatively profitable, lower-risk investment.
That said, many traders and investors, including smaller hedge fund managers, make the mistake of applying the “law of large numbers” to samples of “small numbers.” When we are trading smaller accounts, and over the short term, it’s difficult to anticipate a market trend with historical data. But because of our psychological need to be right and feel mastery over the markets, we allow our beliefs to bias our judgments.
In the markets, history only repeats itself when it does. The rest of the time it doesn’t. That’s all right. The fact that there is variation in the markets does not mean that you can’t trade them. It just means that you cannot trade them and erroneously think that you can do so with more certainty than is reasonable. Everything looks good in hindsight. It’s possible when building a regression model to “over-fit” your available data, in other words, seeing things that aren’t there.
Rather than acknowledge that the model has a wide margin of error, it’s tempting to believe that it is more precise than it actually is, and will make a more accurate forecast than is possible. The reason that the model may not accurately forecast the market is that you may not be using a large enough sample of data to make a forecast. You are using a small sample of data to forecast something that requires a large amount of data. In addition, the sample of data you are using from the past may not be representative of what will happen in the future. Data can only be used to build valid statistical models if it can generalize from one span of time to the next, but that may not always be the case.
It is always useful to have a healthy sense of scepticism when it comes to forecasting the markets with historical data. Don’t place so much confidence in your forecasts that you don’t manage risk. Our mind can play tricks on us. We can scour historical data and because we put in a great deal of time and effort into the building the model, we may wrongly think that we have found the Holy Grail of regression equations, but in reality, you may be placing your trust in an invalid model.
It’s not impossible to forecast the markets but always keep in mind the potential limitations of your forecasts. You may have “unrepresentative data.” Or you may be psychologically based. There are times when everything may look rosy, but in reality, things are too good to be true.