Wall Street & Technology briefly discusses some survey results and concludes: “Wall Street’s Quants Feel Misunderstood.” There’s the obligatory quote from Dr. Wilmott:
“These numbers are alarming,” said Dr. Wilmott. “They indicate that even with the events of the past year, financial institutions are still not taking the importance of financial education seriously, especially as it pertains to improving relationships and understanding between quants and their managers.”
There’s some “alarming” statistics: since last year, quants feel that 86% of their managers have the same or less understanding of their quantitative roles.
But looking at the actual survey results, it’s not quite so bad. In fact, there’s a good deal of exaggeration, depending on how you frame the data. Only 4.5% feel that their managers have less understanding since last year – meaning a full 82% felt that the level of understanding is roughly unchanged. So this largely becomes a question of whether or not the present level of understanding is satisfactory. Most people’s knee-jerk reaction (and that of the original article) will unequivocably be, “Of course it’s not!” However, is that because managers haven’t kept up with quantitative advances, or because quants have run far ahead of their supervisors (and of where they need to be)?
I think it’s a little of both. Certainly, when Things Were OK, supervisors were less incentivized to follow the activities of the mathematicians under them. As long as the numbers danced (higher and higher), it didn’t really matter what they were. Meanwhile, each quant is incentively to pursue ever-more obscure models to squeak out minute bits of alpha. In the end, we wind up with quants doing overly-complex work for managers with too-relaxed supervisory roles. The question isn’t “Does your manager understand what you do?” as much as it is “Do YOU understand why you do what you do?”
The problem here is not that quants ran amuck and screwed up the system (see the replies to question #2), it’s that no one even knew what they were doing in the first place. The article is putting a normative spin on the survey results, but it’s silly to believe that if supervisors understood what quants were doing, everything would be fine. Just the same, if quants only worked within the limits of their supervisors’ knowledge, disaster would result as well (what’s the point of roles, anyway?). What is missing – and what surveys like this fail to address – is the need for proper communication of goals, objectives, methods and ideas. Yes, it might be hard for a mathematician to boil his ideas down to simple English or a supervisor to pick up some mathematical tenets, but the resulting clarity will be well worth the effort in either case.
So in the end, is it bad that quants feel like most of their managers only somewhat understand what they do? It’s hard to say. If the quants are doing their job “properly”, then yes. If supervisors are slacking off, then yes. But if quants are running ahead with inappropriate methods, then although the answer is still yes, the solution isn’t necessarily to educate the supervisors – it’s to teach them how to reign in the quants. Alternatively, it’s to teach the quants a little about their real business objectives.
And speaking of forecasts, I’m reminded today of one of my favorite forecasting errors: the echo. This morning, the manufacturing survey missed the forecasted amount, and many pundits commented that it contributed heavily to the market’s fall.
Here is a plot of the manufacturing survey level as reported each month in red (prior to any revisions, though there haven’t been any substantial revisions) and the forecasted level in green:
You can plainly see that the forecast in each month is more or less the reported level from the month before! This is what I call an “echo” forecast. Note that the echo is more pronounced since 2008. Prior to 2008, the forecast is a near-perfect echo which has been vertically scaled (more on that in a minute).
Recall that one of the hallmarks of a simple random walk is that its expected value at each step is the value of the previous step: 
. Bearing that in mind, the forecasted values of the manufacturing survey are a random walk with respect to the survey itself!
Again, it blows my mind that these forecasts are taken seriously. I can do just as well as this “informed forecast” by using the previous month’s survey value as my naive forecast! As a rule, if the second derivative is negative, then the forecast will be too high. If the second derivative is positive, the forecast will be too low. In a sense, therefore, the data is self-perpetuating as long as the forecast is taken seriously, since good news will look better (beats estimates!) and bad news will look worse (misses estimates!). The fact that the echo is more pronounced since 2007 means that the forecast became more random right when (people though) it mattered most.
To be fair, the model isn’t a pure echo (or it wasn’t before 2007). Instead of taking the previous value as the forecast, it appears that an adjustment was made first. Wherever there was a large surprise (i.e. a reported value well under or over the forecast), then the new forecasted was adjusted in the opposite direction of the surprise. That big dip in 2005 should have been the next forecasted value, but because it was such an outlier, surveyed economists bet on mean reversion and adjusted their forecast upwards. Through the entirety of 2006-7, the forecast is equal to the previous value less about 50% of the forecast. You can quickly build a regression model that includes the surprise as a variable to test this if you think I’m laying too much of my opinion into this analysis. To me, this suggests that in 2008, economists lost faith in their ability to forecast this indicator to the point that they stopped adjusting their naive guesses – a terrifying prospect for anyone following their opinions.
Now for a more nerdy sidenote: one must be extremely careful of the echo problem when running time series regressions, because without careful controls the analysis will return misleading significance. It’s easy to see why if you consider the expectation equation I printed earlier – if the best guess of a random walk’s current value is its previous value, than a statistical model which simply uses the previous value will seem to give extremely good results when in fact it gives no extra information. The R2 in particular will be extremely high. I just generated 100 points from a Gaussian random walk and regressed them naively on their lags, and came up with an R2 of 98%. Of course, time series data can’t simply be regressed in the first place, but let this be an illustrative lesson.
Or: Yet More Ways to Lie With Statistics.
Last Thursday, the month-over-month percent change in factory orders for February was announced at 1.8%. The expected number was 1.5%. Sounds like good news, right? Unfortunately, the January number was revised from -1.9% to -3.5% in the same release.
The easiest way to make a month-over-month change look better is to revise the prior month down.
Let’s walk through with real numbers. Say the index level is 100 on December 31. On January 31, it has dropped 1.9% to 98.1. On February 28, the expectation is that it will have risen 1.5% to 99.6.
Instead, after the numbers came out last week, we learn that the following path was actually realized: 100 on December 31. On January 31, a 3.5% drop to 96.5. On February 28, a better than expected rise of 1.8% to 98.2.
So February’s month-over-month change was better than expected, and yet because of the revision to the January numbers, we are 1.3% lower (98.2 vs 99.6) than the absolute expected level!
I’ve written before about the dangers of misinterpreting comparative statistics; this is an excellent tangible example.