This is Part II of my review of The Numbers Game
The revelation that the US government is spying on all of us has led to a sudden increase in sales of George Orwell’s novel 1984. It’s also true that the publication of Chris Anderson and David Sally’s book, The Numbers Game, has led to an increase sales of Soccernomics, so I for one am very grateful to them. However, in chapter 11 they do rather take us to task, in the politest way possible, on our analysis of managers.
In Soccernomics we argued that managers are not that important- based on the statistical claim that most of the variation in league performance can be explained by wage spending. A statistical model that allowed for the impact of managers showed that most were statistically insignificant and those that were statistically significant only improved league performance by a small amount.
Anderson and Sally’s book is premised on the notion that managers can use data in smart ways to improve performance, and that if they do they will add value to the team, and so much of their book would not fly unless they demolish the Soccernomics argument: hence chapter 11.
The basic point at issue is this. Wage spending seems to account consistently for a very large fraction of the variation in league position, therefore leaving only a small fraction for the managers. Sir Alex Ferguson would not have won 13 titles with the squad of Wigan, and the squad of Manchester United would have done well if operating as a worker’s collective. Anderson and Sally concede that wages account for a lot, but argue that the last fraction might be really important.
Books like The Numbers Game and Soccernomics are in some ways just summaries for lay readers of technical information. Publishers do not like you to dwell for too long on the minutiae of statistical research, judging that most readers will quickly tire (and they must be right, otherwise academic papers would get onto bestseller lists). So the question is always what to retain and what to leave out. Or to put it another way, the argument would go very differently in the seminar room.
I think it would go something like this. We want to know how managers affect performance, so we have to design a test. How would the labcoats do it? They would take a sample of clubs, and randomly divide them into two groups- a treatment group and a control group. The treatment group would get managers, and the control group would be run as workers’ collectives. After a while the labcoats would test whether the performance of the treatment group was significantly better that that of the collectives.
Of course we can’t do this, but the thought experiment shows what is going to be wrong with any statistical analysis using real data. First there are no workers collectives at operation in football (except perhaps the Dutch national team, which seems to do OK), so we never have a proper control group (caretaker managers might be an interesting control group, but there are clearly issues about focusing on transitional periods. Caretakers probably look like very good managers in fact, since as Anderson and Sally point out there is usually a rebound after a sacking. But as they also point out, this has very little to do with management and is just the statistical phenomenon known as reversion to the mean). Some might even be tempted to argue that managers matter just because everybody has one- why employ someone if they do nothing?
Regression analysis enables us to isolate the contribution of individual factors such as wage spending and the manager, but the problem is that the manager is not chosen independently of the wage spending, or the history of the club itself, and so cause and effect is hard to disentangle. Right now I’m working on a paper with Thomas Peeters where we are trying to compensate for these contamination effects, by modelling the productivity of the club separately from the manager, which might give us a better idea, but there is no perfect solution.
One thing that I was relieved about in the Numbers Game is that they didn’t challenge the idea that money buys success. Quite a few academics I know challenge the idea on statistical grounds- how do you know which way causality runs? Maybe success pays better. There’s only so much you can do in terms of statistical modeling to try to address this concern, but it seems to me the ultimate answer is theory. Who really thinks that Wayne Rooney is paid a lot of money just because he is employed by a rich and successful club? He’s paid a fortune because his talent is expected to bring success, and there is a market for his talent (which he appears to be testing right now), and that of every other player. Moreover, because this market is large and transparent, prices paid probably reflect talent quite well.
If I was inclined to whine (and what academic isn’t?), then I would object to footnote 23 which implies that our argument is that Wigan would play better if the players were paid more. I think the value of incentive pay in top professional sports is dubious- players are highly motivated without the need for win bonuses. Our argument is that the market more or less accurately sifts through the data and values a player at his likely contribution to team performance.
As Anderson and Sally say, the data I used for the regression analysis included the total wage compensation of all employees in the club, not just the players, and thus including the manager. Therefore the test for the contribution of the manager is really a test of whether he contributes above and beyond his unknown salary, and if his salary is large then he may be making a very large unmeasured contribution. Frick and Simmons have a paper on the contribution of managers in the Bundesliga, and they find managerial contributions are significant, although they don’t really have accounting data on player remuneration, so it’s hard to know if they really capture the relative contributions of players and managers.
Anderson and Sally admit that the scope for the manager’s influence, once wages are accounted for, is limited, if not so limited as we suggested. They argue that even small differences are important, and rightly point out that the variation unexplained by wages is much larger on a year to year basis. The problem I have with that is that the fluctuations from year to year tend to be random, and so if this variation is due to managers then they don’t seem capable of very much consistency. If management is about small margins, then so is luck, so we’re still unable to identify the managerial contribution reliably.
So if the data can’t settle the issue, what are we to think? Anderson and Sally get close to merely asserting that they matter: “But leaders matter. They really do” – really? I guess in that sense we all matter. But if this is to be something more than the butterfly effect, then we have to have a good account of what is going on inside the club, and as others have pointed out, it all still seems rather hazy. Football clubs are secretive institutions, and managers who think they know something don’t want to let on.
I think part of the argument may actually be a little stale here. Management has clearly changed since the 1980s, and it might make more sense to talk about management teams rather than individual managers. The guy at the top may know relatively little about what is happening beneath him. On the other hand, Simon wrote about Ferguson on his retirement that he obsessively collected relevant details from every information source and then used snippets strategically to manipulate events. But this makes it even harder to identify the contribution of a single individual. After all, we struggle to isolate the contribution of a player, so surely the manager’s contribution is even more nebulous. I’ve heard it said that senior managers in Japanese companies know almost nothing about the business, and rely entirely on their subordinates for decision making. Why have a manager who contributes nothing? Because the top job is a focus of competition for junior managers, who can aspire to reach this level. Someone who does nothing can actually be useful.
I’m not saying that is what is going on in football, but it makes the point that we should be wary of the great man theory. We dismissed this somewhat airily in Soccernomics, The Numbers Game takes us to task and cites several business school studies purporting to show that managers do matter. But here’s the nub, how many businesses are there where the average tenure of the senior manager is barely more than a year? According to a recent article on EPLTalk, following the retirements, sackings and relegations at the end of the season, then excluding Arsene Wenger, the average tenure of Premier League managers is 417 days. So I think what Anderson and Sally really need to explain, if they think that managers really matter, is what it is they are able to do with a 417 day window of opportunity.
“how do you know which way causality runs? Maybe success pays better.”
Can’t we focus only on teams where there was a sudden influx of money, like Chelsea when Abramovich entered the picture, PSG in recent years, Malaga etc.?
There are also reverse cases, like Leeds and Fiorentina at the turn of the century. Looking at it from a layman’s point of view, it seems like money does cause success, not the other way around.
I think your intuition is right, but that’s not the same as statistical evidence. Chelsea and PSG are suggestive, but it’s never the case we can say that money came in and literally nothing else happened. For example, money brought in Mourinho, and money bought the team, how much of these two effects (and there could be others) led to Chelsea winning the League?
Only saw your response now, thanks for replying. If you’re still getting notifications about comments:
If the only problem is we need to control for the new coach, there’s a “trend” where the “new money” keeps the old coach for a season before bringing in big names (Mark Hughes at City, Ranieri at Chelsea). Maybe if we limit ourselves to the performance in those situations, we can get something statistically significant. But I’m guessing there are other variables we need to control for, so that trick alone won’t do.
Will your paper with Thomas Peeters be available online? I must admit, I find this riveting. This may sound slightly ridiculous, but the Football Manager editions have a large database of manager’s wages. I expect they don’t have a high amount of accuracy, but they are well-researched and would perhaps give a ballpark figure. It’s not something you could base an academic paper on though, I grant you. I found both your’s and Simon’s and Sally and Anderson’s books superb. I was convinced by your arguments about managers making no difference, but I have to give credit to Sanderson for raising some valid points too. I always struggle with the concept that with the credit you and Simon award Clough and Taylor on player recruitment, how can it be said Clough made no difference during his tenure? I think the art of being a good manager is just exploiting a knowledge gap, and then you can make a difference. Clough used recruitment, Wenger used recruitment too, with new diet and fitness regimes, and even Michael Laudrup this year with his knowledge of the Spanish players market. If you, specifically, as a manager, sign 3 great players for your club, that otherwise would have gone elsewhere, then I think you have made a significant difference. You can sign these players in a lot less than 417 days. Other than this grey area, which is probably down to my own misunderstanding, I’d say you and Simon were on firmer ground for what it’s worth.
When we have a paper I will post the working paper version here. We have been at this for nearly two years and have made a lot of progress, but we still have quite a lot to do.
I certainly agree with you about Clough- it’s crazy to say he didn’t make a difference. I think statistical models that allow for wage spending usually end up finding that about 10-20% of managers make a difference.
I’m less sure about picking three great players in the transfer market. I think of that like picking stocks in the financial markets. Efficient markets theory says that all relevant information is already in the price. That doesn’t mean that some players/stocks won’t turn out to be better than expected, just that this will be luck, not attributable to the manager/stock picker.
A great response. I see you and Simon as the Richard Dawkinses of the world of football, with football being organised religion in this analogy… (this concept makes a lot more sense in my head). Keep up the excellent work and I look forward to the paper.
Thank you for your books, articles and for this piece.
I feel I need to comment. I think Jack Coles made a good point. And of course, the picking of players can be explained through efficient market theory. The problem as I see it, is that efficient market theory is more of a normative theory than descriptive. Empirically, it is not supported. My opinion is that it is wrong to suggest transfermarket is efficient, I would rather describe it as irrational.
Efficient market theory is testable- and is shown not to hold in some specific circumstances. However, even when anomalies exist, it is usually hard to trade profit systematically from these inefficiencies- it’s very hard to beat the market. I think that’s the position with players- any single player you hire will perform better or worse than you would have expected given the price you pay, but if you hire 100 players it’s quite unlikely that you will do significantly better or worse than would have been predicted given the money you spent. A few exceptional managers do outperform, but the majority are no better or worse than the budget they command.