This is Part II of my review of The Numbers Game
The revelation that the US government is spying on all of us has led to a sudden increase in sales of George Orwell’s novel 1984. It’s also true that the publication of Chris Anderson and David Sally’s book, The Numbers Game, has led to an increase sales of Soccernomics, so I for one am very grateful to them. However, in chapter 11 they do rather take us to task, in the politest way possible, on our analysis of managers.
In Soccernomics we argued that managers are not that important- based on the statistical claim that most of the variation in league performance can be explained by wage spending. A statistical model that allowed for the impact of managers showed that most were statistically insignificant and those that were statistically significant only improved league performance by a small amount.
Anderson and Sally’s book is premised on the notion that managers can use data in smart ways to improve performance, and that if they do they will add value to the team, and so much of their book would not fly unless they demolish the Soccernomics argument: hence chapter 11.
The basic point at issue is this. Wage spending seems to account consistently for a very large fraction of the variation in league position, therefore leaving only a small fraction for the managers. Sir Alex Ferguson would not have won 13 titles with the squad of Wigan, and the squad of Manchester United would have done well if operating as a worker’s collective. Anderson and Sally concede that wages account for a lot, but argue that the last fraction might be really important.
Books like The Numbers Game and Soccernomics are in some ways just summaries for lay readers of technical information. Publishers do not like you to dwell for too long on the minutiae of statistical research, judging that most readers will quickly tire (and they must be right, otherwise academic papers would get onto bestseller lists). So the question is always what to retain and what to leave out. Or to put it another way, the argument would go very differently in the seminar room.
I think it would go something like this. We want to know how managers affect performance, so we have to design a test. How would the labcoats do it? They would take a sample of clubs, and randomly divide them into two groups- a treatment group and a control group. The treatment group would get managers, and the control group would be run as workers’ collectives. After a while the labcoats would test whether the performance of the treatment group was significantly better that that of the collectives.
Of course we can’t do this, but the thought experiment shows what is going to be wrong with any statistical analysis using real data. First there are no workers collectives at operation in football (except perhaps the Dutch national team, which seems to do OK), so we never have a proper control group (caretaker managers might be an interesting control group, but there are clearly issues about focusing on transitional periods. Caretakers probably look like very good managers in fact, since as Anderson and Sally point out there is usually a rebound after a sacking. But as they also point out, this has very little to do with management and is just the statistical phenomenon known as reversion to the mean). Some might even be tempted to argue that managers matter just because everybody has one- why employ someone if they do nothing?
Regression analysis enables us to isolate the contribution of individual factors such as wage spending and the manager, but the problem is that the manager is not chosen independently of the wage spending, or the history of the club itself, and so cause and effect is hard to disentangle. Right now I’m working on a paper with Thomas Peeters where we are trying to compensate for these contamination effects, by modelling the productivity of the club separately from the manager, which might give us a better idea, but there is no perfect solution.
One thing that I was relieved about in the Numbers Game is that they didn’t challenge the idea that money buys success. Quite a few academics I know challenge the idea on statistical grounds- how do you know which way causality runs? Maybe success pays better. There’s only so much you can do in terms of statistical modeling to try to address this concern, but it seems to me the ultimate answer is theory. Who really thinks that Wayne Rooney is paid a lot of money just because he is employed by a rich and successful club? He’s paid a fortune because his talent is expected to bring success, and there is a market for his talent (which he appears to be testing right now), and that of every other player. Moreover, because this market is large and transparent, prices paid probably reflect talent quite well.
If I was inclined to whine (and what academic isn’t?), then I would object to footnote 23 which implies that our argument is that Wigan would play better if the players were paid more. I think the value of incentive pay in top professional sports is dubious- players are highly motivated without the need for win bonuses. Our argument is that the market more or less accurately sifts through the data and values a player at his likely contribution to team performance.
As Anderson and Sally say, the data I used for the regression analysis included the total wage compensation of all employees in the club, not just the players, and thus including the manager. Therefore the test for the contribution of the manager is really a test of whether he contributes above and beyond his unknown salary, and if his salary is large then he may be making a very large unmeasured contribution. Frick and Simmons have a paper on the contribution of managers in the Bundesliga, and they find managerial contributions are significant, although they don’t really have accounting data on player remuneration, so it’s hard to know if they really capture the relative contributions of players and managers.
Anderson and Sally admit that the scope for the manager’s influence, once wages are accounted for, is limited, if not so limited as we suggested. They argue that even small differences are important, and rightly point out that the variation unexplained by wages is much larger on a year to year basis. The problem I have with that is that the fluctuations from year to year tend to be random, and so if this variation is due to managers then they don’t seem capable of very much consistency. If management is about small margins, then so is luck, so we’re still unable to identify the managerial contribution reliably.
So if the data can’t settle the issue, what are we to think? Anderson and Sally get close to merely asserting that they matter: “But leaders matter. They really do” – really? I guess in that sense we all matter. But if this is to be something more than the butterfly effect, then we have to have a good account of what is going on inside the club, and as others have pointed out, it all still seems rather hazy. Football clubs are secretive institutions, and managers who think they know something don’t want to let on.
I think part of the argument may actually be a little stale here. Management has clearly changed since the 1980s, and it might make more sense to talk about management teams rather than individual managers. The guy at the top may know relatively little about what is happening beneath him. On the other hand, Simon wrote about Ferguson on his retirement that he obsessively collected relevant details from every information source and then used snippets strategically to manipulate events. But this makes it even harder to identify the contribution of a single individual. After all, we struggle to isolate the contribution of a player, so surely the manager’s contribution is even more nebulous. I’ve heard it said that senior managers in Japanese companies know almost nothing about the business, and rely entirely on their subordinates for decision making. Why have a manager who contributes nothing? Because the top job is a focus of competition for junior managers, who can aspire to reach this level. Someone who does nothing can actually be useful.
I’m not saying that is what is going on in football, but it makes the point that we should be wary of the great man theory. We dismissed this somewhat airily in Soccernomics, The Numbers Game takes us to task and cites several business school studies purporting to show that managers do matter. But here’s the nub, how many businesses are there where the average tenure of the senior manager is barely more than a year? According to a recent article on EPLTalk, following the retirements, sackings and relegations at the end of the season, then excluding Arsene Wenger, the average tenure of Premier League managers is 417 days. So I think what Anderson and Sally really need to explain, if they think that managers really matter, is what it is they are able to do with a 417 day window of opportunity.