Race and Performance in the NBA
The New York Times article this week about the study of racial bias in NBA officiating by Joseph Price and Justin Wolfers generated quite a bit of commentary. What is amazing is how little people understand, or are willing to understand, about statistics. Here's what the authors claim in the abstract of the study:
We find that--even conditioning on player and referee fixed effects (and specific game fixed effects)--that more personal fouls are awarded against players when they are officiated by an opposite-race officiating crew than when officiated by an own-race refereeing crew.
Much of the reaction among sportswriters has been to take the authors to task for calling the refs racist. (See Mike Wise in his column in Thursday's Washington Post and Kevin Hench at FoxSports.) Having taken a look at the study myself, I am surprised that those who make a living based on the sport would be so dismissive of the result. The main result of the paper is that the foul rate (fouls called per 48 minutes played) increases for black players when the racial composition of the three-person crew of referees goes from black to white. (See Table 4 and the discussion on page 8.) Any honest sportswriter should hold the NBA accountable for the result--why are the outcomes for fouls different across different racial configurations of refs and players?
It is very difficult to posit an explanation for these results that would attribute them to something other than race. First, no one disputes the NBA's claim that it does not assign referees to games based on their race or the racial composition of the two teams. (See page 4 and Table 1 of the study for discussion and evidence.) With (conditionally) random assignment, and the fact that the explanatory variables are fixed characteristics of people (i.e., race), we have the conditions for a clinical trial here, where "controlling" for possibly confounding factors is not likely to be important. Second, the authors do in fact control for a number of "fixed effects," exploiting the fact that their dataset is a panel consisting of a limited number of individuals observed in numerous interactions. This includes characteristics of the player and the refs that don't change over time. As the authors note, the most comprehensive results "are identified only off the differential propensity of teammates to earn extra fouls when the refereeing crew is of the opposite race."
Having said that, I think the authors soft-pedal one possible explanation of the results that would exonerate the refs. The following passage appears on pages 12-13:
The fourth point speaks to a relatively subtle interpretation issue: while we document a correlation between a player’s foul rate and the race of the referee, this may reflect the players responding to the race of the referees, rather than the referees policing opposite-race players more aggressively. Strategic responses by players would lead to an attenuation bias: expecting to receive more fouls for a given style of play, the players may play less aggressively, minimizing the impact of referee discrimination on realized fouls. This suggests that our results understate the amount of discrimination. Alternatively, if players exhibit oppositional responses, they may play more aggressively when policed by the opposite race. Importantly, such oppositional responses suggest that our findings are driven by changes in player behavior, rather than referee behavior. Yet if this were driving our results, one might expect to see effects not just on the number of fouls earned, but on the likelihood of fouling out, as well as other indicators of aggression, including blocks and steals. Instead, we find that blocks and steals actually decline under opposite-race referees.
I'm not persuaded by this reasoning. The player response needn't take the form of aggression--it merely needs to be a general decline in player performance in the presence of opposite-race referees. What if, for example, players find it more difficult to concentrate on their tasks when the refs are of opposite race? Elsewhere in the paper, the authors write, "Player-performance appears to deteriorate at every margin when officiated by a larger fraction of opposite-race referees." So why assume that it's the refs not the players? And why make a statement, "Basically, it suggests that if you spray-painted one of your starters white, you’d win a few more games," even under the possible coaxing of a reporter?
The interpretation of the results that it's the players, not the refs, may also reconcile the results of NBA's internal studies that claim that, on a call-by-call basis, there is no evidence of racial bias. (The NBA has not released the results of these studies, much less the data.) If the players are changing their game based on the racial composition of the refereeing crew, then it is possible that every call or non-call is legitimate, and both studies can be accurate.
4 comments:
In a study such as this, is it necessary to control for whether a foul, or better yet a "potential call," is for what one might deem an "obvious" foul as opposed to a could-go-either-way foul? That is, some fouls in basketball "every" ref would call, so race is irrelevant, and some fouls aren't. The latter are the ones where the effect they find would live. Of course, this doesn't preclude your preferred explanation, Prof. Samwick, of player behavior responding to the race-composition of refs.
Wait, wait. You're surprised that the general American public and sports writers in particular are not familiar with techniques of basic statistical analysis?
To paraphrase one of my favorite movies: "Who's being naive now Andrew?"
On to the consequential part of the study. We need to look at the actual referee assignments. Because the league has three man crews, they are almost always mixed. This means that even swinging to one extreme (all white crews) or the other (all black crews) would not have nearly the 4% headline effect that the authors mention. I also wonder why the authors did not choose to list their data by year. Is this a smooth effect (strong result) or is there a lot of noise (weak result)? Is there a solid trend line indicating whether bias is increasing or decreasing over time?
I know that they probably spent a long time coding this data, but that does not mean that their result says anything significant about the NBA. That is the point most sportswriters were saying. The study can still be impactful in terms of biological/behavioral reactions to race, but that alone would not have gotten them on the cover of the NYTimes...
Andrew,
Certainly some have responded to the study incorrectly, but the authors are also selling it hard and in ways that are a bit sensationalist.
Just one example. The economic magnitude of all coefficients are discussed as going from zero white to all white crews. Only 3% of games are reffed by all black crews. More realistically, what does it mean to switch out 1 ref (white for black), it leads to an increadibly small difference (roughly .06 fouls per 48 minutes).
Also, there are tons of fixed effects in the regression, but the key coefficient is on an interaction. Observable referee characteristics are not interacted with player characteristics. This makes the "but we put in all the fixed effects" defense to omitted variable bias incomplete.
For example: Say black refs are less experienced on average (the NBA was hiring black refs at a higher rate than the current composition, so the black refs had on average fewer years of experience). Say white players are more foul prone (they get more fouls per 48 minutes unconditionally). Say inexperienced refs call more fouls on foul prone players (they are less likely to ignore unimportant fouls to keep the flow of the game going or are more likely to see something awkward and assume its a fould) Put together, black refs will call a higher percentage of their fouls on whites.
Table 3 shows that the fouls per 48 minutes on black players is unchanging across crew composition. The difference is that white refs call fewer fouls on white players (so the share of fouls they call on black players is higher). That fits the previous hypothesis perfectly.
The point is, the magnitude is tiny, other explanations could be there, the unconditional stats in table 3 show NO difference for 1 white 2 black vs. 2 black 1 white. In all, one could have looked at these numbers and reported that race plays a remarkably small role in NBA refereeing (and with a few more interactions maybe none at all). Any number of referee characteristic / player characteristic interactions (former players may ref differently, etc. etc.)
No one in the league thinks this is a problem because even if the numbers are as they are reported, it is a really small problem. The headlines and quotes made it look like a big deal (and most tended to focus on the "spray paint a player white" type of quotes as opposed to noting what the study showed was a slight own-race benefit, not racism on the part of white refs), but I'm not convinced by the study as written that it really is a big problem.
Jay
Jay (and others),
I agree with you on the question of magnitude of the effect. I also think your point about the need for referee characteristics interacted with player characteristics has merit. Thanks for the feedback.
Andrew
Post a Comment