As many of you know, right before the election I made a $50 bet with Hank Campbell that Nate Silver would get at least 48 out of 50 states correct for the 2008 presidential election. I also got one of Hank's readers to sign on to the same bet. Additionally, a few readers and Twitter followers got in on the wager; they were bullish on Romney's prospects, and I was not (more honestly, I was moderately sure they were self-delusional, and willing to take their money to make them more cautious about their self-delusional biases in the future). But there's a major precondition that needs to be stated here: I hedged. Last February a friend told me he was 100% confident that Barack Hussein Obama would be reelected. This prompted me to ask for favorable terms on a bet. The logic was simple, if he was 100% confident, then it shouldn't be a major issue for him, because he was collecting anyhow. As it happens he gave me 5 to 1 odds, so that I would collect $5 for every $1 he might collect. I told him beforehand that I actually thought that Obama had a 60-70% chance of winning, so I went into the wager assuming I'd be out a modest amount of money. But that was no concern. My goal was now to convince those who were irrationally supportive of Romney to take the other side of the bet. For whatever reason people have an inordinate bias toward their hoped-for-candidate in terms of who they think will win, as opposed to who they wish to win. The future ought gets confused with the future is.* I got people to take the other side, which means that I was going to make money no matter who won.
At this point one might wonder about my comment that I suspected that those who were bullish on Romney were delusional. It's rather strong, and my reasoning is actually rather strange. Overall I accepted the polling averages. A few years back I was an economic determinist in election outcomes, but Nate Silver had convinced me that the sample size was too small to get a good sense of the real proportion of variation being predicted here. In short, the economy matters, but I stepped back from the supposition that it was determinative (as it happens, purely economic models that were excellent at predicting past elections face-planted this time). So that's why I relied on the polls. Though I leaned on Nate Silver, I didn't think he was particularly oracular, and I'd say that I'm mildly skeptical of the excessive faith some put in his particular person. When I put a link up to Colby Cosh's mild take-down of Silvermania I received a few moderately belligerent comments. This despite the fact that I was willing to put money on Silver's prediction.
But after soft-pedaling my confidence in polling averages, why did I think the pro-Romney people were delusional? The simple answer is 2004 and 2008. When the polling runs against you consistently and persistently motivated reasoning comes out of the wood-work. There's a particularly desperate stink to it, and I smelled it with the "polls-are-skewed" promoters of 2012. In 2004 there were many plausible arguments for why the polls underestimated John F. Kerry's final tally. And in 2008 there were even weirder arguments for why McCain might win. In 2012 it went up to a whole new level, with a lot of the politically conservative pundit class signing on board because of desperation. The reality is that out of the space of plausible models you can find something congenial to your own proposition. I very studiously avoided reading much about the debates about skewed polls, even in the comments of this weblog. For example, Dwight E. Howell left this note on September 29th:
You might want to go back and look at how accurate polls have been at predicting elections in the past. The track record isn’t great. Even the exit polls in WI were wrong. It appears the Democrats who wanted the governor out stopped and chatted and the people who voted to keep him largely walked on by including a significant number of Democrats who had to have voted for him. There is also the non trivial question of how many of the various sub groups are actually going to show up on election day. If you assume that blacks will turn out in the same numbers as his last election you get one result. If you note that the black community has not fared well during his tenure in office and he has deeply offended many black Christians you have to wonder if some of these people are going to bother to show up and vote for him. The Jews have to know his position relating to the Jewish state, etc. He pretty much had a solid Catholic vote last time but he’s at war with the Catholic church. What does this all mean? You’ll find out after the votes are counted.
The votes have been counted, and Dwight E. Howell was full of shit. In fact I badgered Howell on Twitter and on these comment boards to put a wager down on the election, and he finally begged off after he couldn't evade me, claiming he wasn't a betting man. I have a hard time dismissing the possibility that Howell himself knew he was a delusional crank full of bullshit on some level! And yet what he said wasn't crazy. The reality is that I didn't read most of Howell's comment until after the election. The same with the very similar comments that came through in Howell's wake on that thread (I did not post them, I simply skimmed the first few sentences). A similarity of content across the comments suggested that these individuals were just regurgitating plausible nuggets and feeding their motivating reasoning bugs. And that's why I avoided detailed inquiry into the issues: I didn't want to bias my own perspective! This was part of the source of Hank Campbell's confusion as to my somewhat erratic response on Twitter as I frantically tried to make a bet with him on the election: I didn't really care about Hank's theories about the polls, I suspected that the polls were right because I strongly scented a lot of bullshit on the Republican side. I wanted to get Hank down on some bet, and I wasn't too concerned with the details. In contrast to the odor wafting up from the Republicans, the Democrats seemed sincerely and guilelessly accepting of the polls which favored them. My intuition here could have been wrong, or the perceptions of the parties of interest may have been wrong. But that was really the situation and context which motivated my behavior at the time. After the election was over I actually started reading some of the arguments about why the polls were skewed, and I find that they are extremely plausible to me. And not just me, John Hawks owes me a drink because he simply didn't believe the turnout models which suggested a demographic more like 2008. The reality is that my instinct was to go with John. I too was very skeptical of the proposition that Obama could turnout the same voters as he did in 2008. And yet he did turnout those voters! What does this tell me after the fact? The plausibility of any given datum can't outweigh the aggregate. Dwight E. Howell et al. have a lot of plausible historical data. Granted, you have some obvious bullshit "SHOCK POLL" headlines, but only idiots believe those outliers (there are plenty). Rather, if you have a model, there are plenty of data points you can populate to get the appropriate outcome. That was my suspicion and worry, and I find that I'm highly susceptible to some of the more cogent and eloquent arguments about turnout models (not Dwight's comment specifically, most of the non-specialists signal that they are just echoing the specialists by garbling and muddling transparently). My initial instinct to not allow myself to info-overload, and then filter it out to the subset which confirmed my model, seems to have been wise. And importantly I relied on the expertise of others. I'm just not that motivated or interested in horse-race politics (though I am interested in political history, philosophy, and economy). I assumed that "political junkies" of partisan sentiment would keep track of the likely outcomes, and when Right and Republican leaning individuals started making desperate sounding arguments with the intent of converting themselves, I believed that that signaled that Obama was on the rise. Similarly, I also defer to the collective wisdom of the polls. This does not mean that these two are infallible (my judgement of people bullshitting, or, the wisdom of the polls). But it's better than nothing, and I ended up the richer. All this brings me to Nate Silver's The Signal and the Noise: Why So Many Predictions Fail-but Some Don’t and Jim Manzi's Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society. These two authors are class acts, and I follow both of their pronouncements closely. I have long appreciated Silver's contribution to the broader discourse, and though Jim Manzi may not be as prominent, he is an important voice for empiricism on the modern political Right, which too often seems to simply be a reiteration of old hopeful ideals.
But ultimately if you are a long time reader I'd have to say that you should go with Uncontrolled and not The Signal and the Noise
. First, I will touch upon an issue that may seem superficial to many: style. Jim Manzi may not have the most limpid of prose. He is of course spending a great deal of time on epistemology, history of science, and quantitative business strategy. But Silver is a far better blogger than he is a narrative nonfiction writer. Many of the chapters in The Signal and the Noise have a formulaic quality, insofar as the focus is clearly on the ideas, but there are often pro forma biographical introductions of important thinkers. There are writers who do this well. I doubt I would; I'm more interested in ideas than the people generating them. And I suspect so is Silver. He almost certainly finds Futarchy more fascinating than Robin Hanson. The main exceptions tend to be in areas where Nate Silver has some personal connection. The chapters on the quantitative revolution in professional sports scouting and gambling are more lively, with more loving attention to the dramatis personae. And that makes sense if you have some priors in hand: Silver comes out of a quantitative sports analyst background, and, he was a professional poker player at one point. But more importantly as a work of popularized statistical inference The Signal and the Noise probably would not add much novel data or cognitive tools to the typical core reader of GNXP. Most of you are presumably aware of Bayesian probability, and the abuses of modern Frequentism. If this is Greek to you, then I would recommend The Signal and the Noise! And perhaps check out the Less Wrong Wiki. If you don't know that economists are notoriously bad at predicting recessions, or that political prediction models based on a few economic or social indices are notoriously good at predicting the past but bad at predicting the future, then The Signal and the Noise may also be for you.
And reading this book reiterated to me that Nate Silver is a great blogger whose Weltanschauung is broadly similar to mine
. But The Signal and the Noise did not present to me any grand revelations. It was an exploration of topics which I developed interests in in striking parallel with Nate Silver over the aughts. I suspect this is a function of the change in our relationship to data due to the power of computers in terms of both storage and analysis. Silver is a reflection of the age, a herald, not a prophet. We are part of the same army. Jim Manzi's Uncontrolled is a somewhat different work, insofar as within the author explicitly outlines the relatively constrained scope of his ambition. The core of Manzi's argument is that public policy would benefit from more randomized controlled trials (also known as randomized field trials). This seems a plainly sensible project, but Manzi's assertion is that too often enormous public policy ideas are proposed, and then implemented on a massive and indiscriminate scale. Whether the policy was ever effective or not can often be litigated, because there was never a "control." In the end Uncontrolled is a plea for experimentation, epistemological humility, and incremental gains on the margin. Obviously Manzi is not presenting himself as Prometheus. Rather, this is a small vision executed on a massive scale. Manzi has seen this work in the business work, and he wants to translate these private sector successes to the public domain. But perhaps what Uncontrolled does better than convince you of the efficacy of randomized controlled trials in public policy is that there are limits to the power of elements of the scientific method in particular domains. This is the old hierarchy of knowledge idea, so Manzi is treading over ground familiar to many. In short, physics is easy, and economics is hard. Grand general theories with a few variables have generally failed in economics where they have succeeded in physics. That is not due to the lack of ingenuity of economists (many of whom come from a physics background!), but simply due to the fact that economic phenomena are much more complex than many physical phenomena. In Manzian terms they have "high causal density." There are so many possibilities that simple models and obvious large correlations are not going to be robust or existent. This goes back to why I was very cautious about reading too much about the skepticism of polls before the election. There are so many possibilities it is incredibly easy to conjure up a plausible skepticism of the received wisdom, and present an alternative. True aficionados who wallow in the data can filter the good from the bad, but we civilians rarely can. Importantly I would like to add that this is something Silver acknowledges in The Signal and the Noise. Formal quantitative analysis supplemented by qualitative knowledge trumps quantitative analysis alone. If the pundits who criticize the quants have true knowledge, they will only benefit. If they don't have true knowledge, as Philip Tetlock has reported, then they have much to fear. All things leads us to the common sense conclusion that the process to attain knowledge is hard. Powerful math and statistics can give us only so much. Experiment without theory is not illuminating. A theory devoid of empirical data is not persuasive. Randomized experiments without any guiding model or hypothesis may be lacking in insight. These are the outward aspects. But what about the personal strategies for attaining knowledge? If one is focused on one's domain, one need not over-think this. Presumably, you know your shit. But when you move out of domain your need to be very careful, because you are on alien topography. One suggestion I might make is be careful of looking too hard for data confirming your prejudices; it is all too easy if you are clever. Rather, look only modestly, and withdraw quickly if you don't find what you are looking for. If it was all that clear and obvious, it would have been clear and obvious to you initially. The bold and plain truth does not hide. * For those inquiring about Intrade, it is not that easy to deposit money into that system if you are American. Try it.
