My Recalibrated Model Is Finally Accurate, And The Recalibration Was Deleting The Variable I Am Known For

I said in June that the model required significant recalibration. I have now done the recalibration. It is one line shorter than it was, and the line I removed is the one with my name on it.

In June I wrote in this space that my predictive model had required significant recalibration, and I used the word required because I did not at that point know what the recalibration would consist of. I have spent the fourteen weeks since finding out. The answer is one line of code, and the line is a deletion, and it is the line that contains the variable I have spent eleven years asking other people to put in their models. I am going to describe what I did, in order, because the sequence is the only part of this that is useful to anybody, and because if I describe only the finding it will be read as humility, which it is not. It is arithmetic, and it was available to me at any point since 2016.

The model is the one I have published since 2012. It takes eleven inputs and returns a two-party vote share by state. Its mean absolute error over the four cycles before the last one was 2.4 points, which is respectable, and in the last cycle it was 7.6, which is not. In June I assumed, as everybody in my field assumes in that situation, that the world had changed and the model had not. That is the comfortable diagnosis, because it locates the error outside the model, and it is also sometimes true, which is what makes it dangerous.

The Recalibration Is A Deletion

What I actually did first was the thing you do when you have no hypothesis, which is to drop each of the eleven inputs in turn and re-run. This is not clever and it is not a method, it is a loop, and a graduate student wrote it in an afternoon in 2014 and it has been in the repository ever since. Ten of the eleven drops made the model worse, by between a tenth of a point and 1.9. The eleventh cut the mean absolute error from 4.1 to 2.0.

The eleventh is institutional trust elasticity. It is mine. I defined it in 2015, I named it, I have three papers on it, the most recent of which has been cited four hundred and sixteen times, and I have sat on two panels where I described it as the variable the field was missing. It is measurable, the measurement is good, the instrument is stable, and it is correlated with everything you would expect it to be correlated with. It simply does not predict the thing I built it to predict, and the model is better without it, and those are two different sentences that my field does not reliably distinguish.

"A variable can be real, well measured, theoretically motivated, and correlated with the outcome, and still make a forecast worse. We do not have a word for that. I have been in this discipline for twenty-six years and I am telling you we do not have a word for it, which is why I did not look." The author, to her own department's methods workshop, September 24

It Did Not Stop Working. It Never Worked

This is the part I would like read twice, because it is the part that cost me the summer. Having found that the term hurt the model in the last cycle, I assumed the term had stopped working, which would have been an interesting finding about the electorate. So I re-ran the four earlier cycles without it. The model is better without it in all four. Not dramatically: 2.4 down to 2.1, 1.9 to 1.8, 3.0 to 2.6, 2.2 to 2.0. Small, consistent, and in the same direction every time, which is the shape of a term that is adding noise and not signal.

So there was never a period in which it worked. There was a period in which the other ten inputs were strong enough that the noise did not show, and that period lasted twelve years, and I spent it believing the model was good because of its weakest component rather than in spite of it. The last cycle did not break the model. It was the first cycle in which the other ten were weak enough that you could see what the eleventh had always been doing. I would not have looked if the error had been 3.5. I looked because it was 7.6, and the only reason this finding exists is that I lost badly enough to open the loop.

The Reviewer Asked Me To Put It Back, And The Reviewer Is Right

I submitted this in August. Reviewer two, who is anonymous and who I am confident is somebody whose work I have been rude about in print, asked for a robustness appendix showing the model with the term retained, on the ground that a paper claiming a term is harmful must show the harm under alternative specifications rather than assert it. This is correct. It is exactly what I would have demanded, and have demanded, of a paper arguing the reverse. So the published version of my paper carries a full appendix running my model with institutional trust elasticity in it, under six specifications, and in one of the six, with a particular lag structure I would not defend, the term helps by a tenth of a point.

I want to record what happened next, because it is the whole lesson. Two people have already written to me about that specification. Neither of them wrote to me about the main result. The appendix that exists because the reviewer correctly insisted on it is now the part of the paper being cited back at me by people who would like the term kept, and they are citing it accurately. I put it there. I was obliged to put it there. A robustness check honestly reported will always contain the best available case against your conclusion, because that is what it is for, and the discipline has no mechanism at all for distinguishing one favourable cell in a six-cell table from a result.

What I Am Not Going To Say

I am not going to say that models are useless, because a model that predicts to two points is useful and mine now does. I am not going to say the field is broken, because the loop that found this was written in the field, by a student, and has been publicly available in my own repository for twelve years, and anybody could have run it on my model, and the reason nobody did is that it is my model and I am the one who is supposed to run it. I am not going to say I am humbled. I am irritated, which is more honest and more useful, and the irritation is specifically that the finding was eleven keystrokes away for eleven years.

What I will say is that the recalibration I promised in June is complete and it made the model shorter. There is nothing in this column about a changing electorate, no new term, no appeal to complexity, and no interesting story about the voters. Eleven inputs became ten. The error halved. My name came off the model and stayed on the paper, which is the correct distribution and not the one I would have chosen.

Author's note: the spreadsheet now opens with considerably less hostility than it did in June, and I attribute that entirely to there being one fewer column in it. For the record, I continue to believe institutional trust elasticity measures something real and worth knowing, and I will go on collecting it, and I will not go on forecasting with it. Those are compatible positions and the fact that I have to say so in a note is the finding underneath the finding. My June column said the real story was in footnotes three, seven and eleven. In this one the real story is in a deletion, which cannot be footnoted, and I have had to put it in the body, where I can be held to it.