In statistical modeling you don’t really have right or wrong. You have a level of confidence in a model, a level of confidence in your data, and a statistical probability that an event will occur.
So if my model says RFK has a 98% probability of winning, then it is no more right or wrong than Silver’s model?
If so, then probability would be useless. But it isn’t useless. Probability is useful because it can make predictions that can be tested against reality.
In 2016, Silver’s model predicted that Clinton would win. Which was wrong. He knew his model was wrong, because he adjusted his model after 2016. Why change something that is working properly?
You do it by comparing the state voting results to pre-election polling. If the pre-election polling said D+2 and your final result was R+1, then you have to look at your polls and individual polling firms and determine whether some bias is showing up in the results.
Is there selection bias or response bias? You might find that a set of polls is randomly wrong, or you might find that they’re consistently wrong, adding 2 or 3 points in the direction of one party but generally tracking with results across time or geography. In that case, you determine a “house effect,” in that either the people that firm is calling or the people who will talk to them lean 2 to 3 points more Democratic than the electorate.
All of this is explained on the website and it’s kind of a pain to type out on a cellphone while on the toilet.
You are describing how to evaluate polling methods. And I agree: you do this by comparing an actual election outcome (eg statewide vote totals) to the results of your polling method.
But I am not talking about polling methods, I am talking about Silver’s win probability. This is some proprietary method takes other people’s polls as input (Silver is not a pollster) and outputs a number, like 28%. There are many possible ways to combine the poll results, giving different win probabilities. How do we evaluate Silver’s method, separately from the polls?
I think the answer is basically the same: we compare it to an actual election outcome. Silver said Trump had a 28% win probability in 2016, which means he should win 28% of the time. The actual election outcome is that Trump won 100% of his 2016 elections. So as best as we can tell, Silver’s win probability was quite inaccurate.
Now, if we could rerun the 2016 election maybe his estimate would look better over multiple trials. But we can’t do that, all we can ever do is compare 28% to 100%.
But for the person above to say Silver got something wrong because a lower probability event happened is a little silly. It’d be like flipping a coin heads side up twice in a row and saying you’ve disproved statistics because heads twice in a row should only happen 1/4 times.
Silver made a prediction. That’s the deliverable. The prediction was wrong.
Nobody is saying that statistical theory was disproved. But it’s impossible to tell whether Silver applied theory correctly, and it doesn’t even matter. When a Boeing airplane loses a door, that doesn’t disprove physics but it does mean that Boeing got something wrong.
I see what you’re not getting! You are confusing giving the odds with making a prediction and those are very different.
Let’s go back to the coin flips, maybe it’ll make things more clear.
I or Silver might point out there’s a 75% chance anything besides two heads in a row happening (which is accurate.) If, as will happen 1/4 times, two heads in a row does happen, does that somehow mean the odds I gave were wrong?
I or Silver might point out there’s a 75% chance anything besides two heads in a row happening (which is accurate.)
Is it?
Suppose I gave you two coins, which may or may not be weighted. You think they aren’t, and I think they are weighted 2:1 towards heads. Your model predicts one head, and mine predicts two heads.
We toss and get two heads. Does that mean the odds I gave are right? Does it mean the odds you gave are wrong?
In the real world, your odds will depends on your priors, which you can never prove or disprove. If we were working with coins, then we could repeat the experiment and possibly update our priors.
But suppose we only have one chance to toss them, and after which they shatter. In that case, the model we use for the coins, weighted vs unweighted, is just a means to arrive at a prediction. The prediction can be right or wrong, but the internal workings of a one-shot model - including odds - are unfalsifiable. Same with Silver and the 2016 election.
Person B’s predicted outcome was closer to the truth.
Perhaps person A’s prediction would improve if multiple trials were allowed. Perhaps their underlying assumptions are wrong (ie the coins are not unweighted).
Comparing it to Boeing shows you still misunderstand probability. If his model predicts 4 separate elections where each underdog candidate had a 1 in 4 chance of winning. If only 1 of those underdog candidates wins, then the model is likely working. But when that candidate wins everyone will say “but he said it was only a 1 in 4 chance!”. It’s as dumb as people being surprised by rain when it says 25% chance of rain. As long as you only get rain 1/4 of the time with that prediction, then the model is working. Presidential elections are tricky because there are so few of them, they test their models against past data to verify they are working. But it’s just probability, it’s not saying this WILL happen, it’s saying these are the odds at this snapshot in time.
Presidential elections are tricky because there is only one prediction.
Suppose your model says Trump has a 28% chance of winning in 2024, and mine says Trump has a 72% chance of winning in 2024.
There will only be one 2024 election. And suppose Trump loses it.
If that outcome doesn’t tell us anything about the relative strength of our models, then what’s the point of using a model at all? You might as well write a single line of code that spits out “50% Trump”, it is equally useful.
The point of a model is to make a testable prediction. When the TV predicts a 25% chance of rain, that means that it will rain on one fourth of the days that they make such a prediction. It doesn’t have to rain every time.
But Silver only makes a 2016 prediction once, and then he makes a new model for the next election. So he has exactly one chance to get it right.
His model has always been closer state to state, election to election than anyone else’s, which is why people use his models. He is basically using the same model and tweaking it each time, you make it sound like he’s starting over from scratch. When Trump won, none of the prediction models were predicting he would win, but his at least showed a fairly reasonable chance he could. His competitors were forecasting a much more likely Hillary win while he was showing that trump would win basically 3 out of 10 times. In terms of probability that’s not a blowout prediction. His model was working better than competitors. Additionally, he basically predicted the battleground states within a half percentage iirc, that happened to be the difference between a win/loss in some states.
So he has exactly one chance to get it right.
You’re saying it hitting one of those 3 of 10 is “getting it wrong”, that’s the problem with your understanding of probability. By saying that you’re showing that you don’t actually internalize the purpose of a predictive model forecast. It’s not a magic wand, it’s just a predictive tool. That tool is useful if you understand what it’s really saying, instead of extrapolating something it absolutely is not saying. If something says something will happen 3 of 10 times, it happening is not evidence of an issue with the model. A flawless model with ideal inputs can still show a 3 of 10 chance and should hit in 30% of scenarios. Certainly because we have a limited number of elections it’s hard to prove the model, but considering he has come closer than competitors, it certainly seems he knows what he is doing.
It’s forecasting, not a prediction. If the weather forecast said there was a 28% chance of rain tomorrow and then tomorrow it rained would you say the forecast was wrong? You could say that if you want, but the point isn’t to give a definitive prediction of the outcome (because that’s not possible) it’s to give you an idea of what to expect.
If there’s a 28% chance of rain, it doesn’t mean it’s not going to rain, it actually means you might want to consider taking an umbrella with you because there’s a significant probability it will rain. If a batter with a .280 batting average comes to the plate with 2 outs at the bottom of the ninth, that doesn’t mean the game is over. If a politician has a 28% probability of winning an election, it’s not a statement that the politician will definitely lose the election.
If the weather forecast said there was a 28% chance of rain tomorrow and then tomorrow it rained would you say the forecast was wrong?
Is it possible for the forecast to be wrong?
I think so. If you look at all the times the forecast predicts a 28% chance of rain, then it should rain on 28% of those days. If it rained, say, on half the days that the forecast gave a 28% chance of rain then the forecast would be wrong.
With Silver, the same principle applies. Clinton should win at least 50% of the 2016 elections where she has at least a 50% chance of winning. She didn’t.
If Silver kept the same model over multiple elections, then we could look at his probabilities in finer detail. But he doesn’t.
Just for other people reading this thread, the following comments are an excellent case study in how an individual (the above poster) can be so confidently mistaken, even when other posters try to patiently correct them.
May we all be more respectful of our own ignorance.
He didn’t get it wrong. He said the Clinton Trump election was a tight horse race, and Trump had one side of a four sided die.
The state by state data wasn’t far off.
Problem is, people don’t understand statistics.
If someone said Trump had over a 50% probability of winning in 2016, would that be wrong?
In statistical modeling you don’t really have right or wrong. You have a level of confidence in a model, a level of confidence in your data, and a statistical probability that an event will occur.
So if my model says RFK has a 98% probability of winning, then it is no more right or wrong than Silver’s model?
If so, then probability would be useless. But it isn’t useless. Probability is useful because it can make predictions that can be tested against reality.
In 2016, Silver’s model predicted that Clinton would win. Which was wrong. He knew his model was wrong, because he adjusted his model after 2016. Why change something that is working properly?
Yes. But you’d have to run the test repeatedly and see if the outcome, i.e. Clinton winning, happens as often as the model predicts.
But we only get to run an election once. And there is no guarantee that the most likely outcome will happen on the first try.
If you can only run an election once, then how do you determine which of these two results is better (given than Trump won in 2016):
You do it by comparing the state voting results to pre-election polling. If the pre-election polling said D+2 and your final result was R+1, then you have to look at your polls and individual polling firms and determine whether some bias is showing up in the results.
Is there selection bias or response bias? You might find that a set of polls is randomly wrong, or you might find that they’re consistently wrong, adding 2 or 3 points in the direction of one party but generally tracking with results across time or geography. In that case, you determine a “house effect,” in that either the people that firm is calling or the people who will talk to them lean 2 to 3 points more Democratic than the electorate.
All of this is explained on the website and it’s kind of a pain to type out on a cellphone while on the toilet.
You are describing how to evaluate polling methods. And I agree: you do this by comparing an actual election outcome (eg statewide vote totals) to the results of your polling method.
But I am not talking about polling methods, I am talking about Silver’s win probability. This is some proprietary method takes other people’s polls as input (Silver is not a pollster) and outputs a number, like 28%. There are many possible ways to combine the poll results, giving different win probabilities. How do we evaluate Silver’s method, separately from the polls?
I think the answer is basically the same: we compare it to an actual election outcome. Silver said Trump had a 28% win probability in 2016, which means he should win 28% of the time. The actual election outcome is that Trump won 100% of his 2016 elections. So as best as we can tell, Silver’s win probability was quite inaccurate.
Now, if we could rerun the 2016 election maybe his estimate would look better over multiple trials. But we can’t do that, all we can ever do is compare 28% to 100%.
You’re conflating things.
Your model itself can be wrong, absolutely.
But for the person above to say Silver got something wrong because a lower probability event happened is a little silly. It’d be like flipping a coin heads side up twice in a row and saying you’ve disproved statistics because heads twice in a row should only happen 1/4 times.
Silver made a prediction. That’s the deliverable. The prediction was wrong.
Nobody is saying that statistical theory was disproved. But it’s impossible to tell whether Silver applied theory correctly, and it doesn’t even matter. When a Boeing airplane loses a door, that doesn’t disprove physics but it does mean that Boeing got something wrong.
Would you mind restating the prediction?
He predicted Clinton would win. That’s the only reasonable prediction if her win probability was over 50%
If I say a roll of a 6-sided die has a >50% chance of landing on a number above 2, and after a single roll it lands on 2, was I wrong?
If anything, the problem is in the unfalsifiability of the claim.
I see what you’re not getting! You are confusing giving the odds with making a prediction and those are very different.
Let’s go back to the coin flips, maybe it’ll make things more clear.
I or Silver might point out there’s a 75% chance anything besides two heads in a row happening (which is accurate.) If, as will happen 1/4 times, two heads in a row does happen, does that somehow mean the odds I gave were wrong?
Same with Silver and the 2016 election.
Is it?
Suppose I gave you two coins, which may or may not be weighted. You think they aren’t, and I think they are weighted 2:1 towards heads. Your model predicts one head, and mine predicts two heads.
We toss and get two heads. Does that mean the odds I gave are right? Does it mean the odds you gave are wrong?
In the real world, your odds will depends on your priors, which you can never prove or disprove. If we were working with coins, then we could repeat the experiment and possibly update our priors.
But suppose we only have one chance to toss them, and after which they shatter. In that case, the model we use for the coins, weighted vs unweighted, is just a means to arrive at a prediction. The prediction can be right or wrong, but the internal workings of a one-shot model - including odds - are unfalsifiable. Same with Silver and the 2016 election.
The thing is, Nate Silver did not make a prediction about the 2016 race.
He said that Hilary had a higher chance of winning. He didn’t say Hilary was going to win.
How about this:
Two people give the odds for the result of a coin flip of non-weighted coins.
Person A: Heads = 50%, Tails = 50%
Person B: Heads = 75%, Tails = 25%
The result of the coin flip ends up being Heads. Which person had the more accurate model? Did Person A get something wrong?
Person B’s predicted outcome was closer to the truth.
Perhaps person A’s prediction would improve if multiple trials were allowed. Perhaps their underlying assumptions are wrong (ie the coins are not unweighted).
But in this hypothetical scenario of explicitly unweighted coins, Person A was entirely correct in the odds they gave. There’s nothing to improve.
Comparing it to Boeing shows you still misunderstand probability. If his model predicts 4 separate elections where each underdog candidate had a 1 in 4 chance of winning. If only 1 of those underdog candidates wins, then the model is likely working. But when that candidate wins everyone will say “but he said it was only a 1 in 4 chance!”. It’s as dumb as people being surprised by rain when it says 25% chance of rain. As long as you only get rain 1/4 of the time with that prediction, then the model is working. Presidential elections are tricky because there are so few of them, they test their models against past data to verify they are working. But it’s just probability, it’s not saying this WILL happen, it’s saying these are the odds at this snapshot in time.
Presidential elections are tricky because there is only one prediction.
Suppose your model says Trump has a 28% chance of winning in 2024, and mine says Trump has a 72% chance of winning in 2024.
There will only be one 2024 election. And suppose Trump loses it.
If that outcome doesn’t tell us anything about the relative strength of our models, then what’s the point of using a model at all? You might as well write a single line of code that spits out “50% Trump”, it is equally useful.
The point of a model is to make a testable prediction. When the TV predicts a 25% chance of rain, that means that it will rain on one fourth of the days that they make such a prediction. It doesn’t have to rain every time.
But Silver only makes a 2016 prediction once, and then he makes a new model for the next election. So he has exactly one chance to get it right.
His model has always been closer state to state, election to election than anyone else’s, which is why people use his models. He is basically using the same model and tweaking it each time, you make it sound like he’s starting over from scratch. When Trump won, none of the prediction models were predicting he would win, but his at least showed a fairly reasonable chance he could. His competitors were forecasting a much more likely Hillary win while he was showing that trump would win basically 3 out of 10 times. In terms of probability that’s not a blowout prediction. His model was working better than competitors. Additionally, he basically predicted the battleground states within a half percentage iirc, that happened to be the difference between a win/loss in some states.
You’re saying it hitting one of those 3 of 10 is “getting it wrong”, that’s the problem with your understanding of probability. By saying that you’re showing that you don’t actually internalize the purpose of a predictive model forecast. It’s not a magic wand, it’s just a predictive tool. That tool is useful if you understand what it’s really saying, instead of extrapolating something it absolutely is not saying. If something says something will happen 3 of 10 times, it happening is not evidence of an issue with the model. A flawless model with ideal inputs can still show a 3 of 10 chance and should hit in 30% of scenarios. Certainly because we have a limited number of elections it’s hard to prove the model, but considering he has come closer than competitors, it certainly seems he knows what he is doing.
It’s forecasting, not a prediction. If the weather forecast said there was a 28% chance of rain tomorrow and then tomorrow it rained would you say the forecast was wrong? You could say that if you want, but the point isn’t to give a definitive prediction of the outcome (because that’s not possible) it’s to give you an idea of what to expect.
If there’s a 28% chance of rain, it doesn’t mean it’s not going to rain, it actually means you might want to consider taking an umbrella with you because there’s a significant probability it will rain. If a batter with a .280 batting average comes to the plate with 2 outs at the bottom of the ninth, that doesn’t mean the game is over. If a politician has a 28% probability of winning an election, it’s not a statement that the politician will definitely lose the election.
Is it possible for the forecast to be wrong?
I think so. If you look at all the times the forecast predicts a 28% chance of rain, then it should rain on 28% of those days. If it rained, say, on half the days that the forecast gave a 28% chance of rain then the forecast would be wrong.
With Silver, the same principle applies. Clinton should win at least 50% of the 2016 elections where she has at least a 50% chance of winning. She didn’t.
If Silver kept the same model over multiple elections, then we could look at his probabilities in finer detail. But he doesn’t.
Just for other people reading this thread, the following comments are an excellent case study in how an individual (the above poster) can be so confidently mistaken, even when other posters try to patiently correct them.
May we all be more respectful of our own ignorance.
But what if there’s a 28% chance said poster is right?