we got to get back to doing what's expected of us as free citizens in this democracy that means being tolerant being respectful staying informed and being engaged I'm former Congressman Joe Walsh follow me here and join the millions of Americans who are renewing their social contract with each other the social contract it's on us [Music] hey Labor Day happy Labor Day hope everybody is doing well hope everybody is having a relaxing chill of a uh of a long holiday weekend I hope that everybody well it's still a holiday so I hope everybody is turning all their social media [ __ ] off a little bit I say that as we uh as we um uh broadcast a podcast today on Labor Day that we will promote all week and I'm going to promote this conversation all week back to our weekly conversations and I am so looking forward to this one uh Carl Allen is a researcher uh an analyst of sports and political data uh follow him on Twitter xreal Carl Allen that's Carl with a C I'm fascinated by by this guy because I'm fascinated by polls and I'm fascinated Carl welcome my friend I'm fascinated by how much people don't understand about polls and polling uh and I'm so excited to yank you and grab you on and let's start here you've got a book coming out this week as we speak called the polls weren't wrong Carl welcome work there it is baby came ready Joe came ready I love it and by the way Carl Allan where can people get that book when's it out it is out today it is out today so go to Amazon Amazon today um it's available direct from the publisher tomorrow but I'm encouraging I'm encouraging people to pre-order today because it comes out in waves and uh the shipping date is September 24th so you want to get that as soon possible really really important awesome really important talk about the title of the book is the polls weren't wrong again follow Carl on Twitter or xre Carl Allen that's Carl with a c Carl's also got a substack you ought to be involved with and and uh partaking in it's real Carl Allen at or substack I'm like flumix and blown away and I always have been even though politics is my world by this issue of polls and and let's let's start here the title of your book is the polls weren't wrong but why does it seem like everybody is wrong when they talk about polls today what's going on right no you're you're exactly right so the title of my book comes from the anal is that I did uh starting the with most famously the 2016 Trump Clinton election yeah um when we talk about whether or not the polls were accurate whether or not the polls were right whether or not the polls were wrong we have to have some sort of scientific standard that we measure the polls against right and in this field right now you would be absolutely shocked at how unscientific these polls uh poll analysis the polls themselves not too bad the poll analysis how accurate the polls were is so unscientific so when I talk to folks so my background uh I worked in exercise physiology lab I work um I did quantitative research uh for Major League Baseball NFL the when I talk to people outside of the political Arena yeah and tell them yeah these political analysts these quants who analyze the data they say that the polls the results from the polls should predict pretty pretty closely the result of the election and they say Carl you're you're you're oversimplifying Carl no you're you're you're reading it wrong and I say no look at these quotes from these people and this isn't these aren't quotes for the media to simplify things for folks these are technical articles written in journals that they are objectively and provably incorrect about the function the meaning of pole data so in my book I the the the political analysis comes in about the last quarter right um there's some there's some Foundation wa wait wait wait wait wait wait k back up a sec because I'm loving this and I want to keep up with you hit me over the head if I'm wrong the the the polling data itself you're saying isn't hasn't necessarily been wrong the analysis of that data has been wrong that is spoton if I had to if I had to condense what the book is about to a sentence you just nailed it because okay okay then let me ask this tell me if I'm nailing it still and hit me over the head if I'm not is it then fair to ask or say why are Nate silver 538 in Real Clear Politics why are they all wrong well when it comes to um averaging polls number one um Nate silver largely famous for his poll aggregation but his contributions to the field are much bigger than that real clear politics and I cite them in the book I cite them in the book because they have such a long and transparent track record of averaging polls Nate silver introduced some more Advanced Techniques to how we should average polls how we should wait them but real clear politics their contribution had was look these are the polls we're taking and this is how we're averaging it I I would not say that they that they are um wise in how they average polls it's very easy to manipulate averages if you know how to play the game but and here's the big point there is no objective standard when we talk about bringing scientific standards to this field there is no objective standard for how polls are average so when Nate silver says my poll average says this and this happened in the election and there was this disconnect therefore the polls were wrong no that's not how poll averages work and this is the same thing that happens every election Real Clear Politics 538 real Carl Allen's poll average look we can argue reasonable people can argue about how Poes should be averaged what the best meth methodology should be but at the end of the day this field right now all of the pressure all of the uh criticism is on the pollsters themselves is on the pollsters themselves is that is that is that misplaced that is absolutely misplaced so polling in itself is a science okay let's let's establish this polling is a science but this is where people sometimes um miss or or mis misconstrue and and misunderstand things polling is by definition an inexact science yeah an inexact science okay so in the book chapter I think chapter four very early I introduced this concept called an ideal poll an ideal poll which means if there were no other sources of error non-response uh frame error all of these things that that pollsters talk about and the margin of error itself were the only source of error in the poll what would the polls show meaning and I present this in a very scientific way because I kind of Flip Flip the um analysis from how it is traditionally done here's how it works instead of saying we have these polls what does it say about this population of uh likely voters I say what if you know with 100% certainty who the voters are what if you know with 100 % certainty that none of them will change their mind that no one is undecided all of these other factors and I say if you took a poll from this population what would the polls say yeah and this demonstrates in a very um straightforward Manner and it doesn't oversimplify anything because this is mathematically where pole data Finds Its value if you take a sample from that population what would the pole data say and guess what happens the pole data is erratic the pole data is necessarily imprecise imperfect Carl and that's okay and that's okay because we need and this is where Nate and uh Morris and all these other guys are way way way out of their out of their um uh comfort zone yeah when they are criticizing otherwise trans parent reliable independent pollsters for not being accurate they are forcing them out of the field we need more quality independent pollsters working in this field and the consequences of their analysis which I talk about in the book is that they're they're forcing otherwise good pollsters out of the field because their they think their data should be here when it's here so and that's not how polls work so so so and again Carl H hit me if I'm if I'm not keeping up because I I find this fascinating and and like the average Layman out there I struggle with what pulls mean one of the overriding themes of yours and what I GA garnered from the book is that most of us [ __ ] up screw up in analyzing polls because we look at polls as like like elections we look at polls as predictions of an election outcome where as you say I believe [ __ ] that polls are not predictions of election outcomes like that blows me away are that would blow the away it does it does and when I wrote this book um so I don't know if you know but um rutage and and uh Taylor and Francis who published my book is a very well-known um academic author yeah or publisher publisher and so when they approached me about putting this into a book I said well what do you want who do you want me to write it for they said well we're doing a statistical literacy series a statistical literacy series and I said oh that opens up a whole new um Avenue for me so to get to your to get to your point about polls as predictions this it's not a debate because it's it's not true but in the sense that people argue about it the the debate uh this debate isn't new uh the beginning of political polling or or the beginning of survey data being used for political purposes isn't new it's been around since Gallup Literary Digest Roper like a hundred years ago yeah 80 years ago Joe 80 years ago r and Gallup had a debate Gallup said um the way uh polls are analyzed as predictions keeps us accountable and rer said no no polls are not predictive tools polls are analytical tools polls inform predictions polls are not predictions and that debate 80 years ago what happened nothing nothing this is the only field this is the only scientific field that has maintained their pseudo scientific methods from 80 years ago and it's absolutely blows my mind but Carl like okay KLA Harris versus Donald Trump uh uh a poll will come out today we're still 65 days away from the election but a poll will show Harris up or Trump up and you're you're telling me we can't look at that poll and that that can help us predict who would win in November using polls or viewing polls as predictions is the most fundamental mistake within this field and what you what you said what you said Joe about I'm I'm a Layman you're very well you're very well informed obviously with your background about how polls work and how to use the data but when people who are who are well educated they like polls they they like the data side of things but maybe they're not statisticians um when they talk to me about pole data I don't judge them or talk down to them because they view polls as predictions I want to explain to them and and help them understand why that's not the case and so what I tell people is I say look if you believe that polls should predict the result of the election you are entirely justified in doing so because you have a field of experts who don't know what they're doing telling you that that's what the polls should do G Elliot Morris in his book his most recent book published like two years ago literally said and this is this is a direct quote the Marquette poll predicted the Democratic candidate for governor would win by one point what the the poll that he was the poll that he was referring to by the way the Republican candidate was ahead by one point so he got his his terms backwards but the poll was 474 46 and the winner got to 50 so where did that where did the rest of that come from okay so showan okay so then talk to me my friend by the way everybody uh Carl Allen I'm speaking to follow him on Twitter atre Carl Allen that's Carl with a c uh great new book coming out the polls weren't wrong get it at Amazon today it's it's going to be out just a in fascinating book Carl a poll comes out today that you there it is baby the polls weren't wrong Carl Allen um a poll comes out today Carl that you respect uh and let's say let's say a poll comes out today that you think is a legit poll and it shows Harris up nationally by three or four five points and you think that's a legit poll if you're telling me it's not predictive then what the [ __ ] is the point of that poll what does that poll then tell us a poll that comes out today perfect perfect question let me let me demonstrate for you if I can what that what what what we should do with that poll a quality poll a transparent poll with a good methodology of which there are quite a few neat that's it you look at it you look at it neat and that's it individual polls individual polls um tell us very little um I think in the book I use the term the the graini the gra EST of salt is it's not because that polls don't have value it's because an individual poll by definition by definition can only tell us very little remember and I talked about ideal polls in chapter 4 polls polls in which the only source of error is the margin of error itself well some of my reviewers said but Carl there's no such thing as an ideal poll that's not true there's no such thing as an ideal poll in political application in political applications when these analysts skip all of the basic science that leads up to how to analyze political polls they make this mistake of saying this one poll that says this candidate is up one like glot Morris did in his book this one poll that says this candidate is up one predicts they will win by one no every poll comes with it a margin of error every single poll without Exception by definition so those numbers plus or minus 3% plus or minus 4% neat neat that's all it is it's it's one tiny piece of data okay okay good to answer your question I'm sorry I'm I'm getting no KL keep going okay okay you big jerk I I understand that and I get that one poll doesn't mean squat let's say that tomorrow the average of the 20 best polls shows Harris up three points nationally tomorrow 64 days before the election a range of polls does that tell us anything um it tells us that if we're running a foot race to a th000 meters and kamla Harris is ahead by 3 meters at the 500 meter mark it tells us a little bit I mean it it tells us you like the analogy it tells us a little bit right it's not it's not nothing um and this is why this is why I advocate for more polling better polling independent polling which analysts like silver and Morris um and and a lot of the major well-known experts within the field outright dismiss is because we need a lot of good data to piece together that average that you're talking about yeah because as we talked about there's no um objective way to take an average we can take the most recent polls we can take the most recent high quality polls however you define high quality so um the analogy that I love to use not just because people can relate to it but because it is literally accurate is if you take a snapshot of a foot race we just got past the Olympics the par Olympics are still going on if you take a snapshot of a foot race and you blur it up a blurry snapshot what does that tell us about who's going to win not nothing it doesn't tell us nothing if I had to choose between guessing about who will win with no other data or a blurry snapshot of a race I I'll take the blurry snapshot I can work with that but again just because someone is ahead by whatever poll average that you're using does not predict that they will win and Joe this is this is the hard part this is the last the last thing really quick um the hard part for people to understand is that even if a poll says someone is ahead right now because of factors like the margin of error it's entirely possible because of that blurry snapshot that they're not actually even ahead so when we talk about how close pole data can be within the margin of error so to speak um the the blurry snapshot analogy works on very many levels here so then okay Carl I and I do love that that the the blurry snapshot of a foot race and at the 100 meter foot race and at the 50 meter Point per you you've got one of the racers is a stride and a half ahead of the other one exactly not predictive of who's gonna win but can't you say kind of sort of like like that person who's a stride in a half ahead at 50 meters he's got a much better chance of winning than if he were three strides behind at 50 Metter correct so there's there's some predictive value there no yes yes so here's here's the difference between a snapshot stride and a half ahead and a blurry snapshot a stride and a half ahead plus or minus two and a half strides got it got it is that is that we cannot say with certainty that even at this snapshot that they are ahead if if we could say with certainty that they were ahead right now then that offers a huge amount of predictive value and that's why we take pole averages in the first place and can we ever get there could we ever get there Carl where you could predict with certainty that that guy is a stride and a half ahead at 50 meters or is that I I guess eliminate the margin of error no no because again talking about the discomfort so a lot of people aren't comfortable with numbers they don't like numbers the book is not too technical on that end but there are a few fundamental concepts that we have to understand about pole data and number one is this is a science inferential statistics the umbrella under which survey data Falls is a science but it is by definition an inexact science so we have to we have to be comfortable with our it we have to eliminate this idea in our minds that the poll numbers that we see which we only ever really see the top lines right which is another thing that it's I understand why we have to do it but if a poll says I think you use the example KLA Harris up three let's say those numbers are 47% to 44% we have to be comfortable in our minds and saying okay those are the numbers but if we apply the margin of error to this plus or minus let's say 3% well Harris could be as low as 4445 Trump could be as high as 46 47 and this imprecision is part of the value not not it's not a weakness of pole data it's a value of pole data because the only way we can eliminate the margin of error is take a census literally ask everyone yeah and and and obviously the um that's you know math in political applications it's impossible when I take when I when I do experiments and I take a poll of high school it's very easy to do a census thousand students I knock it out in an afternoon so so Carl the election then happens and then people look back and say ah see those polls back then they were wrong when you AR are arguing no the analysis was wrong because you don't use that poll number from a month ago to tell you who's going to win so you can't say the polls were wrong because you they're not predictive of the outcome to begin with man now you're getting in my wheelhouse okay so this this hindsight if only we would have um this hindsight bias of if we had done this differently then our numbers would have been better this is where the pseudo science comes in this is where the pseudo science comes in so as a researcher when I worked in in an exercise physiology uh when I did um did reports did data for for uh sports teams we have to be able as researchers to identify confounding variables identify confounding variables and what I mean by that is when we take observation a and we see result B we want to see how closely these these numbers or the this this data lines up right yeah but in any good research there's going to be some disconnect here between A and B between what we observe in the result so identifying those confounding variables is not to be too mean here uh this is Middle School level science uh this is very basic this is very basic stuff in election in an election and I and and this is very um this is a very important part of the book in an election there are at least two confounding variables between any poll I don't care if it's a day before a week before month before or 60 days before and the election there are at least two confounding variables number one how undecided voters decide if if there are 10% voters undecided if there are 2% voters undecided we at the poll data doesn't tell us it doesn't try to tell us how they will decide that's number one that's a confounding variable confounding variable number two if voters who say I will Vote for This candidate change their mind between when the poll was taken and in the election now um the cover of the book and I encourage people to have a have a look at it when they get a chance the cover of the book demonstrates this very clearly and and what I say is 538 says this should not exist don't look at this chart um those two confounding variables how undecided decide how people change their mind if people change their mind in our modern polarized mindset of Democrats Republicans and this is an area that you can speak to even better than me because you former Republican you talk to Republican and traditionally conservative voters who say look maybe I supported Trump in 2016 I don't anymore I voted Republican all my life and now I don't and our polarized mindset we say what person who says in a poll that they would vote for Trump would now vote for KLA Harris or vice versa but this is an oversimplification remember remember in 2016 we had these guys this guy uh Gary Johnson yeah ran ran for president yeah um we had uh a very substantive third party push and then on the cover of my book this man Evan McMullen yeah remember him yeah now MC Mullen Nationwide only pulled one or two% of the vote but in Utah his home state he was polling 20 25 even 30% right and so guess guess what 538 guess what these professional alleged scientific analysts do with that data guess tell me they throw it out H they they say these third parties don't matter and from a scientific perspective when you look at a poll and the poll says uh MC Mullen has approximately 25% support plus or minus the margin of error right well what does that mean it means if you ask everyone in that population right now he would probably get somewhere within that margin of error but here's what happens close to the election yeah when people say oh man I don't know if McMullen can win Utah and I really don't want Hillary to win Utah those people are like look look I don't like Trump that much but at least he's not Hillary and then they Chang their mind over here a week before the election they said I think I'm going to support McMullen election day rolls around it comes time to check a ballot and then they say I can't do it I want my vote to count and this third party phenomenon very unique to the United States very unique to the United States but consistently third parties under form their poll numbers yes consistently third parties underperform their poll numbers now that doesn't mean the polls were wrong it means between the time the person was asked the question and election day they changed their mind that's that happens yeah and Carl that's that's really cool information though you called them confounding factors or variables how one decided decide and how people change their minds it and Carl I think then so then there's a bunch of polling we have an election and then the the Nate Silvers and all the experts of the world they R they they then try to reconfigure their machine for the next election to make sure that their polling's more accurate and it's all based on they want to be more predictive of the result and so every cycle we go through this thing where they [ __ ] here and [ __ ] that and you're saying that whole [ __ ] process is flawed because polls can never accurately predict and they their analysis their their methodology is basically them spinning around and chasing their asses trying to figure out what would have made things better last time and and they think oh if we had been right about this let's apply this to the next election and it's just this constant game of chasing their ass and playing this hindsight game you're saying that's impossible it's it's ignorant it's yeah it's not meant to predict the result correct so here's the best example that I can give here's one of the well one of I don't want to say the best but one of the best examples that I can give um in 2016 uh the day after the election 538 put out an article the polls were wrong uh Harry inton was on that was on that uh piece and uh he now works for CNN telling uh misinforming people about polls the uh that the day after the election they said the polls were wrong and and how did they analyze that they took the poll averages their poll averages which obviously can't possibly be wrong they took their poll averages compared it to the election and said based on this and this alone the polls were wrong and what happened we got more data after the election and we found a few a few huge things talking about confounding variables in various swing States where the polls were allegedly wrong 60 to 70% of undecided voters went for Trump the the methodology used and this is another thing that blows my my friends who work in other fields like scientific Fields blows their minds the analysis the methodology used by every reputable the consensus of the experts in the field they say we can't assume undecided split 5050 that's how they judge poll accuracy they say we Bas if this assumption is true then this is what we would see and as a scientist or you know someone who understands middle school science at least um I say hold on there's there's a huge um potential for confounding here uh how did undecided vote 65% for Trump are you kidding me so this number that alone undecideds alone swung the margins 2% 2 % in a swing state is a lot then well they say well 2% wasn't enough so we can we can pretend that our calculations even though our calculations were objectively incorrect we can pretend they're good enough no there's more third party voters people who who uh said that they would support mostly G uh the libertarian candidate uh Johnson 60% of those voters changed their mind they voted for Trump this is these are two huge confounding variables that swung the numbers so the analysis that they did they assumed 5050 undecided they assumed no one changed their mind Jo they assume no one changed their mind can you believe this nonsense this is what masquerades this is what um passes for advanced analysis are you kidding me so so so Carl is it then accurate to say that Carl Allen would say when it comes to Nate silver and 538 and Real Clear Politics blah blah blah blah blah it's not that their polling is necessarily wrong their analysis is wrong the way they analyze polls their polls is wrong that is absolutely correct so so then Carl if polls are not predictions of election out comes what is the value of polling today like a a good poll that comes out today what's the value of that poll what's the value then given that you understand that someone understands what the poll data means which is to say the blurry snapshot analogy understanding the margin of error uh the importance of a poll average and not looking at an individual poll yeah assuming that we understand some of these basic concepts a a very good analyst a very good analyst can produce a decent forecast okay ah a forecast yes so there's a huge huge distinction so some people think a poll and a forecast some people think they're the same um which is not correct some people think they're very very close not really um the analogy or the quote that I use in the book and this is because I I uh love loved his comedy uh Mitch Hedberg uh has a wonderful bit a wonderful bit about um he's a comedian but people always want him to act like oh you're a good comedian can you write me a script he's like that's not fair I worked my ass off to become a really good comedian that's like asking a chef who worked his whole life to become a chef oh you're a chef can you farm these fields these fields are kind of under the same umbrella but asking a chef to farm or a farmer to cook is like asking a holster to forecast these are very different fields we have to understand the distinction so I love Carl that's so cool and you've got this extended quote in the book and I'm G to read it because I love it but I will not give the slightest consideration to the falsehood that failure to predict an eventual result is evidence of a poll's wrongness nor should anyone because it deserves none until we can all agree with what is true polls are not predictions and polls do not try to predict the eventual margin nor that do they attempt to estimate uh winning probability the debate is meaningless and then I love this line a good poll plus a bad forecast do not equal a bad poll yes correct a good poll plus a bad forecast does not equal a bad poll and you're welcome to disagree because you're welcome to be wrong but these are not uh qualifications that I'm comfortable with in a field that claims to be scientific and so I love that you pulled that quote because a lot of people who who I would say are exp and this is something that I really need to point out because I've been critical of silver and Morris and a lot of These Guys these guys are objectively smart these guys are objectively talented um to say that they don't do valuable work would be um would would not be respectful or true but a lot of these people who are very very smart high in the field Etc they want to debate up here they say well let's talk about frequentism versus uh beian fre uh analysis they want to debate up here no no no we're starting at the beginning unless and until we can agree on what is objectively true that is is polls are not predictions of Elections polls do not try to attempt polls do not attempt to try to predict the margin of the election nor do polls or pollsters declare favorites until we can agree on this very very basic Baseline um truth which disproves the analysis done by most everyone in this field until we can agree on those fundamental facts there's no debate to be had it's like trying to debate um astrophysics with someone who doesn't understand arithmetic so not happening so Carl uh by the way you're going to come on again before this election because I I want you on I want to talk to you for two and a half hours um two two other two other let's end with two other questions um how then and again individual polls everybody needs to be smart to those uh uh an aggregate a range of polls for the average person out there listening to us right now Carl how should polling data be understood wonderful question number one number one this is the first thing that I that I'll point out forget the margin not the margin of error forget the margin which is I in the book I call it the spread the difference between the top two candidates forget it throw it out get it out of your vocabulary it is a pseudo scientific metric that proclaims the the the the perceived value of this metric is who is ahead and by how much and it fails both of those tests it does not pass either of those tests even for right now forget predictive value even for right now it doesn't pass that test get it out of your get it out of your system uh get it out of your vocabulary um number two look at the numbers in the poll average the numbers not not the not the margin not to spread the numbers um if Harris is getting 47 48 49 plus or minus 3% in your head right off the bat she could be as low as 45 she could be as high as 55 or 53 or whatever yeah um plus or minus three same with Trump and so what happens when people start applying these scientific which I say they're scientific but they're really fundamental statistical literacy kind of going back to um the original uh home of the book these fundamental plus or minus 3% on a poll average kind of gives you a perspective of like man Anything could happen at this point yeah you know maybe Harris maybe Harris is favored which my forecast says uh 538 I think says maybe Trump is favored which which silver just updated his forecast his forecast goes like this but now he has Trump favored who who should be favored is a is is a respectable debate that that people can have but at the end of the day Joe what you're doing what you're doing is more important to what I'm doing in terms of getting talking to people getting people out to vote the I believe and I may have it may have been edited by myself or the publisher but I believe the last word of the book is vote and at the end of the day we have to treat regardless of what smart people or what people you think are smart yeah uh what good at numbers say about probability we have to vote because if if we go into this with a mindset of oh 99% up by eight whatever no anything can happen between this poll and the election we have to we have to put up numbers so Carl my my last quick and I want a 30 second answer um my last Quick One and we're gonna do this again because I could go with you for three and a half hours from your perspective give me a 30 second answer what does the Hing data tell you right now about where the race for president is between Harris and Trump 30 second answer um and I'll borrow an answer that I saw another analyst use uh in the most recent UK election everything hinges on this 8% undecided tell give or take um and it depend it varies on State and things like that um but if things proceed as they're proceeding which you could argue is the most reasonable hypothesis that we can make as as forecasters or or analysts if things proceed how they're proceeding and there aren't any uh Comey moments between now and the election um it all still rests on this 8% undecided and that's where get out the vote campaigns matter that's where work that you're doing matters that's where um that's where everything is going to make a difference because 6040 versus 5050 makes can swing the election no joke that voice that voice belongs to Carl Allen a researcher longtime analyst of sports and political data I just fascinating dude please everybody listening follow him on Twitter atre Carl Allen that's with a c real Carl Allen he's got a book coming out right now there it is the polls weren't wrong get your hands on that book uh uh subscribe do a substack real Carl allen. substack do.com Carl you're the best I'm having you on again thank you brother man Joe thank you so much for having me sorry the video quality here is not great I don't look as good as you at least not in this light but Joe thank you again thanks so much for having me on thanks all good Keith thank you everybody be brave remember to listen share and follow the social contract with Joe Walsh on Apple podcasts Spotify and everywhere great podcasts are found and be sure to leave a five-star review this has been the social contract with Joe Walsh