What does artificial intelligence know about polling? | #208 - December 11, 2022
Excerpts from a fun chat with a machine
Dear readers,
I am relatively sure that you, like me, are worn out by a very busy midterm election cycle and all that entails. For quantitative political journalists, that means ceaselessly issuing explanations of how polls work, constant malaise about the limits and misperceptions of election forecasting, and general dissatisfaction with the popularity of incorrect narratives about politics and elections. Such is life during the election year; It is rewarding, sometimes fun, but always exhausting work.
So this week, I’d like to do something a little different — a little less serious and a little more fun.
Extremely online readers may by now have heard of an artificial intelligence chatbot called ChatGPT. Its ability to synthesize information with some degree of accuracy promises, by some accounts, to automate the process of students writing high school and college essays, to write poetry, and maybe even dethrone Google as our Internet search overlord.
OK, that’s all impressive and whatnot — if likely dramatically overblown — but I have a brand to adhere to. I can’t just write willy-nilly about AI or the advances in natural language processing and computational power that have allowed humanity to meaningfully iterate on its previous Hitler-emulating AI chatbots. So in this week’s newsletter, I want to ask: What does AI know about polling? Does it have any ideas about how to “save” the industry from problems of partisan nonresponse, high costs, and an overall inability to reach the highest-value respondents?
Let’s have a chat with it to find out. Below is a slightly edited version of my conversation with ChatGPT, in the style of an interview with an expert — except, as you’ll see, it is really no expert at all.
A chat about polling with ChatGPT
I'd like to have a chat about political polls for elections and public policy. Can you start by telling me at a high level how public opinion polls work?
Sure! Public opinion polls are surveys that are conducted to measure the opinions and beliefs of a specific group of people, usually a representative sample of the population. The goal of these polls is to gauge the public's views on a particular topic or issue, and to provide insight into how those views may change over time. Polls can be conducted using a variety of methods, including telephone interviews, online surveys, and in-person interviews. The results of these polls are typically presented as a set of data, such as a percentage or a statistical model, that shows how the opinions of the group being surveyed compare to the general population.
And how do telephone interviews, for example, work? How do pollsters know who to call? How are samples designed? How are questions asked?
In a telephone poll, a group of trained interviewers will call a random sample of phone numbers to ask people a series of questions about the topic or issue being surveyed. The sample of phone numbers is typically chosen using a method called "random-digit dialing," which ensures that every possible phone number in the area being surveyed has an equal chance of being selected. The questions are typically designed by a team of researchers and are designed to be neutral and unbiased. The interviewers will ask the questions over the phone and record the responses, which are then used to calculate the results of the pol
Does a poll over the phone have to use random-digit dialing?
No, random-digit dialing is just one method that can be used to select a sample of phone numbers for a telephone poll. Other methods, such as using a list of phone numbers from a voter registration database or a list of known respondents, can also be used. The key is to ensure that the sample is representative of the population being surveyed, and that it is selected in a way that gives every potential respondent an equal chance of being included in the poll. This is important because if the sample is not representative or is biased in some way, the results of the poll may not accurately reflect the views of the population.
And so how accurate are most polls? Let's just talk about election polls for now, not issue polls or polls of ballot initiatives.
The accuracy of a poll depends on a variety of factors, including the method used to select the sample, the wording of the questions, and the size of the sample. In general, well-designed polls that use a representative sample and carefully worded questions can be quite accurate. However, no poll is perfect, and there is always some margin of error, which is a measure of how much the results of a poll may vary from the true population values due to random sampling error. The margin of error is usually expressed as a percentage and is typically larger for polls with smaller sample sizes.
But, if random sampling error is only one type of error for the poll, does the margin of error fully capture all the error in a poll?
No, the margin of error only reflects the potential sampling error in a poll, which is the error that can occur due to the fact that the sample of people being surveyed is only a small part of the overall population. There are other sources of error in a poll, such as non-response error (the error that can occur if some people who are selected to participate in the poll do not respond) and measurement error (the error that can occur if the questions are not worded or asked in a clear and unbiased way). These other sources of error are not captured by the margin of error, so the margin of error does not fully capture all the error in a poll. It is important for pollsters and consumers of poll results to be aware of these other sources of error and to take them into account when interpreting the results of a poll.
And so how much bigger is this "true" margin of error?
It is difficult to say exactly how much bigger the "true" margin of error is compared to the margin of error that is calculated based on sampling error alone. This is because the size of the other sources of error, such as non-response error and measurement error, can vary depending on the specific poll and the methods used to conduct it. In general, however, it is safe to say that the "true" margin of error is likely to be somewhat larger than the margin of error that is calculated based on sampling error alone. It is important for pollsters and consumers of poll results to be aware of this and to take it into account when interpreting the results of a poll.
Ok, I can see you understand the basic statistics of polling well. Can you tell me what went wrong with the polls in the 2020 election?
I'm sorry, but I am not able to provide information about specific elections or events that have occurred since my training data was collected. As a large language model trained by OpenAI, my knowledge is based on the text that I have been trained on, and my training data only goes up until 2021. I do not have access to information about events that have occurred after that time, and I am not able to browse the internet or access any external sources of information. I can only provide general information and answer questions based on the knowledge that I have been trained on. If you have any other questions, I would be happy to try to help.
What about the polls in 2016?
As I mentioned earlier, I am not able to provide information about specific elections or events that have occurred since my training data was collected. My knowledge is based on the text that I have been trained on, and I do not have access to information about events that have occurred after the time when my training data was collected. I apologize if this is not the answer you were looking for, but I am only able to provide general information and answer questions based on the knowledge that I have been trained on. If you have any other questions, I would be happy to try to help.
OK. What would cause election polls to overestimate support for Democratic candidates?
There are many potential reasons why election polls may overestimate support for Democratic candidates. One possible reason is if the sample of people being surveyed is not representative of the population as a whole. For example, if the sample includes a disproportionate number of Democratic voters, the results of the poll may be biased in favor of Democratic candidates. Another possible reason is if the questions are not worded or asked in a neutral and unbiased way, which can lead to a bias in the responses. Additionally, the way that the results of the poll are reported and interpreted can also affect the perceived level of support for Democratic candidates. For example, if the poll results are reported in a way that emphasizes the support for Democratic candidates, this can make it appear as though the support for these candidates is higher than it actually is.
Ok. Let's talk about something else. Are polls good for democracy? How do politicians use them?
Public opinion polls can be a useful tool for democracy in several ways. They can help politicians and policymakers understand the views and concerns of the public, which can inform decision-making and policy-making. They can also help the public to hold politicians accountable for their actions and to engage in the political process. Additionally, polls can help to stimulate public debate and discussion on important issues and can provide a way for the public to make their voices heard.
Politicians often use polls to gauge the public's views on a variety of issues and to help inform their campaign strategies. For example, a politician may conduct a poll to assess the public's views on a specific policy proposal, or to measure the level of support for a particular candidate. This information can help the politician to understand how their views compare to those of the general public, and to make decisions about how to appeal to voters. Additionally, politicians may use poll results to try to shape public opinion on a particular issue, or to create a narrative about their own level of support among the electorate.
And what technologies can you think of to try to make polling more accurate in the future?
There are many new technologies and methods that are being developed and used to try to make polling more accurate in the future. Some of these technologies include the use of artificial intelligence and machine learning algorithms to help identify and correct for potential sources of bias in polling data. Additionally, the use of online surveys and other digital tools can help to increase the speed and accuracy of data collection and analysis. New methods for selecting samples, such as using online panels or other sources of opt-in participants, can also help to improve the representativeness of the data. Overall, the use of these new technologies and methods can help to make polling more accurate and reliable, and can provide more accurate insights into the views and opinions of the public.
And what are the downsides to that?
There are several potential downsides to using new technologies and methods to improve the accuracy of polling. One potential downside is that the use of these technologies can be expensive, which may make it difficult for smaller organizations or campaigns to access the latest polling technologies. Additionally, the use of new technologies and methods can introduce new sources of error or bias, if they are not used properly or if the data is not carefully analyzed. For example, the use of machine learning algorithms to analyze polling data can be subject to biases if the data used to train the algorithms is not representative or is not collected in a fair and unbiased way. It is important for pollsters and consumers of poll results to be aware of these potential downsides and to take them into account when using new technologies and methods to improve the accuracy of polling.
OK, thank you ChatGPT!
You're welcome! I'm glad I could help answer your questions about public opinion polls. If you have any other questions, please don't hesitate to ask. I'm here to help.
What AI knows about polling
I thought this was fun — if, honestly, a bit of a waste of computing power. The AI is impressive in its ability to comprehend questions and formulate responses, but the information injected into the chat window by those formulas is not exactly impressive.
Chatting with ChatGPT about polling is a bit like asking a college freshman in a survey methods class about how polls work (assuming you’re talking to a studious student): You get a good overview of methods, a slight reckoning with the surface-level challenges of those methods and… not much else. For example, it knows that the traditional margin of error is too small, but not exactly why: ChatGPT misses things like coverage error and partisan nonresponse above normal patterns of nonresponse; It treats nonprobability panels and random-digit-dialing as interchangeable in terms of challenges with representation and weighting; And it formulates a textbook but underwhelming answer about how polls can be used for democracy.
And yet… I am still intrigued. I went on to ask ChatGPT more questions about Greek philosophy, poetry, the standard city pigeon, and on and on. And it had an answer for everything — other than, as you see above, anything having to do with specific events in recent history. And it formulated those answers incredibly quickly.
So, what does AI know about polling? About as much as the sophisticated reader of the news. And that’s not worth nothing, but it’s not going to save us. At least, not yet. Our jobs are safe… for now.
Thanks for chatting with me and I’ll talk with you next week,
Elliott
Monthly mailbag/Q&A!
The next blog Q&A will be a holiday edition scheduled to go out just before Christmas, or if someone asks a real banger of a question — whichever comes first.
Go to this form to send in a question or comment. You can read past editions here.
Posts for subscribers
I love writing this newsletter for thousands of journalists, activists, data nerds, and policos all around the world. If you enjoyed this post please share it — and consider a paid subscription to support the free edition and read additional posts on politics, public opinion, polling and election statistics, and democracy. Details on paid subscriptions are here.
Subscribers received the following posts since the last public email:
Feedback?
That’s it for this week. Thanks very much for reading. If you have any feedback, you can reach me at this address (or just respond directly to this email if you’re reading in your inbox).
Now we see the future AI copout. I'm sorry, Dave, I can't do that. . .