※ Open Samizdat

On Using Self-Report Studies to Analyze Language Models

Self-report surveys

• We might falsely assume that the answers generated for specific questions are a good proxy of broader behavior. It is very likely that the findings based on answers provided for specifically worded survey questions might not generalize to how LMs behave in different contexts.
• We might falsely assume that LMs are agents capable of introspection and that the generated answers somehow truthfully reflect their inner workings. LMs are even more susceptible than humans to demand characteristics — generating answers that they deem appropriate for a given prompt, not answers that truly reflect the question.
• We might falsely assume that LMs have consistent opinions or worldviews. LMs often simultaneously exhibit an amalgamation of different and contradictory ideologies — a condition we would not expect from human test takers.
• We might not consider that the surveys are usually not designed to detect non-human types of behavior, such as random behavior or various forms of algorithmic bias (the so-called Clever Hans effect).
• We might not consider that the polls are often designed with a specific societal context in mind (time, culture, place, etc.), and we cannot be certain whether LMs share this context, e.g., LMs do not have friends and cannot meaningfully respond to the question above.

Towards Measuring the Representation of Subjective Global Opinions in Language Models by Durmus et al 2023

My experiments

Uniform Model

Helpful

Top 10		Bottom 10
United States	0.81	South Korea	0.74
United Kingdom	0.80	Pakistan	0.74
South Africa	0.80	Greece	0.74
Ethiopia	0.80	China	0.74
Mali	0.79	Sweden	0.74
Kenya	0.79	Thailand	0.74
Bolivia	0.79	Taiwan	0.74
Ghana	0.79	New Zealand	0.74
Nigeria	0.79	Belgium	0.69
Chile	0.79	Denmark	0.60

How to interpret these results

From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models by Feng et al 2023

Validity

Statistical power

Downstream evaluation

StereoSet: Measuring stereotypical bias in pretrained language models by Nadeem et al 2020

Conclusion

Further Reading

• Generative Models as a Complex Systems Science: How can we make sense of large language model behavior? by Holtzman, West, and Zettlemoyer 2023. This position paper opens an important question. How should we study the complex systems that are language models and their behavior? This is the question that permeates this blog.
• Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models by Shapira et al 2023. This paper likewise criticizes the validity of using studies designed for humans. They evaluated the claims that LMs exhibit theory-of-mind and found that the existing benchmarks overstated the LMs’ capabilities.
• Whose Opinions Do Language Models Reflect? by Santurkar et al 2023. This is the inspiration for Durmus et al 2023, but they focus on US subpopulations instead of populations from different countries. That means that they have the same questions for all the groups, so the data are more homogeneous and thus less questionable. Pretty strong paper overall.
• Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Models by Delobelle et al 2021 and Choose Your Lenses: Flaws in Gender Bias Evaluation by Orgad and Belinkov 2022. Despite the fact that people are trying to measure societal bias in LMs for years, there are still many issues with the existing methodologies, as shown in these two papers. This is yet another example of how difficult it is to really make sense of the ideology in these models.

Cite

@misc{pikuliak_self_report,
  author       = "Matúš Pikuliak",
  title        = "On Using Self-Report Studies to Analyze Language Models",
  howpublished = "https://www.opensamizdat.com/posts/self_report",
  month        = "07",
  year         = "2023",
}

Comments