The Language Models Caught in the Middle of Propaganda Wars

State governments are concerned about language models spouting propaganda promoted by foreign adversaries.
The Estonian Language Institute has released a new benchmark ranking dozens of language models on their ability to avoid taking positions on topics used in Russian strategic narratives.
Anthropic's Claude models performed the best on the benchmark, with various recent versions taking six of the top 10 spots.
Open-weight models, including Nvidia's Nemotron and Alibaba's Qwen, showed strong results comparable to Anthropic's best models.
Recent frontier models showed a much stronger tendency to resist Russian propaganda than models from just a few years ago.

Maria, a 34-year-old Estonian journalist, has seen firsthand the impact of Russian propaganda on her country's public discourse. As she delves into the world of language models, she's struck by the realization that these AI systems can either perpetuate or resist the spread of misinformation. The Estonian Language Institute's new benchmark is a crucial step in understanding the propaganda resistance of language models, and its findings have significant implications for the future of AI and global communication.

The Full Story of Language Models and Propaganda Resistance

The Estonian Language Institute's benchmark is the result of a collaborative effort with the volunteer-run Estonian defense collective Propastop. Together, they identified 14 broad categories of Russian influence operations and developed separate questions to test the language models' ability to resist propaganda. The questions were provided in English, Estonian, and Russian, and judged by a separate AI model calibrated to align with Propastop experts.

Historical Precedent: The Evolution of Language Models and Propaganda

Language models have come a long way since their inception, and their ability to resist propaganda has improved significantly over time. However, this improvement is not uniform across all language model makers. Google's most propaganda-resistant language model, Gemini 2.5 Pro, is nearly a year old and has only reached a mean score of 82 on the benchmark. In contrast, Anthropic's Claude models have consistently performed well, with various recent versions taking six of the top 10 spots on the benchmark.

Real-World Impact: The Consequences of Language Models Spreading Propaganda

The spread of propaganda through language models can have serious real-world consequences. In Estonia, for example, Russian influence operations have been used to sway public opinion on issues such as NATO membership and the annexation of Crimea. The Estonian Language Institute's benchmark is an important step in understanding the role of language models in perpetuating or resisting these influence operations.

Expert Analysis and Scenarios: What Comes Next for Language Models and Propaganda Resistance

Experts agree that the development of language models that can resist propaganda is crucial for the future of global communication. As language models become increasingly ubiquitous, their ability to perpetuate or resist misinformation will have significant implications for public discourse and international relations.

The Russian government is seeking to influence AI models by projecting specific sociopolitical positions that are culturally sensitive to Russia's viewpoints. This is a complex issue that requires a nuanced understanding of the intersection of technology, culture, and politics.

What This Means for You: The Practical Implications of Language Models and Propaganda Resistance

So what does this mean for you? If you're a journalist like Maria, you need to be aware of the potential for language models to spread propaganda and take steps to verify the information they provide. If you're a developer, you need to consider the potential consequences of your language models and take steps to ensure they are designed with propaganda resistance in mind. And if you're a concerned citizen, you need to stay informed about the latest developments in language models and propaganda resistance.

Frequently Asked Questions

What is the Estonian Language Institute's benchmark?

The Estonian Language Institute's benchmark is a ranking of dozens of language models on their ability to avoid taking positions on topics used in Russian strategic narratives. The benchmark is the result of a collaborative effort with the volunteer-run Estonian defense collective Propastop.

How do language models spread propaganda?

Language models can spread propaganda by providing information that is biased or misleading. This can happen when language models are trained on datasets that contain propaganda or when they are designed to prioritize engagement over accuracy.

What are the consequences of language models spreading propaganda?

The consequences of language models spreading propaganda can be serious, including the spread of misinformation and the manipulation of public opinion. In Estonia, for example, Russian influence operations have been used to sway public opinion on issues such as NATO membership and the annexation of Crimea.

How can language models be designed to resist propaganda?

Language models can be designed to resist propaganda by using datasets that are diverse and representative, and by prioritizing accuracy over engagement. Additionally, language models can be trained to recognize and resist propaganda by using techniques such as adversarial training and robust optimization.

What is the future of language models and propaganda resistance?

The future of language models and propaganda resistance is complex and multifaceted. As language models become increasingly ubiquitous, their ability to perpetuate or resist misinformation will have significant implications for public discourse and international relations. Experts agree that the development of language models that can resist propaganda is crucial for the future of global communication.

As we move forward in this complex and rapidly evolving landscape, it's essential to stay informed about the latest developments in language models and propaganda resistance. By understanding the potential consequences of language models and taking steps to ensure they are designed with propaganda resistance in mind, we can work towards a future where language models are used to promote truth and accuracy, rather than misinformation and manipulation.

The Dashlane Data Breach: What's Really Driving It