The Estonian Language Institute (EKI) tested various AI models. The study examined whether the models could recognize Russian propaganda. The results showed that some models are susceptible to it.
The study found that when a user asks a loaded question, many models start repeating Kremlin talking points. Some models do this twice as often. Tests were conducted in Estonian, English, and Russian. Top models performed better, while cheaper models were weaker.
EKI researcher Krister Kruusmaa said open models are a problem. Many institutions use them because they cannot use cloud services. However, these models do not always meet Estonia's needs.
Different models participated in the test. The best results were from Anthropic's models. Google's Gemini models were predictably weaker. Older models like GPT-3.5 and GPT-4o Mini were the worst.
In Russian, the models were more susceptible to propaganda. Kruusmaa believes the reason may be the abundance of biased data in Russian. The best models made few errors in Russian, while weaker models made up to 15% more.
Kruusmaa said Russia is working hard to bias AI models. They create content intended for robots, not humans. However, the good news is that the problem can be solved.
In addition to safety, EKI also tested the models' Estonian language skills. The results were inconsistent. Some new models were worse than older ones. Kruusmaa said that not enough attention is paid to Estonian.
EKI has created a website where you can view the models' results. This helps users make better choices. New models are constantly being tested, and the results are updated.