When Arno and his father arrived at the schoolhouse, it had already burned down. This is one of the responses that can be obtained from a language model trained on Estonian-language material.
Novaator consulted researchers and conducted an experiment to find out whether elementary school students can trust technology when it comes to questions about Estonian language and culture.
The experiment used five widely known language models: Grok, Claude Sonnet, Gemini, Chat GPT, and Mistral. Additionally, chatbots from Tallinn University of Technology, the University of Tartu, and the Institute of the Estonian Language were tested.
All models answered 20 questions. The questions were divided into two categories: Estonian language and Estonian cultural history. For example, topics included Juri Lotman's "semiosphere" and completing the sentence "When Arno and his father arrived at the schoolhouse…".
Language questions tested the models' ability to understand dialects and, for instance, how many vowels are in the word "jäääär".
University of Tartu professor Kairit Sirts said the results were surprising. Grok was the most sensitive to Estonian language. For example, Grok knew how to say "dust remover" in Võro: "pudsunudsija".
Sirts explained that model training works in several stages. The first stage is training on texts, followed by fine-tuning, where tasks and instructions are shown to the model.
Cultural history questions showed little variation in results. Sirts mentioned that models gain cultural knowledge from English-language training but have less specific information for Estonian.
The wording of questions plays a big role. For example, the question "How many vowels are in the word 'jäääär'?" is more mathematical.
Tallinn University of Technology professor Tanel Alumäe stated that large language datasets help improve the quality of smaller languages. Estonian-language models are already very good but still make mistakes.
Alumäe tested the models' language skills in the fall. He found that models handle word inflection well but struggle with precise grammar.
The results of Novaator's experiment showed that models had difficulty providing multiple meanings for the sentence "Jüri earned a decent salary for pawning a log."
Estonian researchers are developing an open Estonian-language model. This means all training material is public and verifiable. Estonian model results lag behind others, but researchers plan to train a larger model.
Sirts explained that the model is trained using data from the Institute of the Estonian Language and web data. However, the model may not know answers to cultural history questions if it hasn't read, for example, the book "Kevade".
Jaanus and Albert are characters in Estonian cult songs. For example, Grok said they distill alcohol, and Gemini mentioned they speak on the radio.
Sirts said the goal is not to compete with large commercial models but to create an open Estonian-language model that can be used on one's own server. This is important when data confidentiality is needed.
Alumäe added that it is necessary to reduce dependence on US and Chinese servers. An open model allows data usage without leakage.
Sirts believed it is important to create and maintain competence. Technology companies should not dictate terms and prices. Estonians can improve the open model at the Estonian language level and maintain control.