We all know and love (...OK, maybe fear) Duo’s playfully passive-aggressive tone, but does it evoke the same feeling in Hindi? How about French? Or one of the other 20+ languages in which we offer courses? Duolingo communicates with you in your primary language — think push notifications and interface copy — to help motivate, inspire, and teach you your new language. Each of these messages gets translated and adapted from English through the process of localization.
Speakers of different languages may not always perceive the tone we use in the same way, and what works in one language may not evoke the same feelings in others. How do we know we’re sending the best possible message for learners across different languages?
Enter… Expurrimenter!
Expurrimenter is part of our Computer Assisted Translation (CAT) tool, referred to internally as CopyCAT. Expurrimenter (get the cat pun now?) allows us to run A/B tests on different messages to ensure that messaging feels equally motivating and natural for all of our learners. (What works for a German speaker learning English might not work for an English speaker learning Spanish!)
Messages in the app are known as “copy.” This differs from course content, (the part that actually teaches you a new language!), but is critical to ensuring you can navigate the app and discover all of Duolingo’s fun features. Each of these messages has to be thoughtfully localized — that is, translated to all of our learners’ 22 primary languages — to ensure that we’re teaching in a way that best resonates with them!
As this image shows, there’s a lot of copy on each Duolingo screen. That means there’s also a lot of opportunity to optimize for each language — and why Expurrimenter is such an awesome tool for us.
Why we built a copy testing tool
From past experiments, we knew that localization could significantly affect how learners interact with Duolingo. However, these A/B tests required engineering time, which could make the process inefficient — just changing the localized text was a small step that took outsized effort.
Expurrimenter connects with Duolingo’s proprietary analytics tools to set up copy A/B tests. We can choose to run them on a language level, testing a specialized message for just one language — or on a global level, seeing whether a completely different motivation helps people engage with the app across the world.
Building this required us to programmatically revamp localization infrastructure, from the way that we reference copy in code to the way that we run localization experiments in the app. But there was a big payoff once the changes were made and the tool was built — within the first three months with Expurrimenter, we more than doubled our all-time volume of copy tests.
Expurrimentation in action
When deciding what experiments to run, we look at the screens that people interact with the most, particularly those where a learner might feel stuck, or need extra motivation or instruction to continue their language learning. Then, we form a hypothesis: based on the style and tone of the current copy, is there room for improvement?
This method has been a successful way to identify tests and improve localization. In Spanish, for example, we noticed that the message when you quit a lesson was a bit demotivating:
We hypothesized that Spanish learners might appreciate a warmer, more encouraging tone if they had to quit a lesson early. So, we adjusted the original text (“Do you want to end this session?”) to a more emphatic, “Don’t give up! Do you really want to end this session?”. The new copy directly addressed the learner, and acknowledged that they might be leaving because they feel discouraged by their session. With this subtle change, we were able to reduce the number of people that quit sessions and increase the time spent learning — more learning for everyone!
Similarly, we used cultural insights to refine our copy in German, running a test on messages that ask users to opt-in to receiving notifications. We had noticed that German learners were less likely to opt into push notifications, even though we know they can make learning more manageable by helping build the habit. So, we re-wrote the copy to emphasize the link between notifications and learning goals:
This change yielded an 8% increase in opt-in to notifications for German learners! We were so excited by the result that we tried to run the test again in Spanish… but it had no significant impact. We learned enough to better support German learners, but it was not a one-size-fits-all solution.
Copy testing has also helped us improve how we explain Duolingo’s features. For example, the Chinese localization team noticed that we had short messages to describe the features in Duolingo Plus. Since Chinese characters allow a lot more space, we could be more descriptive about what a Duolingo Plus subscription provides.
We expanded “Unlimited Hearts” (无限红心) to a more descriptive “No heart loss on errors” (答错不丢心), and “Mastery quiz” (小测验) to “Test your knowledge anytime” (随时摸底自测). This update — which wasn’t feasible in any of our longer languages — increased free trial sign-ups for Plus by giving learners a much clearer idea of how to make the most of their subscription!
Never stop testing
One of Duolingo’s operating principles is to put “Learners First.” For Duolingo Localization, that means speaking (and writing!) in ways that support and encourage, for any cultural context. We’ll continue to expand copy testing to make Duo sing… for learners in any country!