As a linguistics and computer science double major, I always wondered about the methods Duolingo uses to teach and how effective those methods are. I had the opportunity to look into exactly that as part of my internship on Duolingo’s Core Learning team. This past summer, I worked with the team to create a tool that visualizes information about Duolingo users’ learning patterns, helping both myself and the company answer the question of how to measure learning. In doing so, we found that the Learning Quiz, a tool that already exists in Duolingo, provides a stable and reasonable measurement of learning.
What is the Learning Quiz?
The Learning Quiz is a language assessment tool built by Duolingo’s learning experts. It consists of a large pool of items testing the beginner, intermediate, and advanced language skills. The quiz assesses the knowledge of vocabulary and grammar, as well the holistic skills of reading and listening. The quiz appears every two weeks and consists of five questions, all created independently of other Duolingo content to avoid biased results. The quiz has no impact on users’ progress, but instead gives users an opportunity to earn some extra XP while giving the company an opportunity to measure learning for these courses. Currently, the quiz is available for all Android users learning English and some Android users learning Spanish.
How does Duolingo use the Learning Quiz to measure learning?
It might seem surprising that five questions are enough to tell how well users are learning a language. While the quiz isn’t enough to analyze a single user’s progress in a language, the data averaged across all users gives us a good estimate of how well the app is teaching overall.
Since we wanted to know how much users learn on Duolingo, we decided to look at data for users who have seen a large amount of Duolingo content.
Another piece of information we used in analyzing the Learning Quiz comes from a question before the Learning Quiz starts. The first time a user sees the Learning Quiz, they are asked to report their knowledge of the language prior to using Duolingo, on a scale of 0 to 10, where 0 means that a user had no prior knowledge, and 10 means that the user is already fluent in the language. We placed users into four ranges of prior knowledge: users with little prior knowledge, who selected 0 through 2, users with intermediate prior knowledge, who selected 3 through 5, users with substantial prior knowledge, who selected 6 through 8, and users with expert prior knowledge, who selected 9 or 10.
To gain insight into how well Duolingo teaches, we looked at data for each range of users, averaging scores for each language concept at each level. These results allowed us to learn about how well our users are learning compared to each other, as well as how well we teach certain levels and concepts.
What did we learn from the Learning Quiz?
Visualizing data from the Learning Quiz revealed that the Learning Quiz is a stable and reasonable measurement of learning. For each month, we visualized data for all users two-thirds of the way through a course or further, and this data showed no unexpected large spikes or drops. Since each month shows data for a semi-random group of users and average scores are similar even with these different groups (shown in the figures below), the Learning Quiz seems to provide a good way to measure overall learning.
In addition to learning about the Learning Quiz, we learned that Duolingo does have a positive impact on users’ language knowledge. The following figures show Learning Quiz scores, on a scale of 0 to 1, for users learning English from Spanish.
These figures tell us a few different things. One insight is that Duolingo seems to teach vocabulary and grammar equally well, since the scores look similar regardless of language level or concept. Another is that Duolingo positively impacts users’ language learning. Across all of these figures, on average, users with little prior knowledge score around 0.6, users with intermediate prior knowledge score around 0.7, and users with substantial prior knowledge score around 0.8. Since these users are at similar points in the course, that means that prior knowledge of the language is not the only factor in Learning Quiz scores. We can also rule out the possibility that users are doing well on the Learning Quiz just by chance, since this data looks at a large enough amount of users that luck shouldn’t be a factor. In other words, if these users were given a ten-question version of the Learning Quiz, users with little to no language knowledge before Duolingo are likely to only miss two more questions than more experienced users, which is a pretty promising find!
What’s next?
Learning scientists at Duolingo have been working on bringing the Learning Quiz to other languages, starting with French, and our engineers are also working to make the Learning Quiz accessible to our iOS and web users. Knowing now that the Learning Quiz offers us meaningful data about how well users are learning, these efforts are even more worth the while. Using the existing Learning Quiz and the improvements to come, we can continue to tailor Duolingo to provide the best education in the world.
