We are always trying to make Duolingo a more effective learning tool. To do so, we first need to understand what our users are learning from Duolingo – and, more importantly, what they’re not learning. Our Learning Assessment team spends all of their time working on this problem. In this post, we will share our approach to measuring learning in the app, as well as some of the challenges that we face and how we are overcoming them.
Language learning is really complicated!
Learning a language requires mastering many different skills. For an activity as simple as ordering a cup of coffee, you have to be able to read the menu to decide what your options are, listen to the server greet you and ask you what you want ("Hi, how are you? What can I get you?"), and then prepare and say a response ("I’m fine, thanks. I’d like a large cappuccino, please"). The conversation can go on, but the point is: as a language learner, you need to learn a large number of words, learn how to put them together correctly, learn to understand what you read and hear, and learn the culturally-appropriate ways to speak to someone. We want Duolingo to teach you all the skills you need to communicate in a new language, so we work hard to ensure that we are measuring the effectiveness of Duolingo across all of these skills.
Duolingo vs. The Classroom
Historically, most language learning has happened in a classroom setting, where all students are taught at the same pace and come to the class with similar levels of prior knowledge. Learners on Duolingo, however, are very different! Because of the flexibility of our app, users can learn anywhere at any time at their own pace. Some learners opt to use Duolingo while commuting, while others use it before going to sleep; some learners use Duolingo for 15 minutes a day, while others use it more than an hour. Similarly, learners come to the app with very different levels of prior experience in their target language. Some learners are true beginners while others use Duolingo to refresh the languages they learned in college. Some learners use Duolingo as the only tool for learning, while others use it as a supplement when they take language classes. Due to these individual variations, it is a big challenge for us to measure how much learners learn from Duolingo. A lot of knowledge has been generated in the field of educational measurement on how to assess learning in a classroom context, but unfortunately much of it is not applicable to Duolingo. This means that we have to get creative!
The Checkpoint Quiz
One tool that we use to measure how much our users are learning is the checkpoint quiz. The checkpoint quiz is embedded between sections of a course as shown in the screenshot below. Learners need to take the checkpoint quiz to unlock the next section of the course.
The exercises in this quiz are designed specifically to help us measure how much vocabulary and grammar our users have learned in the section and how well they are able to integrate this knowledge in reading and listening. If you have ever taken a checkpoint quiz, you might notice that it can be quite difficult. This is because some of the questions test concepts that you have not been taught yet. Each quiz is split into two parts: a “pre-test” and a “post-test.” The pre-test measures how well you know the material in the section that you are about to start. This allows us to measure how much you knew before you started learning the material. The post-test then accounts for how well you know the material in the section you just completed. By comparing how much you knew when you started the section and how much you knew when you finished it, we can measure how much you learned from that section of the course. Of course, this approach does not allow us to exclude learning from other sources during a Duolingo course.
The primary goal of the checkpoint quiz is to assess how well we teach. Currently, we don’t provide scores or feedback to individual learners regarding how well they do. Instead, we combine all our learners’ scores to get a more detailed view of what Duolingo teaches well and what we could improve upon. We then analyze this data in different ways to help us decide where to focus our resources. Every time you complete a checkpoint quiz, you are helping us measure the effectiveness of Duolingo, so that we can make Duolingo a better learning experience for learners like you!