Duolingo’s mission is to develop the best education in the world and make it universally available. But how do we know whether learners in our courses are achieving their goals? That's where our research team comes in!
We conduct efficacy research about how well Duolingo works. When a learner uses Duolingo to study a new language, we want to be sure they're gaining the skills they need to communicate in that language!
To guide our efficacy research program, we developed a framework to help us evaluate the most important aspects of the learning experience in the app. Our framework also helps us understand how our learners use language outside the app—after all, most people aren’t learning just to earn XP; they’re learning for important real-world purposes!
Here are the four questions we outline in our efficacy framework to help guide our research program.
1. Do learners think Duolingo is fun, engaging, and useful?
This question is all about engagement and whether learners like studying with Duolingo. After all, it doesn't matter how expertly-crafted our lessons are if you don't want to come back to them!
We answer this question by gathering feedback directly from learners. One way we do this is through user experience interviews, where we meet one-on-one with real learners to talk about their experiences, goals, and challenges. We also run surveys with thousands of learners about their attitudes toward their lessons and how useful they feel their course is. And of course, we also analyze the data we get from learners right in the app! For example, are learners returning day after day? How much time do they spend learning? What kinds of lessons do they like to do each day? Each of these questions tells us about how much Duolingo learners love using the app!
2. Do learners achieve the knowledge and skills we're teaching?
Just because we think we're teaching something, doesn't mean our learners are learning it! That’s why we focus on whether our learners master the skills and information we teach in our courses. Testing learners' mastery of course content helps us understand what we're teaching well and where we can improve.
To answer this question, we test how well learners know the vocabulary and grammar taught in their lessons and whether they develop skills in reading, listening, speaking, and writing from using the Duolingo app. We also plan and develop courses to have maximum impact on learning outcomes. For example, we evaluate whether our teaching methods are scientific and based on the latest research, whether our course content is of high quality, and whether our AI models are powerful enough to provide personalized learning that tailors lessons to each learner’s current level and continues to adapt as they improve.
3. Are learners able to use what they learn in the course for real-life communication?
For most learners, getting everything correct in the app is not the ultimate goal of learning. Learners want to use their new language out in the world, in real-life activities! This part of the framework focuses on application and addresses how much learners are able to transfer what they learn within Duolingo to other learning contexts or real-life scenarios.
To answer this question, we conduct research studies using tests and simulated tasks that focus on a learner's performance, or how they use the language in realistic situations. For example, if a learner’s motivation is travel, are they able to perform travel-related tasks (e.g., booking plane tickets, checking into a hotel) with the language they learn on Duolingo?
4. Do learners reach the intended level of proficiency as a result of learning?
All those questions help us evaluate the learning process, but learners' goals with the language matter, too! Ultimately, we want our learners to reach their intended level of proficiency. Duolingo courses are aligned to the Common European Framework of Reference (CEFR), an international language proficiency standard that defines learning goals for Basic (A1 and A2), Independent (B1 and B2), and Proficient (C1 and C2) users, so each section of our courses has an intended proficiency goal. For example, when learners finish the A2 content on Duolingo, we expect them to reach the A2 proficiency level.
To answer this question, we assess learners’ language skills with standardized proficiency tests and interpret their scores based on established proficiency scales. We do this by conducting research studies with learners at different points in the course and having them take independent language tests to assess their language ability. Once learners attain their desired proficiency levels, they'll be able to accomplish their own language learning goals, which might include passing an immigration test, watching original films in the new languages, meeting language requirements for academic programs, traveling to target countries, finding a corporate job, and so much more!
Driven by efficacy
This four-part efficacy framework represents Duolingo’s mission to provide the best education in the world. At Duolingo, our efficacy research answers questions such as how effective our courses are, in which areas and levels we teach best, and where we can continue to improve in order to help our learners succeed! You can learn more about our efficacy research program here.
