For most people, high-stakes exams are high-stress situations. But it doesn’t have to be this way!
At Duolingo, we learned a long time ago that the best way to do something challenging is to make it fun — and we take this same approach to the Duolingo English Test. Just as we are learner first, we are also test taker first; we are committed to ensuring that our exam is as low-stress (and even fun!) as possible, all while ensuring the test is accurate and secure.
So how did we create a rigorous, high-stakes exam that’s not a chore to take? In this post, we’ll take a look at how our unique approach to test development lets us prioritize the test-taker experience — or TTX — in every decision we make.
Frameworks make tests work
The Duolingo English Test measures English language proficiency, often for the purposes of university admissions. Experts from many fields collaborate to build the test: AI and machine learning engineers, linguists and natural language processing experts, assessment scientists and psychometricians all have a hand in making sure the test does everything it was designed to do, while being the world’s most widely accessible high-stakes exam.
As with all high-stakes exams, our experts rely on assessment frameworks to coordinate their efforts and guide their decision making. You can think of an assessment framework as a blueprint, informing everything from which skills are assessed, to the way in which test items are written, to how exams are scored.
There are many frameworks for language assessment design, and assessment design more generally. These frameworks were developed based on a traditional model of testing (think fixed-form, often paper-based exams), and they offer different perspectives on building and evaluating assessments to ensure they are valid. That is, they function as intended to assess whatever skill or skill set they were designed to — in our case, English language proficiency.
Thinking outside the box
But the Duolingo English Test is not a traditional test and can’t rely on traditional frameworks. Our digital-first, computer-adaptive exam is available online, on demand, 24 hours a day, anywhere in the world. As a first-of-its-kind exam, its theoretical design is entirely novel. So while there are a lot of assessment frameworks out there, none of them provide guidance for the full set of processes and considerations involved in supporting a digital-first and test-taker first exam like the Duolingo English Test.
To accommodate this first-of-its-kind assessment, we went beyond just one framework, and instead created a whole ecosystem. Our test-taker ecosystem guides our whole approach to English language assessment and instead of leaning on just one framework, we've developed a comprehensive and integrated set of frameworks that work together to guide our multidisciplinary team of experts as they develop and evaluate the test.
In this ecosystem, our experts collect and interpret evidence about the test to ensure that it yields its expected impact, or purpose — in our case, accurately assessing and reporting test takers’ English language proficiency.
Our ecosystem provides us with a well-defined set of processes for everything from developing test items to measuring and scoring test-taker responses. It includes frameworks for language assessment design, expanded evidence-centered design, computational psychometrics, and test security.
“Innovative item types on our test may involve multiple interactions, potentially in different modalities, such as speaking and listening ,” says principal assessment scientist Jill Burstein. “The ecosystem supports the Duolingo English Test team’s decisions as they design these complex item types.”
A good TTX
The element that unites this entire ecosystem is the test-taker experience, or TTX. TTX is distinct from UX, or user experience; while UX is typically associated with design elements related to visuals and navigation in digital platforms, we use TTX to refer to the full test-taker experience, from test administration to score reporting.
“Every decision we make about the test considers TTX,” says Burstein. “We don’t want anything about the test to negatively impact test takers — their experience is the center of the ecosystem, and ensuring that they have a positive experience drives all of our decisions.”
We aim to provide a positive TTX at every stage of the test taker’s journey, including free test-readiness resources, a more affordable price point, intuitive UX design, shorter testing time, and fast score turnaround processes. Of course, accessibility and accommodations are critical to ensuring a positive TTX for everyone; Duolingo English Test designers follow federal and industry standards to continually build accessibility and accommodations into the exam.
Fairness plays a role, too: a test taker’s experience can vary across culture, first language, and computer setup. We consider these and other factors when developing test items and when running quality control evaluations on the exam, to ensure that the test is fair for everyone.
Driving innovation
Ensuring a delightful TTX serves Duolingo’s driving mission to lower barriers to education. Making our test affordable, efficient, and even pleasant means more people around the world have the chance to certify their English proficiency. But as Burstein explains, our approach also drives innovation.
“The ecosystem helps the Duolingo English Test team to expand the test's research agenda, by exposing new areas in which the test can grow,” says Burstein. “We can use the ecosystem to systematically work through the set of processes we need to continually innovate and improve our test.”
To learn more about how our ecosystem serves to ensure a positive TTX, check out Burstein et al’s recent white paper, A Theoretical Assessment Ecosystem for a Digital First Assessment.
The full TTX
