Reporting bugs is fast, but fixing them was slow. At Duolingo, admin and beta users can shake their phone to open a bug report form. We call these Shake-to-Report, or STR, bugs. Reports submitted by admin users automatically become Jira tickets for engineers to investigate. These STR Jira tickets include lots of key debugging information like a screenshot, logs, device metadata, and more. With more tickets coming in than we could resolve, our bug backlog was ever growing.
Insight #1: We need help… so let’s throw AI at it
I’m on the DevX AI team, whose mission is to accelerate developer productivity using AI, so of course we decided to throw AI at it. We started with a simple prompt: “Claude, fix this bug.”
It turns out there’s a lot more to fixing a bug than just making the code change. Engineers first need to piece together what happened. That might mean digging through logs, reconstructing the bug reporter’s activity, checking which A/B experiments were enabled, searching through the codebase for the relevant code, grabbing a device and trying to reproduce the bug, investigating pull requests that recently touched the feature, or even trying to track down the engineer who implemented the feature in the first place.
These were all processes Duos were already doing manually to fix bugs. Asking Claude to “fix this bug” meant implicitly asking it to work through that entire process without ever explaining the steps engineers already know to take.
Insight #2: Fixing bugs is hard… so let’s make it easier
What if AI did the investigative work engineers were already doing by hand? We leaned into what AI is fast at Duolingo, so engineers can focus on what humans are good at.
We built AI STR Fix using the Claude Agent SDK, with Sonnet handling most tasks and Opus taking on harder ones. For each task, we provide a system prompt, a user prompt, a set of helpful skills, and a working directory containing the Jira ticket’s information, plus a copy of the relevant repository. As AI STR Fix progresses, we document each step’s findings as files to the working directory (not the prompt) so later agents can build on earlier work without polluting their context window, much like how engineers use what they’ve learned to guide the next step of an investigation.
AI STR Fix automates the grunt work engineers were doing manually to gather what we call AI context and post it to the STR ticket. Managing context is essential for AI agents, but the “AI context” that AI STR Fix gathers helps humans and agents alike triage, investigate, and fix bugs.
Finding the relevant code context
Engineers told us that just finding where to make a change could take hours, particularly in unfamiliar parts of the app. AI STR Fix surfaces and explains the parts of the codebase most likely relevant to the bug. That’s less time digging through the codebase and more time to fix the bug.
Reconstructing relevant user activity
Engineers often sift through thousands of tracking events to understand what happened leading up to a bug report. We don’t want to pay to have AI agents sift through all those events either. To avoid spending tokens on clearly irrelevant events, we apply a heuristic pre-filter: Keep events from the relevant client and backend, prioritize errors and events closest to the report, and collapse repetitive events. The agent then turns the filtered events into a focused timeline, giving engineers more time to fix the bug. That’s less time digging through thousands of tracking events and more time to fix the bug.
Identifying relevant experiments
Duolingo loves a good A/B test. Most client code changes at Duolingo are gated behind experiments. If a bug is caused by an experiment, shutting down the experiment is a quick fix. AI STR Fix gathers the reporter’s experiment information at the time of the report and again applies a heuristic pre-filter to avoid wasting tokens on irrelevant candidates. We drop experiments targeting only a different client—an Android experiment wouldn’t cause an iOS-only bug—and keep only experiments that were running or launched when the bug was reported. The agent then examines experiment descriptions and code to identify relevant experiments. That’s less time digging through 100s of experiments and more time to fix the bug.
Finding relevant pull requests
We merge A LOT of pull requests at Duolingo, so having an agent examine every one would be expensive. Instead, AI STR Fix gets to benefit from work it’s already done. AI STR Fix uses the relevant code files identified earlier to narrow the search to recent changes touching those files. The agent then examines the associated pull requests for connections to the bug. Sometimes it finds a likely cause. Other times it surfaces an existing fix, helping engineers close a stale bug ticket. That’s less time digging through git blame and more time to fix the bug (or, if it has already been fixed, more time to fix other bugs).
Generating reproduction steps
AI STR Fix combines relevant user activity, relevant code, and experiment information to determine what the reporter was doing, where in the app the bug occurred, and which experiment states are needed to reproduce it. It then generates step-by-step instructions, using the app’s debug tools to get the user there faster. A human or downstream AI agent can follow those steps to try to reproduce the issue. That’s less time guessing how to trigger the bug, and more time to fix the bug.
Suggesting a relevant engineer
With nearly 1,000 Duos at Duolingo, it isn’t always clear who should own a bug. AI STR Fix recommends someone familiar with the affected area by considering who frequently modifies the code context, authored relevant pull requests, or contributed to relevant experiments. That’s less time tracking down the right engineer and more time to fix the bug.
Insight #3: The easier bugs are to fix, the more bugs AI can fix
All that AI context makes fixing bugs easier for both humans and agents. The initial AI STR Fix workflow was straightforward: gather the AI context and ask the agent to implement a fix. This gives engineers a starting point instead of starting from scratch and shifts their role from writing the fix to verifying and reviewing one.
As we used it, we found more manual steps to automate.
We added a self-review step so the agent would check its work for obvious mistakes, broken tests, and downstream effects before handing it to an engineer.
We added a Fix CI stage to monitor CI and iterate on failures.
We also found that some changes didn’t follow the conventions of the repository or feature, creating extra work for reviewers. We added a feature research step before implementation so the agent could study how the feature worked and how its change should fit into the existing code.
Finally, we added automatic beta builds. Once the fix branch is ready, AI STR Fix automatically kicks off a beta build. Anyone can install it and verify the fix, meaning anyone can fix bugs.
Insight #4: Fixing bugs is hard
AI STR Fix has fixed release-blocking bugs. It automates hours of manual work on each STR bug ticket it runs on. Teams are now adding their own AI context steps to automate manual triage work specific to the bugs they own.
AI STR Fix is all about making the lives of engineers easier. We took the tedious processes engineers were doing manually and passed them off to AI, so engineers can get to the interesting part faster.
If you want to work at a place where AI helps engineers spend less time on tedious work and more time solving fascinating problems, we’re hiring!