Duolingo runs hundreds of microservices for 130M+ monthly active users, with engineers juggling GitHub, Jenkins, Sentry, Grafana, Honeycomb, PagerDuty, Jira, AWS, BigQuery, and more. We saw a chance to connect AI tooling to all of it. That’s how local experiments by a few engineers grew into an AI Slack app used weekly by 30% of the company. It is now open-sourced at github.com/duolingo/slack-ai-agent.
The MCP Journey
Early Adopters (Nov 2024)
In November 2024, Anthropic introduced the Model Context Protocol (MCP)—an open standard that connects AI agents to external tools. A few engineers immediately saw the potential and started wiring up MCP servers in their local editors. But setup was painful: each one required hunting down the right config and credentials, and then hand-editing mcp.json. The barrier was so high that only a handful bothered.
Centralized Setup Page (May 2025)
To lower the barrier, we built a centralized setup page where engineers could pick an MCP server and install it with one click:
But new problems surfaced: Every server had a different runtime (Node, Python, Go, or Docker) and a different transport (stdio, SSE, HTTP). “It works on my machine” became a constant refrain—mismatched npm versions, resource-hungry containers, and zombie processes made the experience fragile.
Standardization (Aug 2025)
We standardized on HTTP-based MCP servers hosted internally. Every server now follows the same configuration pattern: a single auth token and a predictable URL. Setup became dramatically simpler—but we realized that even one click is still too much. Not every engineer wants to configure a local AI environment; most just want answers.
Zero Setup: The Slack App (Sep 2025)
In September 2025, we thought, “If people won’t configure MCP themselves, we can bring MCP to them.” That’s how @DuolingoAI, our Slack app, was born.
Architecture
The Slack app is built on two foundations: the Claude Agent SDK—Anthropic’s agent framework that autonomously determines which tools to query—and Slack Bolt for event handling and conversation management. MCP servers provide structured access to Honeycomb, Grafana, Sentry, PagerDuty, GitHub, Jenkins, Jira, AWS, BigQuery, and more. Adding a new tool means deploying a new MCP server—no changes to the app’s core logic.
The Slack app automatically responds to help desk and incident response channels, each with its own channel-specific context. Write operations—like creating pull requests or filing Jira tickets—are gated behind Approve/Cancel buttons for human oversight. Under the hood, it uses Claude Skills and sub-agents to delegate specialized tasks. Responses include upvote/downvote buttons so engineers can rate them, feeding a signal we use to improve quality over time.
The DuolingoAI Slack App in Action
Help Desk Auto-Responses
The Slack App monitors 20+ help desk Slack channels and automatically responds when it thinks it knows the answer to the person’s question. In the example below, someone came asking about a recurring Cursor login popup issue, and the DuolingoAI Slack App searched internal info and public sources to suggest how to fix it.
Incident Triage
In the example below, a DET PagerDuty alert fired, and the DuolingoAI Slack App investigated using Sentry and Honeycomb, pinpointed a slow endpoint as the likely cause, and suggested mitigation steps. It also identified domain experts in case the on-call engineer needed to escalate.
Human-in-the-Loop Operations
Engineers can ask the bot to create pull requests, file Jira tickets, fix CI, or address review comments—all gated behind “Approve”/“Cancel” buttons. In the example below, once the engineer clicks “Approve,” a Temporal workflow spins up a sandboxed Claude agent that writes the code and pushes a branch. The session stays alive for follow-ups so people can continue iterating on it.
Each Slack thread is an independent agent session. Engineers can run multiple concurrent tasks—one creating a pull request, another fixing CI, another researching a service—directly from their phone, no laptop required.
Lessons Learned
Getting the Slack App to work on a single query was straightforward. Getting it to work reliably across thousands of queries per week for hundreds of people was a different story. Here are a few things we learned:
- AI agents hallucinate without the right context and tools. We mitigate this with channel-specific system prompts and a validation sub-agent that cross-checks links and names before responses go out.
- Security and privacy matter for in-house AI agents. Engineers shouldn’t gain access through the bot that they couldn’t get directly, so we enforce a role-based tool allowlist. We also don’t log DM messages or private-channel content.
- Dogfood aggressively. We build, debug, and analyze the Slack App using the Slack App itself.
- Evals keep agents from regressing. We maintain a benchmark suite that runs on every change, so quality drops get caught before users see them.
- Convenience wins. A Slack App people can just @-mention is far more adoptable than any MCP-setup guide, no matter how polished.
Usage Metrics
Since we introduced the Slack App last September, usage has grown to ~300 weekly active users—roughly 30% of the company:
The Slack App’s upvote rate has stabilized at ~80%:
Open Source
We’ve open-sourced the core framework at github.com/duolingo/slack-ai-agent. If you’re on Slack and Claude, use it as-is. If not, fork it and ask AI to port it.
If you want to work at a place where improving developer productivity is a first-class engineering priority, we’re hiring!