Automating Golden Path upgrades at scale: A journey from manual upgrades to an AI-powered workflow

Earlier this year, the Engineering Studio team manually upgraded multiple Java services to modern versions of the stack, including JDK, Spring Boot, and Gradle. To accelerate this work, we paired OpenRewrite, an automated refactoring tool, with AI and saw promising results. This led us to systematize the process and create an AI-powered workflow to upgrade all of our repos.

Objective

At a high level, the goal of this project is to get all JVM services to follow a “golden path,” also referred to by other companies as a “paved road.” We use the term Golden Path and define it as a set of clear, opinionated standards designed to reduce complexity and ensure consistency across microservices and resources within an organization.

In practice, what this means right now is that services have to be on the following versions:

• Gradle 8
• Kotlin 2.1.20
• Spring Boot 3.x
• Base Library 3.x
• JDK 21
• OpenTelemetry 1.33.6
• Debian Bookworm

(There are some other things involved in the Golden Path, like Galaxy, pre-commit, migrating to workbench etc., but we focused solely on application-level changes.)

Part 1: Manual upgrades with OpenRewrite and AI

We started by using OpenRewrite, a refactoring tool, to automate as much of the upgrade process as possible. In our upgrades, OpenRewrite handled standard elements of the upgrades that had existing recipes. For example:

• Updating Gradle build files and dependency versions
• Migrating from javax to jakarta
• Refactoring code to use new methods from Java 21 and replacing deprecated ones

However, once the automated refactoring was done, we were still left with build errors, runtime issues, and deprecated code. That’s where AI agents came in. We used it to help us diagnose and fix post-upgrade errors, including deprecated methods, misconfigured Dockerfiles, JVM flags, and more.

Highlights

OpenRewrite was easy to use. All of the recipes we needed for the major upgrades already existed and we were able to use them without making changes.

The AI agents provided significant help debugging issues and fixing build errors. They were able to determine the cause of build errors and, for deprecated methods, provided information on the correct replacement method.

After upgrading the first service, the next upgrade was much easier. Overall, using OpenRewrite together with AI agents sped up the process by weeks.

Lowlights

One of the OpenRewrite recipes added a conflicting dependency that caused build failures. Tracking down the root cause was challenging and slowed us down. While the AI agents were good at providing debugging strategies, it wasn’t good at looking through the code and finding where the conflicting dependency was coming from.

There were also plenty of hallucinations. AI agents often detected deprecated methods but, when they didn’t, they would go in circles trying to fix the code.

Part 2: The workflow

So we proved that combining OpenRewrite with AI agents can massively accelerate Golden Path upgrades. The next step was to systematize this process in a way that would allow us to further scale and accelerate the upgrades.

The goal when designing and building the workflow was to have something that would not only get all JVM services to current Golden Path standards but also have something that could be reused for future upgrades/migrations.

Approach

The first thing we did was create a YAML file structure that would serve as the foundation for the workflow. This would allow us to do a few key things to accomplish our goal:

1. Support future upgrades, since each upgrade would have its own config file 
2. Define upgrades as a collection of smaller upgrades
   • This follows our manual-upgrade approach, results in safer changes, and is easier for AI to do and humans to verify

For the upgrades we are currently targeting, the steps look like this:

1. Gradle 8, JDK 17 and Kotlin 2.1.20
2. Spring Boot 3.x and Base Library 3
3. JDK 21
4. Debian Bookworm
5. OpenTelemetry 1.33.6

In these steps, each individual framework, library, dependency, etc. that is being upgraded is a component. So, for example, the first step listed above has three components: Gradle 8, JDK 17, and Kotlin 2.1.20.

For each component, the YAML file specifies the OpenRewrite recipe (if it has one), whether the upgrade requires AI to make changes, and, if it does, which prompt to use.

The workflow starts by processing this config file. Then, for every upgrade step, the workflow will run the OpenRewrite recipe and/or AI prompt to perform the upgrades of each component. Once all of the upgrades are complete for a single step, the workflow enters a loop that deterministically checks if there are build errors and, if they exist, uses AI to fix them.

Once there are no more build errors, a PR is created and the workflow moves on to the next upgrade step until everything has been successfully upgraded.

Challenges

The core challenge of this project was making a workflow that could consistently generate valid upgrade PRs across all of our repos. Getting the workflow to work on a single service was easy, getting it to work for hundreds was a different story.

This comes down to the fact that services vary in many, often unexpected, ways. Here are a few specific challenges we faced:

• There’s no consistent way to build a service.
  • As changes are made to the microservice template, repos end up with build setups.
  • On top of that, some services have unique changes to handle a service-specific situation.
• Repos have different dependencies that need to be upgraded, and you can’t know what has to be upgraded until you start running into build errors.
• Several repos had issues flagged by static analysis tools after doing the upgrades and, while AI would make fixes, it wasn’t easy to determine if it was the correct fix.

Takeaways

In building the workflow and dealing with those challenges, we came across a few key learnings:

1. Breaking upgrades into smaller steps saves a lot of time. Smaller steps made it easier to improve prompting, debug issues, re-run the workflow in the case of errors, and deploy changes.
2. Don’t over-rely on AI. Learn to love determinism. Being able to apply OpenRewrite recipes to start off allowed us to get to a very solid starting point for our upgrades. We only used AI to fill in the gaps. Also, having a separate and deterministic validation step made the workflow more reliable.
3. Better prompting is not just adding more context. When updating the prompts, we had to find the right balance between adding solutions from completed upgrades and generalizing those solutions. It’s often useful for AI to know the exact diff that resolved an issue. But if you give AI the diffs for everything, you’ll end up bloating the context window and creating an overly-specific prompt that doesn’t work well at scale. On the other hand, if you never provide the exact diff for something that comes up for basically every repo and requires the same fix, AI won’t consistently fix the issue even though it easily could if you provided the code change needed.

What’s next

So far, using the workflow, we have only upgraded a handful of repos. We will continue to do upgrades in small batches while refining the prompts to reduce manual work required to have production-ready pull requests. After that, we will kick off upgrades for all of our JVM repos. Additionally, since we designed the workflow to be usable by other upgrades and migrations, we will explore ways to do that—for example, by getting our Python repos on the Golden Path too.

This work is only the beginning. With both AI tooling and models improving every day, there is so much potential to build on this work and push the boundaries of automated large-scale code migrations.

If you’re interested in building cutting-edge, AI-driven systems you won’t find anywhere else, we’re hiring!