Upgrading Rails Using AI: When It Helps and When It Lies
We used AI throughout a full Rails upgrade — six framework hops, five Ruby hops, eleven versions in total. Not as an experiment. As a real tool, integrated into the daily workflow from the first hop to the last. It saved us significant time. It also gave us confident wrong answers on at least two occasions.
The pattern behind both experiences took a few hops to see clearly. There were tasks where AI accelerated our work dramatically, and tasks where it slowed us down by giving us plausible but incorrect information to act on. The difference wasn't about the tool getting better or worse over time. It was about which kind of task we were handing it.
Where It Delivers
The tasks where AI consistently added value shared a common shape: the problem was already understood before we touched it. We knew what the broken pattern was, we knew what the correct form looked like, and we'd already made the call on what needed to change. What remained was execution — the kind of work that's expensive for a human not because it's hard, but because it's slow and repetitive at scale.
Mass Pattern Fixes
When Ruby 3.0 made keyword argument separation strict, our codebase had hundreds of call sites using the old pattern — passing a hash where keyword arguments were expected. Controllers, models, gems, initializers. The fix for each instance was mechanical: find the pattern, understand whether it's a positional-to-keyword issue or a double-splat issue, apply the correct form.
That's exactly what AI is built for. We described the pattern, provided examples from the codebase, and let it classify and fix every instance systematically. Work that would have taken days of error-prone manual scanning finished in hours.
The same applied to deprecated
update_attributes calls, renamed constants across engines, and a handful of alias_attribute usages that changed behavior in Rails 7.x. In every case, one engineer understood the problem and made the call on what the fix was. AI executed it at scale.What made it work: the task was well-defined. Input shape known, output shape known, decision already made by a human.
Repetitive Test Coverage
After fixing hundreds of keyword argument call sites, we needed specs to verify the fixes held. The specs followed a consistent structure — same setup pattern, same assertion, different method under test.
Writing eighty specs that follow the same template is exhausting for a human and completely trivial for AI. The output was solid. We reviewed each batch, caught a few cases where the generated spec didn't match the actual method signature, corrected those, and moved on.
AI doesn't get tired on spec number 47. For this kind of work, that matters.
CI Failure Triage — With One Condition
Paste a failing test output into the conversation with enough surrounding context, and AI can trace the root cause quickly. Across eleven upgrade hops, this worked reliably — but only for one category of failure: technical, localized errors where all the relevant information is in a single output.
A
NoMethodError on a removed method, a mismatched keyword argument in a specific controller action, a failing spec because an ActiveRecord method was renamed — these AI handled well. We'd paste the failure, get a traced path to the root cause in seconds, verify it against the code, and apply the fix.That last condition — all the relevant information in a single output, no cross-system reasoning required — is also the exact boundary where usefulness ends. And the first time we hit something on the other side of it, the contrast was sharp.
So... Where It Doesn't?
The failures weren't about AI making obvious mistakes. They looked reasonable in the moment. The problem was that we were using the tool on tasks that required something it can't do: reason across incomplete information, hold the nuance of a specific version combination, or weigh tradeoffs against context it doesn't have access to. Understanding that distinction earlier would have saved us from at least two situations where we had to backtrack.
Multi-System Failures
One of the production deployments triggered an incident that lasted 36 hours and ultimately required a rollback. The symptoms looked like a Rails problem. They turned out to be infrastructure: a database cluster where PgBouncer was running in session mode on two of three nodes and in transaction mode on the primary — a misconfiguration that had existed for years but had never been triggered because the primary had never failed over during production operation.
The Rails deployment was the first event that put enough CPU pressure on the primary to take it out of service. Patroni failed over to the misconfigured nodes. In session mode, every Rails thread held a database connection for its entire lifetime instead of returning it after each transaction. Under production load, connections were exhausted across all services within minutes.
We tried using AI to help triage. It was useful for explaining PgBouncer modes and what the error messages meant. It could not reason about the interaction between our specific Patroni failover behavior, the Hetzner firewall configuration on the failover node, the per-node PgBouncer discrepancy, and the application's connection pool settings across four service types running at different concurrency levels. That chain of causality required a DevOps engineer reading actual server logs and tracing the failure manually across multiple nodes over two days.
This is the hard boundary of what AI can do in triage. Localized, self-contained failures: yes. Failures that span multiple systems with non-obvious causal chains: no.
Version-Specific Answers
A subtler limitation showed up throughout the upgrade, and it's one that's easy to miss because the answers AI gives are usually plausible. AI's knowledge is bounded by what was documented, discussed, or written about at training time — and the specific edges of a multi-hop Rails upgrade are not well-documented territory.
When Kaminari broke against a newer Rails version — not because of a documented change, but because it called a private ActiveSupport method that had been made inaccessible — AI didn't know about it. It couldn't warn us. We found it because CI broke and we traced it back through the call stack.
The same was true for several other edge cases: specific behaviors in how
belongs_to scopes interact with eager_load in a particular Rails version, and a gem that called File.exists? after Ruby 3.2 removed it. None of these were in any changelog. None were things AI could have predicted.More subtly: when AI described how a Rails method behaved "in 7.x," it was sometimes describing 7.1 behavior and applying it to 7.2. Or describing a gem's current API based on a version older than what we'd already updated to. These answers weren't obviously wrong — they were plausible, stated with confidence, and slightly off in ways that only became visible when we verified against the actual source.
The fix: treat version-specific AI output as a lead, not a conclusion. Verify against the release notes, the gem's changelog, or the installed source before applying anything version-sensitive.
Decisions That Require Judgment
When we had to decide whether to patch a broken gem, replace it, or defer the migration entirely, AI could enumerate options. It could explain the tradeoffs of each approach in general terms. What it couldn't do was weigh those tradeoffs against our specific codebase — how large the gem's footprint was across multiple engines, how much runway we had in the engagement, and what the realistic risk profile of each path was for this particular app.
We made those calls ourselves. AI was a useful research assistant in the process — it surfaced options we might not have considered, and helped us think through the implications of each one. But the decision was ours.
The pattern: AI is good at generating options. It is not good at making context-sensitive tradeoffs. The more a decision depends on knowing your specific codebase, your specific timeline, or your specific client, the less you should delegate it.
The Underlying Rule
Looking back across all eleven hops, every situation where AI helped us had something in common: the problem was already understood by a human before AI touched it. We knew what the broken pattern was. We knew what the correct form was. We knew what the spec should verify. We gave AI a well-defined task with a known output shape.
Every situation where AI failed or misled us had something in common too: we were asking it to understand something we didn't fully understand yet ourselves. To reason about a multi-system failure from incomplete information. To tell us the definitive behavior of a specific version combination it might have only partial data on. To make a judgment call that depended on context it didn't have access to.
The mental model that held up across the entire engagement: AI executes. It doesn't investigate, and it doesn't decide.
Use it for the "how" once you already know the "what" and "why." When you're still figuring out what the problem is, or what you should do about it — that's your job. AI is available after that.
Share: