We Upgraded to MySQL 8. Our Overnight Jobs Went From 12 Hours to 30.
How a production rollback, a weekend project, and AI turned into an open-source MySQL replay engine.
It was a Friday deployment.
The migration itself was uneventful. We upgraded one of our production MySQL 5.7 databases to MySQL 8.0 after successful QA validation. The database came back online, the applications connected, smoke tests passed, and everything looked healthy.
Until our overnight business jobs started running.
The same workflows that normally completed in around 12 hours were now taking close to 30 hours.
Nothing had changed except the database.
Within days, we rolled back to MySQL 5.7.
That rollback raised a simple question:
How do you prove a MySQL upgrade won't regress your production workload before you deploy it?
Surprisingly, I couldn't find a good answer.
Surely Someone Already Solved This
What I wanted sounded straightforward.
Record what production actually does.
Replay that exact workload against MySQL 8.
Compare the results.
The same queries.
The same concurrency.
The same timing.
Exactly what production executed.
I assumed there had to be a mature tool for this.
Instead, I found a graveyard.
Percona's pt-upgrade compares two servers, but it replays everything through a single connection. That's useful for correctness, but not for discovering concurrency regressions.
Percona Playback had exactly the architecture I wanted, but it has effectively been abandoned for years and predates modern MySQL authentication.
Tools like sysbench and mysqlslap generate synthetic workloads. They're great for stress testing hardware but can't answer whether your application's workload became slower.
GitHub was full of replay projects that solved parts of the problem, but every one had a deal-breaker: replaying queries in the wrong order, requiring external middleware, lacking MySQL 8 support, or even executing write statements by default.
The more I searched, the more obvious it became.
The reason a new replay tool appears every few years is because nobody ever finishes the problem.
Fine. I'll Build It.
Like many engineers, I started in Python.
The idea seemed simple enough:
- Parse the slow query log.
- Preserve session ordering.
- Replay queries concurrently.
- Compare latency distributions.
Then I pointed it at a real production capture.
It consumed gigabytes of memory.
CPU usage spiked.
Latency measurements varied between identical runs.
The benchmark itself had become the bottleneck.
The Moment I Realized Python Was the Wrong Tool
The problem wasn't Python being "slow."
The problem was that a benchmark tool has one job:
It must disappear from the measurement.
Mine wasn't disappearing.
It was allocating millions of objects, spending time in garbage collection, buffering huge result sets, and competing with MySQL for CPU.
Even if every query executed correctly, I could no longer trust the latency numbers.
When you're trying to detect regressions measured in milliseconds — or even microseconds — your measurement tool can't become another source of noise.
At that point I wasn't benchmarking MySQL anymore.
I was benchmarking my Python script.
Why I Switched to Rust
Once the requirements became clear, the language choice almost made itself.
I needed:
- True parallelism across thousands of concurrent sessions.
- Predictable latency without garbage collection pauses.
- Bounded memory usage while streaming massive captures.
- A single static binary that could run directly on production RHEL hosts.
Rust checked every box.
Instead of loading millions of events into memory, the workload streams through an on-disk spool.
Instead of a global interpreter lock, thousands of concurrent sessions execute across all available CPU cores.
Instead of deploying Python environments, dependencies, and version compatibility, the replay engine is just one executable.
The tool itself should never become the experiment.
The Part That Wouldn't Have Happened Five Years Ago
Here's the part that makes this a 2026 story.
I'm not a career Rust systems programmer.
Yet the project exists.
I built it with AI acting as a pair programmer.
That doesn't mean AI wrote the project.
It means AI dramatically reduced the cost of learning unfamiliar systems programming concepts.
Instead of spending a week reading protocol documentation, I could discuss design ideas, generate implementations, challenge assumptions, and iterate quickly.
The important part wasn't writing code faster.
It was making ambitious projects feel achievable.
Every AI-generated component still had to earn its place through testing.
The replay engine is fuzz-tested, validated against real MySQL 5.7 and 8.0 containers in CI, and verified against intentionally injected regressions.
AI accelerated implementation.
Testing established trust.
The Result
The project eventually became sql-replay, an open-source tool for validating database upgrades against real production workloads.
Instead of generating synthetic traffic, it can:
- Capture production workloads from slow query logs or packet captures.
- Replay the original concurrency and session ordering.
- Compare latency distributions between database versions.
- Detect result differences, not just performance regressions.
- Fail CI automatically when regressions exceed configured thresholds.
Most importantly, it answered the question we actually cared about:
Will our production workload perform differently after this upgrade?
Was Building It Worth It?
Halfway through the project I discovered another GitHub repository with almost the same goal.
For a moment I wondered whether I'd wasted days reinventing something that already existed.
Then I realized something.
Pieces of the solution already existed.
One project replayed traffic.
Another compared results.
Another captured packets.
Another handled concurrency.
But there still wasn't a maintained tool that combined all of those capabilities into something you could confidently drop into a production migration pipeline.
And that's exactly what we needed.
More importantly, it caught our regressions before another production cutover.
The migration eventually shipped based on data instead of hope.
That's why the project exists.
Final Thoughts
This project started because a routine database upgrade failed.
It became an exercise in performance engineering, systems programming, and one unexpected lesson about modern software development.
Five years ago I probably wouldn't have attempted something this ambitious outside my comfort zone.
Today, AI changes that equation.
Not because it replaces engineers.
Because it lowers the cost of becoming one in a new domain.
Try It Yourself
If you're planning a MySQL migration — or just want to replay your real production workload instead of relying on synthetic benchmarks — you can find the project here:
⭐ GitHub: https://github.com/Shaked98/avraham-sql-tools
Issues, feature requests, and pull requests are always welcome.
If the article helped you, I'd also appreciate a ⭐ on the repository.