Photo by Oleksandr Chumak on Unsplash
It started with a message from a platform engineer from another team.
“The API service keeps crashing and restarting.”
According to their logs, the service was running out of memory.
For the meantime, they increased the memory allocation so users could continue performing their routine analyses. It worked, but everyone knew it was only a temporary fix.
More memory meant higher infrastructure costs.
But not increasing the memory meant users couldn’t do their work.
And there was something more concerning.
This wasn’t a new problem.
A Problem That Kept Coming Back
The issue had been reported several times since last year.
But each time, it looked slightly different.
One report said:
Users get stuck when creating 10 or more jobs.
Another said:
The database service randomly crashes.
Different symptoms. Different tickets. Different engineers.
But nobody had been able to completely solve it.
Then the issue came back again, this time with a much clearer description:
“API is constantly exiting due to Out-of-Memory (OOM) (Error code 137) in Production.”
This time, our team was responsible for the API service.
The engineer who originally developed the affected module had already left the company.
So I decided to take it.
My goal wasn’t to find a way to give the service more memory.
I wanted to answer a much more important question:
Why does the service need so much memory in the first place?
Sprint 1: Making the Invisible Visible
I started where most investigations start: the logs.
I reviewed the existing errors, previous investigations, and suggestions from other engineers. One of the proposed solutions was to use AWS SQS for outgoing requests.
But there was a problem.
I couldn’t simply accept the assumption.
I needed to prove it.
The API relied heavily on AWS services, while the bug happened in production. Reproducing the exact workflow locally wasn’t easy.
And I had experienced this problem before.
Before this ticket, I had worked on background processing that also relied on AWS services. At that time, developing and testing locally was painful because replicating the AWS environment wasn’t straightforward.
That experience contributed to roughly three months of development before we finally released the background-processing solution.
So this time, I knew what I wanted to try.
Ministack.
Bringing AWS to My Laptop
Ministack was recommended by a colleague as a way to simulate AWS services locally.
The concept was surprisingly simple.
Run it through Docker, expose it on:
http://localhost:4566
Then interact with it using the AWS CLI.
For example:
aws --endpoint-url=http://localhost:4566 s3 lsThat’s it.
But running an S3 command wasn’t the difficult part.
The real challenge was understanding the entire system.
I needed to answer questions like:
Which AWS services does the application actually use?
Which service calls another service?
What creates the queue?
Where are the Lambda functions?
How are permissions configured?
Where does Cognito fit into the flow?
Which parts can actually be reproduced locally?
In other words, I didn’t just need to simulate AWS.
I needed to understand how our application talks to AWS.
Turning Infrastructure Into a Script
Once I understood the architecture, I started rebuilding the environment locally.
With the help of Kiro AI, I created scripts that automated the setup by executing a series of AWS CLI commands.
The infrastructure configuration came from YAML files describing the required resources.
Instead of manually creating everything every time, I could essentially run a script and recreate the environment.
That was a huge improvement.
But it wasn’t perfect.
Some parts of production simply couldn’t be reproduced directly.
SSL certificates were one example.
Another was dynamic access-token retrieval through Cognito, which was handled by another API service owned by another team.
There were also differences between communicating with real AWS services and the local Ministack environment.
So I introduced a local-development flag.
When Ministack was enabled, the application could:
Skip SSL validation for local endpoints
Bypass dynamic token retrieval where necessary
Switch between real AWS services and the local environment
It wasn’t an exact copy of production.
But it was close enough to reproduce the workflow.
And that was what I needed.
But AWS Was Only Half the Problem
At this point, I had a local environment.
But I still had another problem.
I didn’t fully understand the codebase.
If I wanted to find where the memory was going, I first needed to understand how the affected module worked.
So I started reading.
I followed the workflow.
I traced how data moved between components.
I looked at how database queries were constructed.
And then I started isolating the problem.
But reading code alone wasn’t enough.
I needed numbers.
Measuring the Memory
With the help of Kiro AI, I created a simple Python decorator for memory tracing.
I could place it on top of a method and let it report how much memory the method consumed during execution.
Suddenly, the problem became measurable.
Instead of saying:
“This part looks like it consumes a lot of memory.”
I could actually see which operations caused memory usage to increase.
This became one of the most useful tools during the investigation.
I also built a small collection of supporting tools:
A log parser that converted pasted logs into a readable table
Scripts for consolidating logs
A simple web UI for browsing and downloading S3 objects
Scripts for repeatedly reproducing the workflow
None of these tools were the actual fix.
But together, they made the investigation dramatically easier.
Then I Found It
By the end of the first sprint, I finally had an answer.
The problem wasn’t SQS.
It wasn’t simply the number of jobs.
And it wasn’t the database randomly crashing for no reason.
The application was performing database queries that loaded unnecessary relationship tables and data into memory.
The methods were retrieving much more data than the operation actually needed.
With a small dataset, this wasn’t obvious.
With a production-sized dataset, it became expensive.
And when several of these operations happened as part of a workflow, memory consumption could grow dramatically.
The OOM error suddenly made sense.
The API wasn’t running out of memory because it randomly needed more.
It was loading too much data.
Sprint 2: Fixing the Problem Without Breaking Everything
Finding the root cause was only half the battle.
The problematic methods were reused throughout the system.
Changing them could affect multiple components:
The API
The background worker
The UI
A quick fix could easily turn into a new regression.
So instead of changing everything at once, I mapped out the dependencies and broke the optimization into smaller pieces.
I traced which components were using the affected methods, what data each component actually needed, and which relationships could safely be removed from the queries.
Then I optimized them one by one.
This approach allowed me to gradually improve the system while continuously validating that existing functionality still worked.
The Unexpected Lesson About Reusability
This investigation also changed the way I think about code reusability.
The problematic methods were reused in many places.
At first glance, that sounds like good software engineering.
And it often is.
During the early stages of development, reusability can make engineers much faster.
Build something once.
Reuse it everywhere.
Move on.
But there’s another side to that story.
Imagine you have a method that returns 20 pieces of information because one consumer needs all 20.
Later, another component needs only 2.
Instead of creating a smaller query, it reuses the existing method.
The application now retrieves all 20 pieces of information just to use 2.
The code is reusable.
The code works.
But the application is paying for data it doesn’t need.
That’s what happened here.
The same reusability that helped us move faster during development eventually made optimization harder during maintenance.
What I Would Do Differently
If I could go back in time, I would keep one rule in mind:
If you don’t need it, don’t include it just for the sake of reusability.
Keep the method simple.
Return what the caller actually needs.
Every unnecessary relationship, field, or object that gets loaded becomes an overhead that someone may have to optimize later.
There’s another lesson I took from this:
Functional testing isn’t enough.
A feature can work perfectly with a development dataset.
Then users adopt it.
The dataset grows.
The number of jobs increases.
And suddenly, the problem appears.
By then, the application is already in production and optimization becomes much more expensive.
So performance needs to be tested alongside functionality, preferably using data that resembles the real workload.
And if possible, those performance tests should be automated to prevent regressions.
The Result
What initially looked like an infrastructure problem turned out to be an application-level performance problem.
By addressing the way data was loaded and processed, I was able to reduce memory consumption in one of the most critical workflows from 2.99 GB to 211 MB. The system’s capacity also increased from around 10 jobs to 23+ jobs, without requiring additional production resources. Most importantly, users could continue their workflows without the service repeatedly running out of memory or becoming unavailable.
The improvements extended beyond the main workflow. Data browsing also became significantly more efficient, with memory consumption reduced from 1.4 GB to 182 MB, response time improved from 8.39 seconds to 1.70 seconds, and response size reduced from 3.82 MB to 152.34 KB.
Overall, the application became more efficient, capable of handling more work, faster when browsing data, and less expensive to operate.
Instead of simply increasing the memory allocation, I was able to trace the issue back to inefficient data loading.
But perhaps the most valuable result wasn’t the optimization itself.
It was having a way to reproduce the problem.
I now had a local environment that could simulate the required services, automated infrastructure setup, memory profiling, log analysis tools, and a repeatable workflow for testing the application.
That changed the investigation from:
“The API randomly crashes.”
to:
“This operation loads more data than necessary, causing memory consumption to increase significantly.”
Those are two very different problems.
The first is a symptom.
The second is something you can investigate, measure, and fix.
The Bigger Takeaway
Looking back, the first sprint might seem like I spent a lot of time building tools instead of fixing the bug.
But those tools were what made the fix possible.
Without a reproducible environment, I would have been guessing.
Without memory profiling, I would have been guessing.
Without understanding the codebase, I would have been guessing.
And without realistic data, I could have fixed the code without actually fixing the problem users were experiencing.
The biggest lesson I took away from this experience is simple:
Don’t optimize based on assumptions. Reproduce the problem, measure it, understand it, and then fix the actual cause.
Sometimes solving a production issue isn’t about immediately changing the code.
Sometimes, the first thing you need to build is the ability to see the problem clearly.
Once you can see the problem, measure it, and reproduce it, the path toward fixing it becomes much clearer.