On Technical Debt...and Pyramids
Are you drowning in a legacy of shortcuts? Are you a victim of the lava anti-pattern? Technical Debt comes to us all. Here's when to create it, when to pay it down and when to draw the line.
Stakeholder: “We’re way behind schedule, can you just use limestone from the lower quarry instead of granite and get that base filled out?”
Builder: ”My lord, limestone’s not nearly as strong; the weight of the upper chambers won’t hold! We’ll have cracks within a generation!”
Stakeholder: “But the Pharaoh will never know the difference. Just ship it!!!”
Witness the earliest Technical Debt conversation between a builder and a stakeholder. I’m taking some creative license here but anyone who’s seen the famous “Bent Pyramid of Sneferu” might wonder how the standup went on that day when the team realized they were building on an unstable foundation.
Let me take you to a scene that plays out in engineering teams every day:
The Sprint is humming along nicely and a request comes from a stakeholder (could be a Product Manager, could be an Executive etc.) "We need this feature by XYZ date. Can we just hardcode these values for now and clean it up later?"
You know the answer should be "no" but you also know the quarter is ending, the feature is important, and you know there’s a hacky way to get it done that while it won’t be pretty will probably work. So you slap something together, your teammate writes "LGTM" on the PR and it’s approved. You promise yourself you'll fix it properly next sprint.
You just took out a high-interest technical loan, and the payments will come due at the worst possible time.
What is Technical Debt?
It’s a broad term but I think of it as anything that wasn’t done right the first time. It might be a hardcoded value that should’ve been read from a config. It might be a piece of code you wrote really inelegantly, e.g. tightly coupling references instead of dependency injection. It might be just a piece of code that doesn’t have good coverage.
These all add up over time to make modifications and maintenance of the code harder than it needs to be. That’s Tech Debt.
How do you define it? Drop a comment below with your worst example.
The True Cost
How do you explain to someone that this 'simple change' would require updating 23 different services, each with its own precarious set of dependencies? That modifying one seemingly innocent parameter would cascade through layers of technical debt accumulated over five years of “we'll fix it later”?
Technical debt isn't just about messy code or outdated dependencies. It's a compound tax on:
Your time - figuring out how something works instead of building features.
Your speed - when something’s designed well it’s easier to maintain. When it’s band-aided together you’re afraid to touch it.
Your reputation - do you want to be known for moving slow and maintaining legacy systems or innovation?
Your learning - if you can’t update your codebase easily you’ll end up stuck using outdated patterns and technologies.
Your job satisfaction - who wants to live that high cyclomatic complexity life?
Your frustration level - you know things could be simpler but feel powerless to stop the never-ending layering of workaround.
A Lived Example
Probably the greatest test of any organization is saying “Hello World” meaning how quickly can you push a change to production?
When I joined Audible/Amazon I noticed our team was all working in a single jar (these were Java days, don’t judge me). The codebase was huge (hundreds of thousands of lines) and developers had turned off unit testing as it took too long to run.
This was crippling tech debt that the entire organization had normalized. Everyone complained about it but no one wanted to take on the task of splitting up the library into smaller more manageable components. It was clear what needed to be done so as a fun little side project I took on this task and over the next few weeks broke that library down into more manageable pieces.
I worked across the group and got to know the team, and slowly areas of domain ownership emerged; Content, Identity, Account, and so on. This effort became known as “The Package Split” and gained its own momentum. When I was done we had about 15-20 much smaller libraries, many of which were rarely touched so we could leave them alone and focus on code coverage for the more frequently worked code bases. It was a thankless task but everyone thanked me 😀 and I got to learn the codebase pretty well as a bonus.
A Lifestyle Change
Managing Tech Debt is a lifestyle. You can’t just "refactor aggressively" or "write clean code." Would you tell someone in financial debt to "spend less." The reality is more nuanced:
Measure - It always starts with data. Track time spent on bug fixes vs new features. Document which parts of the system generate the most support tickets. Quantify the onboarding time for new team members. These data points can help you make your case to leadership that you need to pay down your debt.
Learn to “Speak Business” - Every deprecation needs a story. Don't say "We need to upgrade the database." Say "Our current data infrastructure costs us $X in support hours and limits our ability to handle Y% more traffic, which directly impacts our Q4 growth targets."
The Power of Small Wins - The Boy Scouts say “Leave the campsite cleaner than you found it” so in our world try to "leave the codebase better than you found it." Every ticket, every feature, every bugfix is an opportunity to pay down a small amount of debt. It's the engineering equivalent of paying an extra $50 on your mortgage each month.
Envision your Future Regret - What’s the worst-case scenario? Imagine the system has failed. What types of questions would you ask yourself in the retro? How would you feel in that moment? e.g. “We knew this would happen, why didn’t we address it?”. I don’t want to live with that feeling. Model what the customer impact would be should your system fail and convey this to your team. Is it really something you want to end up in the Wall Street Journal over?
The Career Benefits
Engineers who can successfully navigate technical debt often accelerate their careers. Why? Because technical debt is really about making trade-offs, communicating effectively, and thinking long-term – that’s Engineering. It’s all trade-offs and knowing where to compromise is a key skill for senior engineering roles.
A Call to Action
It’s your turn - Where is your systems’ debt stored? Where are the pain points? What's the interest rate? Think about your team - what skills aren't they learning because they’re maintaining legacy systems? Drop a comment below and let’s talk about it.
Can you create a "debt payment plan" that aligns with business objectives?
“Don’t Let the Pyramid Bend.”
Technical debt, like financial debt, isn't inherently bad. The problems come from pretending it doesn't exist. Sometimes you have to hit your debt ceiling before you learn to manage it properly.
Technical debt doesn’t mean you’ve failed as an engineer, it just means the solution isn’t as elegant as it could be. Engineers who thrive are those who learn to make choices strategically and who can then communicate effectively about those trade-offs. Entropy should decrease over the long term (but the Universe will fight you at every turn).
4,500 years later the Bent Pyramid’s still bent and although the ancient Egyptians did manage to “Ship It” I’m sure the stone masons were bothered by it for the rest of their days…and it probably bugged the Pharoah too.
There was a cartoon that was circulated when I was first starting in the late 70s that said " if builders built houses like coders built applications, then the 1st woodpecker to figure that out would destroy civilization.." The issue of technical debt has only gotten worse as aging systems that house "system of record, gold copies" and other vital information that can only be obtained through some wild magic. The layers of knowledge required to get basic API management and gateways implemented, are very complex. even though their usage should make things easier, getting there is so difficult. Technical debt has always been a business decision that can rarely be argued effectively due to the mismatch in skills, language, and perspectives, let alone values. Solid insight Francis
Great balanced points....most people say tech debt is bad. In finance it can be highly advantageous. It's all about tradeoffs and priority. A wise man once said: "There are only two types of migrations: Incremental ones and failed ones."