Appsec and Technical Debt

Technical debt is a fact of life for anyone working in software development: work that needs to be done to make the system cleaner and simpler and cheaper to run over the long term, but that the business doesn't know about or doesn't see as a priority. This is because technical debt is mostly hidden from the people that use the system: the system works ok, even if there are shortcuts in design that make the system harder for developers to understand and change than it should be; or code that’s hard to read or that has been copied too many times; maybe some bugs that the customers don’t know about and that the development team is betting they won’t have to fix; and the platform has fallen behind on patches.

It’s the same for most application security vulnerabilities. The system runs fine, customers can’t see anything wrong, but there’s something missing or not-quite-right under the hood, and bad things might happen if these problems aren't taken care of in time.

Where does Technical Debt come from?

Technical debt is the accumulation of many decisions made over the life of a system. Martin Fowler has a nice 2x2 matrix that explains how these decisions add to a system’s debt load:

I think that this same matrix can be used to understand more about where application security problems come from, and how to deal with them.

Deliberate Decisions

Many appsec problems come from the top half of the quadrant, where people make deliberate, conscious decisions to short cut security work when they are designing and developing software. This is where the “debt” metaphor properly applies, because someone is taking out a loan against the future, trading off time against cost – making a strategic decision to save time now, get the software out the door knowing that they have taken on risks and costs that will have to be repaid later.

This is the kind of decision that technology startups make all the time. Thinking Lean, it really doesn't matter if a system is secure if nobody ever uses it. So build out important features first and get customers using them, then take care of making sure everything’s secure later if the company lasts that long. Companies that do make it this far often end up in a vicious cycle of getting hacked, fixing vulnerabilities and getting hacked again until they rewrite a lot of the code and eventually change how they think about security and secure development.

Whether you are acting recklessly (top left) or prudently (top right) depends on whether you understand what your security and privacy obligations are, and understand what risks you are taking on by not meeting them. Are you considering security in requirements and in the design of the system and in how it’s built? Are you keeping track of the trade-offs that you are making? Do you know what it takes to build a secure system, and are you prepared to build more security in later, knowing how much this is going to cost?

Unfortunately, when it comes to application security, many of these decisions are made irresponsibly. But there also situations when people don’t know enough about application security to make conscious trade-off decisions, even reckless decisions. They are in the bottom half of the quadrant, making mistakes and taking on significant risks without knowing it.

Inadvertent Mistakes

Many technical debt problems (and a lot of application security vulnerabilities) are the result of ignorance: from developers not understanding enough about the kind of system they are building or the language or platform that they are using or even the basics of making software to know if they are doing something wrong or if they aren't doing something that they should be doing. This is technical debt that is hidden even from people inside the team.

When it comes to appsec, there are too many simple things that too many developers still don’t know about, like how to write embedded SQL properly to protect an app from SQL Injection, or how important data input validation is and how to do it right, or even how to do something as simple as a Forgot Password function without messing it up and creating security holes. When they’re writing code badly without knowing it, they’re in the bottom left corner of the technical debt quadrant – reckless and ignorant.

But it’s also too easy for teams who are trying to be responsible (bottom right) to miss things or make bad mistakes, because they don’t understand the black magic of how to store passwords securely or because they don’t know about Content Security Policy protection against XSS in web apps, or how to use tokens to protect sessions against CSRF, or any of the many platform-specific and situation-specific security holes that they have to plug. Most developers won’t know about these problems unless they get training, or until they fail an audit or a pen test, or until the system gets hacked, or maybe they will never know about them, whether the system has been hacked or not.

Appsec Vulnerabilities as Debt

Thinking of application security vulnerabilities as debt offers some new insights, and a new vocabulary when talking with developers and managers who already understand the idea of technical debt. Chris Wysopal at Veracode has gone farther and created a sensible application security debt model that borrows from existing cost models for technical debt, calculating the cost of latent application security vulnerabilities based on risk factors: breach probability and potential breach cost.

Financial debt models like this are intended to help people (especially managers) understand the potential cost of technical debt or application security debt, and make them act more responsibly towards managing their debt. But unfortunately tracking debt costs hasn't helped the world’s major governments face up to their debt obligations and it doesn't seem to affect how most individuals manage their personal debt. And I don't think that this approach will create real change in how businesses think of application security debt or technical debt, or how much effort they will put in to addressing it.

Too many people in too many organizations have become too accustomed to living with debt, and they have learned to accept it as part of how they work. Paying off debt can always be put off until later, even if later never comes. Adding appsec vulnerabilities to the existing debt that most managers and developers are already dealing with isn't going to get vulnerabilities taken care of faster, even vulnerabilities that have a high “interest cost”. We need a different way to convince managers and developers that application security needs to be taken seriously.

Frankensystems, Half-Strangled Zombies and other Monsters

There are lots of ugly things that can happen to a system over time. This is what the arguments over technical debt are all about – how to keep code from getting ugly and fragile and hard to understand and more expensive to maintain over time, because of sloppiness and short-sighted decision making. But some of the ugliest things that happen to code don’t have anything to do with technical debt. They’re the result of conscious and well-intentioned design changes.

Well-Intentioned Changes can create Ugly Code

Bad things can happen when you decide to re-architect or rewrite a system, or start some large-scale refactoring, but you don’t get the job done. Other more important work comes up before you can finish transitioning all of the code over to the new design or the new platform – or maybe that was never going to happen anyways, because you didn't have the budget and the mandate to do the whole job in the first place. Or the somebody who started the work leaves, and nobody else understands their vision well enough to carry it through – or nobody that’s left cares about it enough to finish it. Or you get just far enough to solve whatever problems your or the customer really cared about, and there’s no good business case to keep going.

Now you’re left with what a colleague of mine calls a “Frankensystem”: different designs and different platforms spliced together in a way that works but that is horribly difficult to understand and maintain.

Why does this happen? How do you stop your system from turning into a monster like this?

Branching by Abstraction

One way that code can get messed up, in the short-term at least, is through Branching by Abstraction, an idea that has become popular in shops that Dark Launch changes through Continuous Deployment or Continuous Delivery.

In Branching by Abstraction (also known as “branching in code”), instead of creating a feature branch to isolate code changes, and then merging the changes back when you’re done, everyone works in trunk. If you need to make bigger code changes, you start by writing temporary scaffolding (abstraction layers, conditional logic, configuration code like feature switches) to isolate the changes that you’ll need to make, and then you can make your changes directly in the code mainline in small, incremental steps. The scaffolding serves to protect the rest of the system from the impact of your changes until all of the work is complete.

Branching by Abstraction tries to address problems with managing the misuse of feature branches (especially long-lived branches) – if you don’t let developers branch, then you don’t have to figure out how to keep all of the branches in sync and manage merge conflicts. But with Branching by Abstraction, until the work is complete and the temporary scaffolding code removed, the code will be harder to maintain and understand, and more brittle and error-prone, as James McKay points out:

“…visible or not, you are still deploying code into production that you know for a fact to be buggy, untested, incomplete and quite possibly incompatible with your live data. Your if statements and configuration settings are themselves code which is subject to bugs – and furthermore can only be tested in production. They are also a lot of effort to maintain, making it all too easy to fat-finger something. Accidental exposure is a massive risk that could all too easily result in security vulnerabilities, data corruption or loss of trade secrets. Your features may not be as isolated from each other as you thought you were, and you may end up deploying bugs to your production environment”.

If you decide to branch in code like this (we do branching in code in some cases, and feature branching in others – branching in code is good for rolling out behind-the-scenes plumbing changes, not so good for big functional changes), be careful. Review your scaffolding to ensure that your code changes are completely isolated, and test with old and new configurations (switches off and on) to check for regressions. Minimize the number of changes that the team rolls out at one time, so that there’s no chance of changes overlapping or colliding. And to keep Branching by Abstraction from becoming a maintenance nightmare, make sure that you remove temporary scaffolding as soon as you are done with it.

Half-Strangled Zombies

Branching by Abstraction can lead to ugly code, at least for the few weeks or months that it will take to roll out each change. But things can get much worse in the code if you try to do a major rewrite or re-architecture of a system incrementally, for example “strangling” the existing system with new code and a new design (another approach coined by ThougtWorks), and slowly suffocating the old system.

Strangling a system lets you introduce a new design or change over to a new and modern platform without having to finish a long and expensive rewrite first. The strangling work is done in parallel, usually by a separate team, letting the rest of the team to maintain the old code – which of course means that both teams need to keep in sync as changes and fixes are made.

But if you don’t finish the job, you’ll be left with a kind of zombie, a scary half-dead and half-alive thing with ugly seams showing, as Nat Pryce warns against in this Stack Overflow post:

"The biggest problem to overcome is lack of will to actually finish the strangling (usually political will from non-technical stakeholders, manifested as lack of budget). If you don't completely kill off the old system, you'll end up in a worse mess because your system now has two ways of doing everything with an awkward interface between the two. Later, another wave of developers will probably decide to strangle what's there, writing yet another strangler application, and again a lack of will might leave the system in an even worse state, with three ways of doing things….

I've seen critical systems that have suffered both of these fates, and ended up with about four or five "strategic architectural directions" and "future state architectures". One large multi-site project ended up with eight different new persistence mechanisms in its new architecture. Another ended up with two different database schemas, one for the old way of doing things and another for the new way, neither schema was ever removed from the system and there were also multiple class hierarchies that mapped to one or even both of these schemas."

Strangling, and other incremental strategies for re-architecting a system, will let you start showing benefits to the customer early, before all of the work of writing the new system is done. This is both an advantage and a problem. Because once the customer starts to get what they really care about (some nice new screens or mobile access channels or better performance or faster turnaround on rules changes or…) you may not be able to make the business case to finish up the work that’s left. Everyone understands (or should) that this means you’re stuck with some inconsistencies – on the inside certainly, and maybe on the outside too. But whatever is there does the job, and keeping this mess running may cost a lot less than finishing the rewrite, at least in the short term.

Frankensystems and Zombies are Everywhere

Monster-making happens more often than it should to big systems, especially big, mission-critical systems that a lot of different people have worked on over a long time. As Pryce warns, it can even happen multiple times over the life of a big system, so that you end up with several half-realized architectures grafted together, creating all kinds of nasty maintenance and understanding problems.

When making changes or adding features, developers will have to decide whether to do it the old way or the new way (or the other new way) – or sometimes they will need to do both, which means working across different architectures, using different tools and different languages, and often having to worry about keeping different data models in sync. This complexity means it’s easy to make mistakes or miss or misunderstand something, and testing can be even uglier than the coding.

You need to recognize these risks when you start down the path of incrementally changing a system’s direction and design – even if you believe you have the commitment and time to finish the job properly. Because there’s a good chance that you’ll end up creating a monster that you will have to live with for years.

Don’t take the Technical Debt Metaphor too far

Because “technical debt” has the word “debt” in it, many people have decided that it makes sense to think and work with technical debt in monetary terms, and treat technical debt as a real financial cost. This is supposed to make it easier for technical people to explain technical debt to the business, and easier to make a business case for paying debt off.

Putting technical debt into financial terms also allows consultants and vendors to try to scare business executives into buying their tools or their help – like Gartner calculating that world wide “IT debt” costs will exceed $1.5 in a couple of more years, or CAST software’s assessment that the average enterprise is carrying millions of dollars of technical debt.

Businesses understand debt. Businesses make a decision to take on debt on and they track it, account for it and manage it. The business always knows how much debt they have, why they took it on, and when they need to pay it off. Businesses don’t accidentally take on debt – debt doesn't just show up on the books one day.

We don't know when we're taking technical debt on

But developers accidentally take on debt all of the time – what Martin Fowler calls “inadvertent debt”, due to inexperience and misunderstandings, everything from “What’s Layering?” to “Now we know how we should have done it” looking at the design a year or two later.

"The point is that while you're programming, you are learning. It's often the case that it can take a year of programming on a project before you understand what the best design approach should have been."

Taking on this kind of debt is inevitable – and you’ll never know when you’re taking it on or how much, because you don’t know what you don’t know.

Even when developers take on debt consciously, they don’t understand the costs at the time – the principal or the interest. Most teams don’t record when they make a trade-off in design or a shortcut in coding or test automation, never mind try to put a value on paying off their choice.

We don’t understand (or often even see) technical debt costs until long after we've taken the costs on. When you’re dealing with quality and stability problems; or when you're estimating out a change and you recognize that you made mistakes in the past or that you took shortcuts that you didn't realize before or shortcuts that you did know about but that turned out to be much more expensive than you expected; or once you understand that you chose the wrong architecture or the wrong technical platform. Or maybe you've just run a static analysis tool like CAST or SONAR which tells you that you have thousands of dollars of technical debt in your code base that you didn't know about until now.

Now try and explain to a business executive that you just realized or just remembered that you have put the company into debt for tens or hundreds of thousands of dollars. Businesses don’t and can’t run this way.

We don't know how much technical debt is really costing us

By expressing everything in financial terms, we’re also pretending that technical debt costs are all hard costs to the business and that we actually know how much the principal and interest costs are: we’re $100,000 in debt and the interest rate is 3% per year. Assigning a monetary value to technical debt costs give them a false sense of precision and accuracy.

Let's be honest. There aren't clear and consistent criteria for costing technical debt and modelling technical debt repayment – we don’t even have a definition of what technical debt is that we can all agree on. Two people can come up with a different technical debt assessment for the same system, because what I think technical debt is and what you think technical debt is aren't the same. And just because a tool says that technical debt costs are $100,000.00 for a code base, doesn't make the number true.

Any principal and interest that you calculate (or some tool calculates for you) are made-up numbers and the business will know this when you try to defend them – which you are going to have to do, if you want to talk in financial terms with someone who does finance for a living. You’re going to be on shaky ground at best – at worse, they’ll understand that you’re not talking about real business debt and wonder what you’re trying to pull off.

The other problem that I see is “debt fatigue”. Everyone is overwhelmed by the global government debt crisis and the real estate debt crisis and the consumer debt crisis and the fiscal cliff and whatever comes next. Your business may be already fighting its own problems with managing its financial debt. Technical debt is one more argument about debt that nobody is looking forward to hearing.

We don’t need to talk about debt with the business

We don’t use the term “technical debt” with the business, or try to explain it in financial debt terms. If we need to rewrite code because it is unstable, we treat this like any other problem that needs to be solved – we cost it out, explain the risks, and prioritize this work with everything else. If we need to rewrite or restructure code in order to make upcoming changes easier, cheaper and less risky, we explain this as part of the work that needs to be done, and justify the costs. If we need to replace or upgrade a platform technology because we are getting poor support from the supplier, we consider this a business risk that needs to be understood and managed. And if code should be refactored or tests filled in, we don’t explain it, we just do it as part of day-to-day engineering work.

We’re dealing with technical debt in terms that the business understands without using a phony financial model. We’re not pretending that we’re carrying off-balance sheet debt that the company needs to rely on technologists to understand and manage. We’re leaving debt valuation and payment amortization arguments to the experts in finance and accounting where they belong, and focusing on solving problems in software, which is where we belong.

Are bugs part of technical debt?

Everybody is talking about technical debt today: developers, testers, consultants, managers - even executives.

But the more that people talk about technical debt, the fuzzier the idea gets, and more watered down the meaning of “technical debt” becomes.

Philippe Kruchten is trying to solve this by suggesting a narrower definition of what technical debt is and what it isn't. He breaks out all of the work on a system into work that is visible or invisible, and that has positive or negative value:

	Visible	Invisible
Positive Value	New Features	Architecture and Structure
Negative Value	Defects	Technical Debt

In this model, defects that have been found but haven't been fixed yet aren't technical debt - they are just part of the work that everyone can see and that has to be prioritized with the rest of the backlog.

But what about defects that you've found, but that you've decided that you’re not going to fix – bugs that you think you can get away without fixing (or that the people before you thought they could get away without fixing)?

These bugs are technical debt – because you’re pretending that the bugs are invisible. You’re betting that you don’t have to take on the cost of fixing these problems in the short term at least, just like other kinds of technical debt: copy and paste code and conscious short-cuts in design and compounding complexity and automated tests that should have been written but weren't and refactoring that should have been done that wasn't and code that you wrote that you wished you hadn't because you didn't know the language well enough back then and anything else that you may have done or not done in the past that has a chance of slowing down your work and making it harder in the future.

The same rule applies to results from static analysis bug finding tools like Findbugs and Klocwork and Fortify which point out coding problems that could be real bugs and security vulnerabilities. After you filter out the false positives and motherhood, and the code that works but really should be cleaned up, you’re left with code that is wrong or broken – code that should be fixed.

Keep in mind that these are problems that haven’t been found yet in testing, or found in production by customers – or at least nobody knows that they've run across these bugs. These are problems that the team knows about, but that aren't visible to anybody else. Until they are fixed or at least added to the backlog of work that will be done soon, they are another part of the debt load that the team has taken on and will have to worry about paying back some day. This is why tools like SONAR include static analysis coding violations when calculating technical debt costs.

I agree that bugs aren't technical debt – unless you’re trying to pretend that the bugs aren't there and that you won’t have to fix them. Then it’s like any other technical debt trade off – you’ll need to see if your bet pays off over time.

Technical Debt – when do you have to pay it off?

There are 2 times to think about technical debt:

1. When you are building a system and making trade-off decisions between what can be done now and what will need to be done “sometime in the future”.
2. “Sometime in the future”, when have to deal with those decisions, when you need to pay off that debt.

What happens when “sometime in the future” is now? How much debt is too much to carry? When do you have to pay if off?

How much debt is too much?

Every system carries some debt. There is always code that isn’t as clean or clear as it should be. Methods and classes that are too big. Third party libraries that have fallen out of date. Changes that you started in order to solve problems that went away. Design and technology choices that you regret making and would do differently if you had the chance.

But how much is this really slowing the team? How much is this really costing you? You can try to measure if technical debt is increasing over time by looking at your code base. Code complexity is one factor. There is a simple relationship between complexity and how hard it is to maintain code, looking at the chance of introducing a regression:

Complexity	% Chance of bad fix
1-10	5%
20-30	20%
>50	40%
100	60%

Complexity by itself isn’t enough. Some code is essentially complex, or accidentally complex but it doesn’t need to be changed, so it doesn’t add to the real cost of development. Tools like Sonar look at complexity as well as other variables to assess the technical risk of a code base:

Cost to fix duplications + cost to fix style violations + cost to comment public APIs + cost to fix uncovered complexity (complex code that has less than 80% automated code coverage) + cost to bring complexity below threshold (splitting methods and classes)

This gives you some idea of technical debt costs that you can track over time or compare between systems.But when do you have to fix technical debt? When do you cross the line?

Deciding on whether you need to pay off debt depends on two factors:

1. Safety / risk. Is the code too difficult or too dangerous to change? Does it have too many bugs? Capers Jones says that every system, especially big systems, has a small number of routines where bugs concentrate (the 20% of code that has 80% of problems), and that cleaning up or rewriting this code is the most important thing that you can do to improve reliability as well as to reduce long the term costs of running a system.
2. Cost – real evidence that it is getting more expensive to make changes over time, because you’ve taken on too much debt. Is it taking longer to make changes or to fix bugs because the code is too hard to understand, or because it is too hard to change, or too hard to test?

While apparently for some teams it’s obvious that if you are slowing down it must be because of technical debt, I don’t believe it is that simple.

There are lots of reasons for a team to slow down over time, as systems get bigger and older, reasons that don’t have anything to do with technical debt. As systems get bigger and are used by more customers in more ways, with more features and customization, the code will take longer to understand, changes will take longer to test, you will have more operational dependencies, more things to worry about and more things that could break, more constraints on what you can do and what risks you can take on. All of this has to slow you down.

How do you know that it is technical risk that is slowing you down?

A team will slow down when people have to spend too much time debugging and fixing things – especially fixing things in the same part of the system, or fixing the same things in different parts of the system. When you see the same bugs or the same kind of bugs happening over and over, you know that you have a debt problem. When you start to see more problems in production, especially problems caused by regressions or manual mistakes, you know that you are over your head in debt. When you see maintenance and support costs going up – when everyone is spending more time on upgrades and bug fixing and tuning than they are on adding new features, you're running in circles.

The 80:20 rule for paying off Technical Debt

Without careful attention, all code will get worse over time, but whatever problems you do have are going to be worse in some places than others. When it comes to paying back debt, what you care about most are the hot spots:

• Code that is complex and
• Code that changes a lot and
• Code that is hard to test and
• Code that has a history of bugs and problems.

You can identify these problem areas by reviewing check-in history, mining your version control system (the work that Michael Feathers is doing on this is really cool) and your bug database, through static analysis checks, and by talking with developers and testers.

This is the code that you have to focus on. This is where you get your best return on investment from paying down technical debt. Everything else is good hygiene – it doesn't hurt, but it won’t win the game either. If you’re going to pay down technical debt, pay it down smart.

Software Development Metrics that Matter

As an industry we do a surprisingly poor job of measuring the work that we do and how well we do it. Outside of a relatively small number of organizations which bought into expensive heavyweight models like CMMI or TSP/PSP (which is all about measuring on a micro-level) or Six Sigma, most of us don’t measure enough, don't measure the right things or understand what to do with the things that we do measure. We can’t agree on something as basic as how to measure programmer productivity or even on consistent measures of system size –should we count lines of code (LOC, SLOC, NCLOC, ELOC) or IFPUG function points or object oriented function points or weighted micro function points or COSMIC Full Function Points, or the number of classes or number of something else…?

Capers Jones, who has spent most of his career trying to understand the data that we do collect, has this to say:

The software industry lacks standard metric and measurement practices. Almost every software metric has multiple definitions and ambiguous counting rules… The result of metrics problems is a lack of solid empirical data on software costs, effort, schedules, quality, and other tangible matters. Strengths and Weaknesses of Software Metrics, 2006

But how can we get better, or know that we need to get better, or know if we are getting better, if we can’t or don’t measure something? How can we tell if a new approach is helping, or make a business case for more people or a rewrite, or even justify our existing costs in hard times, without proof?

What do you really have to measure?

Different metrics matter to management, the team and the customer. There are metrics that you have to track because of governance or compliance requirements, or because somebody more important than you said so. There are metrics that the team wants to track because they find them useful and want to share. There are stealth metrics that as a manager you would like to track quietly because you think they will give you better insight into the team’s performance and identify problems, but you aren’t sure yet or you don’t want people to adapt and try to game what you are measuring. And most important, the measures that the customer actually cares about: are you delivering what they need when they need it, is the software reliable and usable?

Like most people who manage development shops, it’s much more important to me to get work done than to collect measurement data. Any data must be easy and cheap to collect and simple to use and understand. If it takes too much work, people won’t be able to keep it up over the long term – they’ll take shortcuts or just stop doing it. The best metrics are the ones that come for free, as part of the work that we are already doing. http://www.amazon.com/Managing-Design-Factory-Donald-Reinertsen/dp/0684839911 I need a small number of metrics that can do double duty, that can be combined and used in different ways.

Size

In order to compare data over time or between systems and teams, you need a consistent measure of the size of the system, the size of the code base and how this is changing.Counting the Lines of Code, whether it is Source Lines of Code (SLOC)or Effective Lines of Code (ELOC) or NCLOC (Non Commented Lines of Code), is simple and cheap to measure for a code base and easy to understand (as long as you have a consistent way of dealing with blank lines and other formatting issues), but you can’t use it to compare code written in different languages and on different platforms.

This is where Function Points come in – a way of standardizing measurements of work for estimation and costing independent of the platform. Unfortunately it’s not easy to understand or measure Function Points – while there are lots of tools that can measure the number of lines of code in a code base and every programmer can see and understand what 100 or 1000 lines of code in a certain language looks like, Function Point calculations need to be done by people who are certified Function Point counters (I’m not kidding), using different techniques depending on which of the 20 different Function Point measurement methods they choose to follow.

Function Points have been around since the late 1970s but this way of measuring hasn’t caught on widely because it isn’t practical – too many programmers and managers don’t understand it and aren’t convinced it is worth the work. See “Function Points are Fantasy Points”.

A rough compromise is to use “backfiring” rules-of-thumb to convert from LOC in each language to Function Points (you’ll need to find a good backfiring conversion table). With backfiring, 1 Function Point is approximately 55 lines of Java (on average), or 58 lines of C#, or 148 lines of C. There are a lot of concerns about creeping inaccuracies using this approach, but for rough comparisons of size between systems in different languages it’s all that we really have for now.

Where are you spending time and money, and are you doing this responsibly?

Everybody has to track at least some basic cost data – its part of running any project or business. What’s interesting is coming up with useful breakdowns of cost data. You want to know how much people are spending on new stuff, on enhancements and other changes, on bug fixing and other rework, on support, on large-scale testing work, on learning – and how much time is being wasted. Whatever buckets you come up with should be coarse-grained – you don’t need detailed, expensive time tracking and other cost data to make management decisions. If everyone is tracking time for billing purposes or project costing or OPEX/CAPEX financial reporting or some other reason it’s easy. Otherwise, it’s probably enough to take samples; ask everyone to track time for a while, work with this data, then track time again for a while to see what changes. With a common definition of size across systems (see above), you can compare costs across teams and systems, as well as look at trends over time.

For maintenance teams, Capers Jones recommends that teams track Assignment Scope: the amount of code that one programmer can maintain and support in a year. You can use this for planning purposes (how many people will you need to maintain a system), watch this change over time and compare between teams and systems (based on size again), or even against industry benchmarks. According to Capers Jones’s data, an average programmer should be able to maintain and support around 1,000 Function Points of code per year, depending on the programmer’s experience, the language, quality of code and tools.

Speed

Everybody wants software delivered as fast as possible. Agile teams measure Velocity to see how fast they are delivering at any point in time, but the problem with velocity, as Mike Cohn and others have explained in detail, is that it can’t be compared across teams, or even within the same team over a long period of time, because the definition of how much is being delivered, in Story Points, isn’t stable or standardized – Story Points mean different things to different teams, and even different things for the same team over a long enough period of time (as people enter and leave the team and as they work on different kinds of problems). Velocity is only useful as a short-term/medium-term predictor of one team’s delivery speed.

Teams that are continuously delivering to production can measure speed by the number of changes they are making, and the how big the changes are (we’re back to measuring size again). Etsy measures the frequency of changes and the size of the change sets, and correlates this with production incident data (frequency and impact and time to recover) to help understand whether they are changing too much or too quickly to be safe.

Probably the most useful measure of speed comes from Lean and Kanban. To determine if you are keeping up with customer demand, measure the turnaround or cycle time on important changes and fixes: how long it takes from when the customer finds a problem or asks for a change to when they get what they asked for. Turnaround is simple to measure and obvious to the business and to the team, especially if you use something like Cumulative Flow Diagrams to make cycle time visible. It’s easy to see if you are keeping up or falling behind, if you are speeding up or slowing down. Focusing on turnaround ties you directly to what is important to the business and your customers.

Reliability and Quality

But speed isn’t everything – there’s no point in delivering as fast as possible if what you are delivering is crap. You also need some basic measures of quality and reliability.

For online businesses, Uptime is the most common and most useful measurement of reliability: measuring the number of operational problems, the time between operational failures (MTTF), and the time to recover from each failure (MTTR). Basic Ops stuff.

And you need some kind of defect data, which come free if you are using a bug tracking system like Bugzilla or Jira. Identify areas in the code that have the most bugs, measure how long it takes to fix bugs and how many bugs the team can fix each month. Over the longer term, track the bug opened/closed ratio to see if there are more bugs being found than normal, if the team is falling behind on fixing bugs, if they need to slow down and review and fix code rather than attempting to deliver features that aren’t actually ready.

To compare between teams or systems or to watch trends develop over time, one of the key metrics is defect density: the number of bugs per KLOC (or whatever measure of code size you are using – back to the size measurement again). With enough data, you can even try to use defect density to help determine when code is really ready to be shipped.

How Healthy is the Code?

You also want to measure the health of the code base, how much technical debt you are taking on, using code analysis tools. Code coverage is a useful place to start, especially if you are putting in automated testing, or even if you just want to measure the effectiveness of your manual testing. Tools like EMMA and Clover instrument the code and trace execution as you are testing to give you a picture of what code was covered by testing, and more important, what code wasn’t. It’s easy to drill down and see where the holes in your testing are, and to build trends from this data, to set code coverage targets and watch out for people cutting back on testing under pressure.

Measuring Code Complexity, identifying your most complex code and watching to see if complexity is increasing over time helps you make decisions about where to focus your reviews and testing and refactoring work. If you are measuring both test coverage and complexity, you can correlate the data together to identify high risk code: complex routines that are poorly tested. Code health can also be measured by static analysis checkers like Findbugs and PMD or Coverity or Klocwork – tools that look for coding mistakes and security vulnerabilities and violations of good coding practices. Higher-level code quality management tools like Sonar and CodeExcellence, which consolidate and correlate data from multiple static analysis tools, give you an overall summary picture of the health of all of your code, and let you look at code health over time and across teams and systems.

Using Metrics in Ways that aren’t Evil

Many programmers think that metrics are useless or evil, especially if they are used by management to evaluate and compare programmer productivity. IBM for example, apparently uses static measurement data to scorecard application developers and teams and in an attempt to quantify programmer performance. Using this kind of data to rate developers for HR purposes can easily lead to abuse and reinforcing of the wrong behavior. If you are measuring “productivity” by size and simple code quality measures, no programmers will want to take on hard problems or step up to maintain gnarly legacy code because their scores will always look bad. Writing lots of simple code to make a tool happy doesn’t make someone a better programmer.

Only measure things to identify potential problems and negative trends, strong points and weak points in the code and your development process. If you want to set goals for the team to improve speed or cost of quality, let them decide what to measure and how to do it. And measure whatever you need to build a business case – to prove to others that the team is moving forward and doing a good job, or to make a case for change. Make metrics work for the team. You can do all of this without adding much to the cost of development, and without being evil.

Technical Debt - How much is it Really Costing you?

The idea behind the technical debt metaphor is that there is a cost to taking short cuts (intentional technical debt) or making mistakes (unintentional technical debt) and that the cost of not dealing with these short cuts and mistakes will increase over time.

The problem with this metaphor is that with financial debt, we know how much it would cost to pay off a debt off today and we can calculate how much interest we will have to pay in the future. Technical debt though is much fuzzier. We don’t really know how much debt we have taken on – you may have taken on a lot of unintentional technical debt – and you may still be taking it on without knowing it. And we can’t quantify how much it is really costing us – how much interest we have paid so far, what the total cost may be in the future if we don’t take care of it today.

Some people have tried to put technical debt into concrete financial terms. For example, according to CAST Software’s CRASH report

“applications carry on average $3.61 of technical debt per line of code”.

For some reason, the average cost of Java apps was even higher: $5.42 per line of code. These numbers are calculated from running static structural analysis on their customers’ code.

Sonar, an Open Source dashboard for managing code quality, also tries to calculate a technical debt cost for a code base, again using static analysis findings like code coverage of automated tests, code complexity, duplication, violations of coding practices, comment density.

Thinking of technical debt in this way is interesting, but let’s stop pretending that these are hard numbers that we can use to make trade-off decisions. Although the numbers appear precise, they’re arbitrary, guesses. And they assume that technical debt can be calculated by a tool looking at the structure of the code. Unfortunately, dealing with technical debt is not that straightforward.

But if debt is too fuzzy to be measured in detailed cost terms, how do you know what kind of debt is hurting you the most, how do you know when you have too much? Let’s look at different kinds of technical debt, and how much they might cost you, using a fuzzier approach.

$$$ Making a fundamental mistake in architecture or the platform technology – you don’t find out until too late, until you have real customers using the system, that a key piece of technology like the database or messaging fabric doesn’t scale or isn’t reliable, or that you can’t scale out your architecture like you need to because of core dependency problems, or you you made some fundamentally incorrect assumptions on how the system is supposed to work or how customers will use it. Now you have no choice but to start again or at least rewrite big chunks of the system to get it to work or to keep it working, and you don’t have the time to do this properly.

$$-$$$ Error-prone code – the 20% of the code where 80% of bugs are found. Capers Jones says that all big systems have a small number of routines where bugs and problems cluster, code that is hard to understand and expensive and dangerous to change because it was done badly in the first place or it went to hell over time because of the accumulation of short-sighted fixes and changes. Not rewriting this code is one of the most expensive mistakes that developers make.

$-$$ The system can’t be easily tested – because you don’t have good automated tests, or the tests are brittle and slow and keep falling apart when you change the code. Testing costs can make up more than half of the cost of making any change or fix – sometimes testing can take much more time and cost much more than making the fix – and testing costs tend to go up over time as you write more code, as the system adds more interfaces and options.

$-$$ Not taking care of packaging and release and deployment. Relying too much on manual steps and manual checks, leading to mistakes and problems in production, late nights. Like testing, release and deployment costs don’t go away, they just keep adding up incrementally.

$-$$ Code that mysteriously works, but nobody is sure how or why – usually performance-critical or safety-critical low-level plumbing code written by a wizard who has long since left the company. It might be beautiful code, but if nobody on the team understands it, it’s a time bomb – someday, somebody is going to have to change it or fix it, or try to.

$-$$ Forward and backward compatibility adapters and compromises. This is necessary, short-term debt. But the cost rises the longer that you have to maintain these compromises.

$-$$ Out of date libraries and middleware stack – you’ve fallen behind on applying patches and upgrades. Even if the code that you have now is stable, you run some risk of unpatched security vulnerabilities. The longer that this goes on, the further behind you are, the higher the risk – at some point if the software is no longer supported or supportable, and your hand is called.

$-$$ Duplicate, copy-and-paste code. This is one of the bugaboos of technical debt and static analysis tools. Almost everybody has it. But how bad is it, really? The cost depends on how many clones developers have made, how often they need to be changed, how many subtle differences there are between the different copies, and how easily you can find the copies and keep track of them. If the developer who made the copies is still on the team and does a good job of keeping track of all of them, it doesn't cost much if anything.

$-$$ Known, outstanding bugs in code and unresolved static analysis findings. The cost and risk depends on how many bugs and warnings you have, and how nasty they are. But if they are real problems, they should have been fixed by now. Is a bug really a bug if it isn't bugging anyone?

$-$$ Inefficient design or implementation, “throwing hardware at it”, using too much memory or network bandwidth or CPU. Hardware is cheap, but these costs can add up a lot as you scale out.

$ Inconsistent use of programming idioms and patterns – developers either didn’t understand the existing patterns, or didn’t like them and introduced new ones, or didn’t care and just wanted to get their change done. It's ugly, and it can be frustrating for developers. But the real cost of living with the situation is often less than trying to clean it all up.

$ Missing or poor error handling and exception handling. It will constantly bite you in the ass in production, but it won’t cost a lot to at least get it mostly right.

$0.01 Hard coding, magic numbers, code that isn’t standards compliant, poor element naming, missing comments, and code that needs tidying. This is a pain in the ass, but it’s the kind of thing that is easy to clean up as part of standard refactoring work.

$0.01 Out of date documentation – another issue that is commonly considered in technical debt. But let’s be honest, most programmers don’t read documentation anyways. If nobody is using it, get rid of it. If people are using it, why isn’t it up to date?

$0.00 Hand-rolled code that could have and should have been done using built-in language features or libraries, or existing framework or common services. It’s disappointing when somebody recognizes it, but unless this hand-rolled code has lots of bugs in it, it’s a sunk cost, not a cost that is increasing over time.

There are different kinds of debt, with different costs. Figuring out where your real costs are, and what to do about it, isn't easy.