On Unit Test Coverage and Code Complexity

A team lead approached me asking what should they define as minimum unit test coverage percentage and code complexity metrics. I had to think about my answer a bit and came up with following, which I think is worth sharing to others as well.

Unit-test Coverage %

The more good unit tests there is covering the code, the easier it will be to read, understand and change the code later on. Excellent answer to the question of what should the coverage be can be found from StackOverflow: http://stackoverflow.com/questions/90002/what-is-a-reasonable-code-coverage-for-unit-tests-and-why

If you still insist on having a target percentage, I’d personally recommend to aim for 100%, which is achievable if you use strict TDD. By strict I mean following the Red-Green-Refactor pattern with discipline. There’s a good presentation about this on InfoQ: http://www.infoq.com/presentations/TDD-as-if-You-Meant-It.

Being more reasonable, specially if you already existing code base, you might set the target to 70-80%. Keep in mind though not to stare at the numbers only! Bad tests also give you high coverage but not much confidence or easier refactoring. For this as well, there’s an excellent article available: http://www.sustainabletdd.com/2011/12/lies-damned-lies-and-code-coverage.html

Code Complexity

Code complexity  figure tells quite well how easy it is to read and change the code. It also affects unit testability as smaller methods are generally easier to unit test. I’d aim for cyclomatic complexity of less than 10 per method. More complex code usually contains more bugs as well, so if you find some parts of the code to be more complex, do focus your testing efforts on that area.

We have one project where we have set up our Jenkins job so that it becomes unstable if the amount of complex methods grows. This project now has average method complexity of 5.25. Still there is one methods with high complexity (23 in fact) and about 50 methods of medium level of complexity (10-20).

There might be a good reason for high complexity number e.g. an event handler with a long switch clause. While this might be avoided with polymorphism (see http://sourcemaking.com/refactoring/replace-conditional-with-polymorphism), but this might also make the code more difficult to maintain.

Summary

It’s good to early enough think about code maintainability and the speed of development in the future. Unit test coverage and code complexity are factors which affect those a lot. Other things worth investing to are code reviews as a standard practice and pair programming. Both work as fast feedback cycles to catch not only defects but also code which is difficult to maintain.

In the end it’s up to you and your Product Owner to decide how much do you invest in code quality. Code which is simple, produced with TDD and covered with good unit tests is cheaper to maintain in the future. On the other hand, if these factors are not taken in account, it’s like taking a loan from the bank. You’ll manage a long time by just paying the interest, or parts of that, but eventually you need to start paying back the debt as well. Or take another loan.

It’s more likely that you will produce Clean Code when you have proper measures in place to early detect code going bad. Also worth to note is that you should always aim for writing as less code as possible. A nice article on this subject: http://mikegrouchy.com/blog/2012/06/write-less-code.html

Advertisements

Get the Most Out of Static Code Analysis

Static code analysis is a cheap way to improve code quality. One might argue with this by stating that tools like Klocwork and Coverity are actually fairly expensive. And here comes one common misconception related to static analysis, some people think it’s about tools while it’s not. Sure good tools help, but there are a lot more important issues related to this. There are several problems related in doing static code analysis and only some of those might be solved with proper tools.

Problems / Solutions

Problem: Too many problems reported

Amount of reported problems is devastating and developers feel overwhelmed. As a result, the problems aren’t fixed. Sometimes a large number of reported problems are actually not real problems but e.g. style issues or even false positives. Good tools help in this but truth is all of them require manual work in tuning the results.

Solution: Define baseline and focus only on new problems

Measure the number of problems reported and define that as a baseline. Then make sure that no new problems are introduced. This ensures that the codebase doesn’t at least get worse. Later on its possible to tighten up the thresholds to get rid of older reported problems as well.

Problem: Tools are not used

It doesn’t really help if you have the best tools money can get but nobody’s using them.

Solution: Run tools automatically

If you’re using continuous integration system such as Jenkins you can run static analysis for every commit. In case the analysis takes long time, run it at least once a day (or night). Make sure all committers are being informed should the analysis reveal new problems. In my opinion, if static analysis reveals new problems, it should be considered a stop-the-line situation. So stop everything, fix problems, and only then continue.

You can also take this to extreme by introducing a pre-receive (or pre-commit) hook in your VCS. There a commit is rejected if static analysis doesn’t pass. In this case analysis needs to be really fast.

Problem: Results are not understood

It might well be that tools produce fine reports pointing out the problems but if developers don’t understand why something should be fixed then it doesn’t get fixed. Commonly heard reasoning is “but the code works why should we fix it?”

Solution: Educate developers, chew the results automatically

In the worst case you need to do like was suggested in the previous solution, declare stop-the-line in case analysis reveals problems or even reject commits to VCS in such a case. I’ve noticed one of the most effective ways is to just show and tell. So show developers a recent problem which has been found, point out the source code location and tell what’s wrong with the code. For example complex code usually doesn’t look to be “wrong” but if you try refactoring it turns out to be difficult. Or if code is not covered by unit tests, retro-fitting them is not easy.

It makes sense to do some pre-analysis for the results. So not to show raw results to developers but try to filter out non-problems and possibly also old problems. The simpler the better.

Problem: Tools are used by wrong people

Some companies have solved some of the previously presented problems by devoting a separate team to perform code analysis tasks. Unfortunately it might be that this team is capable of only finding the problems but not fixing them. Sure it might be more effective if there’s number of people telling that here are these problems which should be fixed but it’s still likely the problems will not get fixed.

Solution: Automatic analysis, everyone responsible for fixing problems

A basic rule of thumb which works in several different issues is “if you break it, you fix it”. Making problems visible by automatic analysis and rapid feedback makes it more likely that problems get fixed. With the tools available it is fairly trivial to point out who caused codebase to become worse. If nothing else helps, public humiliation is the last option to try. And that was a joke, being constructive is of course always really important.

 

So what kind of different static source code analysis tools and methods are out there? I’ll tell something about the ones we are using in the next post. Stay tuned, meanwhile you can for example run some static analysis.