ryan kennedy’s blog

My current project has a new requirement that in order to be feature complete we must reach 80% code coverage with our unit tests. At first, this seems like a good idea. You want to ensure code quality. Unit tests can help with that. So you figure you’ll come up with a way of measuring how much of your code is being tested. Soon, however, deadlines get tighter and actual features need to be finished. The code coverage is short of the required 80%. So you take the following code:

   if(unreachable) {
      doTheUnreachable();
   }

   int codeCoverageRequired = 80;
   cout < < "Feature complete requires " << codeCoverageRequired << "% code coverage" << endl;

This code has 4 executable lines of code, only 3 of which are being executed. You’re only at 75% code coverage. So you make the following change:

   if(unreachable) {
      doTheUnreachable();
   }

   int codeCoverageRequired = 80;
   cout < < "Feature complete requires ";
   cout << codeCoverageRequired;
   cout << " % code coverage";
   cout << endl;

This code has 7 lines of executable code, 6 of which are being executed. Now you’re at 86% code coverage. You’ve boosted your code coverage numbers by 10% simply by adjusting some lines of code. The quality is no better, but your code coverage is higher.

You can also play games by under-reporting the total lines of code in your project. As it turns out, this is easier than you might think. The code coverage tool we’re using (gcov) appears to have an issue with not reporting the lines of code in files that don’t get tested at all.

By gaming the system you’re able to give management the warm, fuzzy feeling that the code quality is high when, in truth, the opposite may true. Even a high code coverage number without gaming the system doesn’t necessarily mean that the code quality is high. Take the following, for example:

   size_t function( void *ptr, size_t size, size_t nmemb, void *stream) {
      ((string*)stream).append(ptr, size * nmemb);
      return size * nmemb;
   }

   string requestUrl(string url) {
      static string buffer;

      CURL* ch = curl_easy_init();
      curl_easy_setopt(ch, CURLOPT_URL, "http://unclehulka.com/ryan/blog/");
      curl_easy_setopt(ch, CURLOPT_WRITEDATA, &buffer);
      curl_easy_setopt(ch, CURLOPT_WRITEFUNCTION, writehandle);
      curl_easy_perform(ch);
      curl_easy_cleanup(ch);

      return buffer;
   }

If you call requestUrl() from your unit test, you’ll end up with 100% code coverage, however this code is as buggy as it gets (see the ’static string’ declaration).

The lesson is if you want to ensure code quality, use something that actually measures code quality.

This entry was posted on Sunday, April 20th, 2008 at 10:13 am and is filed under Development. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.