Oovaide C++ Code Coverage

Last Updated: Sept. 24, 2014

Test Coverage

The Oovaide project has some tools to support flow path test coverage analysis. It can show the percentage of paths that have been tested in each source file and shows the number of times that each instrumented line was executed. This can sometimes be used as a profiler and also to find dead code. The CLang compiler is used to parse the source files.

The Oovaide program contains a build system that will work on C++ projects that will cover many but not all project directory structures. Even if the build system cannot be used to build the project, it should be able to instrument many of the source files.

Code Instrumentation

The Oovaide project has a program named oovCovInstr that will read a source file and produce an instrumented source file. The oovCovInstr is a command line utility that works on an individual source file, and the Oovaide program provides a GUI that will run the utility on all files in a project.

An example function in the source file is:

  void Location::setPosition(Location loc)
    {
    mX = loc.mX;
    mY = loc.mY;
    }

The instrumented source file will look like:

  void Location::setPosition(Location loc)
    {
    COV_IN(COV_mary_cpp, 0);
    mX = loc.mX;
    mY = loc.mY;
    }

The COV_IN line is a macro that simply increments an element in an array, and the "COV_mary_cpp" is simply a #defined offset into the array. The array will store the count of the number of times (hit count) that each instrumented line has been run. The second argument is the index to an instrumented line in the source file.

The COV_IN macro is intended to have a very low cost in execution time so that it does not increase the running time too significantly. The oovCovInstr program will instrument any statements and compound statements, but will only instrument a set of statements within a compound statement once for efficiency.

The macro is defined as the following.

    #define COV_IN(fileIndex, instrIndex) gCoverage[fileIndex+instrIndex]++;

An example of a simple statement in the code is the following.

  for(int i=0; i<3; i++)
    mBags.push_back(WoolBag());

This code would be modified to include extra braces so that it could be instrumented as follows:

  for(int i=0; i<3; i++)
    {
    COV_IN(COV_BlackSheep_cpp, 1)
    mBags.push_back(WoolBag());
    }

The oovCovInstr program will also create a header (OovCoverage.h) and source file (OovCoverage.cpp) that can be compiled into the final executable that will be tested. The header file contains the macro and defines for the array offsets.

The source file contains code that will save the array into a file. If there is already an existing file, the code will first read the file, modify the counts, and write to the file. This allows multiple runs of the program to append the counts from the multiple runs. The default name of the file is "OovCoverageCounts.h". The default configuration is that the array is saved when the program exits.

Modifying and Building the Program Under Test

There are many ways that a program may be modified:

The hit count array may be saved at different times while the program is running.
The hit counts could be made a different type to hold higher counts.
The hit count array file could be saved to a different location.

The Oovaide program provides a build system that will automatically create a new project, and add the OovCoverage files as a library to the link for the project.

Generating Statistics

The Oovaide program provides a menu to run the program that generates statistics. This program reads the OovCoverage.h file to get the offsets of the instrumented lines for each source file, and reads the hit counts file (OovCoverageCounts.h) to get the number of times that each instrumented line was executed.

The statistics are created in the project and named "oovCovStats.txt". The file contains the names of the source files, along with the percentage of instrumented lines that were executed in each file. A simple example for two source files is the following.

    mary.cpp 100
    mary.h 100

This indicates that 100% of the paths in both files were executed.

Another form of output is that each instrumented file is marked with a comment indicating the number of counts that a particular line was run. The following shows that the instrumented line was run 3 times.

  void Location::setPosition(Location loc)
    {
    COV_IN(COV_mary_cpp, 0);    // 3
    mX = loc.mX;
    mY = loc.mY;
    }

To find lines that were never executed, simply search for "//<space>0".

Using with Externally Built or Run Systems

If the program to be tested cannot be built with Oovaide, or the program must run on another system (Ex. embedded systems), then there are some simple steps to generating statistics and hit count results.

First instrument the code as described above. Many times much of the code will be instrumented even if not all compile errors can be removed.
Run the program on the target system.
Make sure there is an oov-cov-oovaide directory under the oovaide project directory. Create an out-Debug directory under the oov-cov-oovaide directory. Copy the OovCoverageCounts.txt file to the oov-cov-oovaide/out-Debug directory.
Run the Oovaide program and select the Coverage/Statistics menu item.

Using with Microsoft Code

The Analysis/Settings menu item in Oovaide can be used to set build arguments. These are examples of some of the arguments that may be needed for an older version of Visual Studio.

  -c
  -x c++
  -DWIN32_PC
  -DWIN32
  -D_WINDOWS
  -D_USRDLL
  -DUNICODE
  -D_WCHAR_T_DEFINED
  -D__int64=long long
  -D_M_IX86
  -D_MSC_VER=1100
  -D__cplusplus
  -IC:\Dave\SMISVN\Programmer\trunk\Source\Common
  -IC:\Program Files\Microsoft Visual Studio 9.0\VC\atlmfc\include
  -IC:\Program Files\Microsoft Visual Studio 9.0\VC\include
  -IC:\Program Files\Microsoft SDKs\Windows\v6.0A\Include
  -fms-extensions
  -fmsc-version=1500
  -ferror-limit=5000