The C/C++ series - code coverage of template code

Key takeaways:

• Testing C++ templates is challenging due to the need to cover all instantiations.
• TrustInSoft Analyzer can help analyze and show coverage of template instantiations.
• Strategic selection of instantiations is crucial to ensure good coverage.

The C/C++ series - code coverage of template code

Testing C++ templates can be particularly challenging, especially when aiming for high code coverage. Calculating code coverage itself for C++ is tricky. First, there are several criteria to consider: reached lines, reached statements, or Modified Condition/Decision Coverage (MC/DC)… Then, determining the coverage of a function template adds another layer of complexity: simply covering all statements of a single instantiation is clearly insufficient. For a given program, the goal should be to cover all instantiations of all class and function templates. Achieving this level of coverage requires a meticulous approach to testing, including the use of advanced techniques and tools designed specifically for template-heavy code. Using TrustInSoft Analyzer, ensures that the template code is thoroughly tested across various scenarios, providing a more robust and reliable codebase.

Example: Testing a simple template function with TIS Analyzer.

// the function template to be analysed
template <typename T>
T add(T a, T b) {
    return a + b;
}

// Analysis driver
template<typename T>
void
test()
{
  T x;
  tis_make_unknown(&x);
  add(x, x);
}

int
main(void)
{
  test<unsigned>();
  test<float>();
  return 0;
}

In this toy example, we have a simple program with a template function add(). The only instantiations used in this program are with the types unsigned and float. Using TrustInSoft Analyzer, we wrote an analysis driver1 covering these two cases.

If your program only deals with unsigned types, this test is enough to guarantee that there will be no undefined behavior in the add function. If you generate a report, it will show you 100% coverage across all instantiated types.

TrustInSoft Analyzer report - Showing 100% coverage across all instantiated type

Testing the add function as a library

If you use TrustInSoft Analyzer to analyze your program and have it parse all your source files, you can check the coverage of all instantiations of templates in the given program.

However, this obviously does not give any guarantee across all possible instantiations. This is not an issue for a program because you can observe the coverage of all used instantiations, but this is problematic if you want to analyze a library.

If we treat the function template add from the previous example as a library, we should also test unsigned values. But if your program also deals with a signed type like int, you also need to analyze the add<int> instantiation:

int

main(void)

{

  test<unsigned>();

  test<float>();

  test<signed>();

  return 0;

}

Running TrustInSoft Analyzer on this new driver will find and report integer overflow undefined behaviors.

template.hpp:3:[kernel] warning: signed overflow. assert -2147483648 ≤ a+b;

                  stack: add<int> :: signed.cpp:11 <- test<int> :: signed.cpp:19 <- main

template.hpp:3:[kernel] warning: signed overflow. assert a+b ≤ 2147483647;

                  stack: add<int> :: signed.cpp:11 <- test<int> :: signed.cpp:19 <- main

TrustInSoft Analyzer report - Integer Overflow Undefined Behavior

Some other types might generate different kinds of undefined behaviors. For instance, more complex types can introduce additional risks, like pointer arithmetic or uninitialized values.

In our (very) simple add function, it is more likely that undefined behavior will come from the source code using the library rather than the library itself. However, in more complex functions or classes, it is crucial to analyze different instantiations to cover a broad range of use cases effectively.

Conclusion

Covering all possible instantiations presents a challenge and can be time-intensive. But the analyst can carefully select and craft a series of instantiations leading to good global coverage of all possible instantiations. Using TrustInSoft Analyzer with this approach, significantly contributes to the development of a thoroughly tested and highly reliable library that performs consistently well across a diverse range of types and scenarios.

1. An analysis driver is an auxiliary function that can be written by the user or automatically generated by TrustInSoft Analyzer. It specifies the possible initial states of an analysis and calls one or multiple functions in the analyzed code base with a specified set of possible input vectors. It may also perform additional setup by calling builtins. It is then used as the entry point of the analyzer.