TrustInSoft Analyzer Drives Results on the Toyota Infotechnology Center (ITC) Test Suite

Context

The Toyota Infotechnology Center (ITC) test suite, created for C/C++ static analysis tools, was designed to quantitatively evaluate industry tools and their ability to discover vulnerabilities in the code as well as their efficiency in doing so.

 Static analysis is used to ensure that software is of high quality and successful in reaching high-level software assurance. Static analysis tools should help identify errors like runtime exceptions and other dangerous bugs/undefined behaviors from the source code.

 Previous testing methods such as vehicle-level testing have been around for years, but vehicle-level testing does not always ensure safety because it does not take into account the interactions between different systems and components, does not account for all possible scenarios, and does not provide sufficient detail. To ensure the safety of a vehicle, it is necessary to perform additional testing and verification at the component and system levels.

Why was the test suite developed by Toyota?

Prior to this report, in 2010 Toyota’s accelerator software cost them approximately $130k/LoC on 280,000 LoC due to litigation, fines, recall costs, dealership losses, and devaluation in the market.

 To combat these losses and continue their innovation they took a plan of action to research static analysis as a solution for safe and secure automotive software.

 This test suite was created to evaluate the effectiveness of static analysis tools in finding critical bugs in C and C++: more information is available in the research paper created by ITC.

 As cited in the report, “The E/E (electrical and electronic) components are gaining dominance in automotive systems. For example, most contemporary vehicles have more than 50 ECUs (Electronic Control Units) [1]. This implies that the software part of existing automotive systems has already been large. Moreover, ADAS (Advanced Driving Assistance Systems) are turning into a competitive area among car makers and Tier 1 suppliers. Obviously, this kind of new system is based on large-scale software.” 

With the growing usage of electronic and electrical components, automotive software is growing in complexity and vehicle suppliers need to follow strict safety rules such as ISO 26262.

Additionally, automotive components are safety critical and need to stay compliant and ensure safety to consumers. As a result of growing concerns for the safety of their vehicle components occurred, ITC launched research into static analysis as a solution.

 [1] (Shiraishi, Shinichi & MOHAN, Veena & MARIMUTHU, Hemalatha. (2015). Test Suites for Benchmarks of Static Analysis Tools. 10.1109/ISSREW.2015.7392027.)

What does the ITC Test Suite consist of?

This test suite was conceived by the ITC researchers and consisted of:

• 1272 test cases:Half of the test cases have defects to be detected by a tool and the other half doesn’t have any defects. These cases contanied two code examples for each defect type: one with defects and one without. This allows one to see if the tool is detecting all the vulnerabilities and to determine if the tool excessively reports alarms where it should not (false positives).
• Tests written in C and C++
• Tests come in 50 different categories: Some of the possible categories for the tests consist of:bit shift bigger than integral type or negative,dead code,memory leakage,etc.

TrustInSoft’s Analysis Methods

The TrustInSoft R&D team used the ITC test suite from Toyota to do the same tests from the aforementioned report’s test suite to determine the effectiveness of our exhaustive static analysis tool, TrustInSoft Analyzer, in both finding these code defects and reporting a minimum of false alarms.

The tests were run by TrustInSoft Analyzer and returned errors/no errors respectively for each corresponding test case in the test suite. It found every bug and did not mislabel any of the tests which did not contain bugs, meaning that every bug found was a real one.

TrustInSoft Analyzer’s Results on the Test Suite

With exhaustive detection of defects on fully supported tests, 100% of the defects were found within the 1008 tests that cover undefined behavior in C/C++  code. This was done with no false alarms(no false positives), meaning that only real defects were detected.

 268 tests are excluded from our analysis because they have been deemed out of the scope of TrustInSoft Analyzer at this time (2023).

• Defects related to concurrency (dead locks, live locks, race conditions): Testing parallel applications is not easy. Concurrent bugs are difficult to detect due to the non-deterministic behavior of parallel applications. Even if these bugs are detected, it is difficult to reproduce them consistently. The TrustInSoft team can provide multithreaded code analysis on a service basis to address the architecture and design of the parallel program.
• Defects that cannot cause Undefined Behavior (for example unused variables): One of the main functionalities of TrustInSoft Analyzer is to detect undefined behavior. These types of errors that cannot cause undefined behavior are not part of the analysis.
• Stack overflow defects: Stack overflow is not an undefined behavior. TrustInSoft Analyzer is designed to detect undefined behavior. Consequently, these types of errors are not part of the analysis.

 Some tests in the tests suite that were not fully supported do not relate to undefined behavior (difficult to detect), and others (also not related to undefined behavior) are on the tool roadmap.

 TrustInSoft Analyzer found the following:

• In red/green you can see how TrustInSoft Analyzer has identified the code example. To the right of that you can see the name of the test and whether the code example was expected to produce undefined behavior or not (OK). 

• These tables depict the types of defects; their subtypes, and the number of corresponding tests for each category that are being tested in the test suite to determine if the code examples that contain defects are correctly identified as having defects. Inversely it should also identify the test cases of the code that do not have defects as being error-free, thus not producing false alarms.
• This graph demonstrates TrustInSoft Analyzer’s ability to cover 100% of static memory defects, dynamic memory defects, pointer-related defects, resource management defects, and misc. defects. Full details on the 1272 tests covered can be found in the Annex.

• From the Toyota test suite, 1008 (or 79%) of the tests were fully supported by TrustInSoft Analyzer: These supported tests allow you to check for bugs that can be very subtle and difficult to find.

• And the remaining 21% is out of the scope of the analysisdefects related to concurrency, e.g., dead locks, live locks, race conditions (9 categories);bad practices that cannot cause Undefined Behavior, e.g., unused variables, redundant conditions, math.h exceptions (7 categories);the stack overflow defects (1 category).

View the build on TIS-CI

This build shows the analyzer’s results for each defect type and describes whether an Undefined Behavior is expected in the test or if it was found or not. It also includes a link to the GitHub repository with the test suite used for the analysis.

Not only can TrustInSoft Analyzer guarantee coverage for the most critical test cases, but it does so with zero false positives and zero false negatives. This is significant because it allows testers to spend their time on the real issues without having to sort through false alarms or worry if there are still bugs left in the code that were not identified during the analysis.

For the automotive industry, this is especially important. Automotive manufacturers need to know that every software component will behave correctly and not put humans at any risk while using the vehicle. Undefined Behaviors are frequently the source of safety or security problems and safety and cybersecurity must be assured by exploring all possibilities with static analysis.

Conclusion

In conclusion, TrustInSoft Analyzer masters dangerous C/C++ defect detection due to its ability to perform thorough static and dynamic analysis with a mathematical guarantee. There are zero false negatives and it is able to satisfy the requirements of the ITC test suite. All undefined behavior is detected with TrustInSoft Analyzer without wasting time on detecting bugs that aren’t actually there (false alarms). This ensures efficiency during the testing process and mathematical proof that all of these dangerous vulnerabilities have been removed from the code.

Talk to our experts or ask for a demo here: https://trust-in-soft.com/contact/

Annex