Learn more about the benchmark of static analysis tools for the purpose of comparing solutions for automotive safety and security practices & TrustInSoft Analyzer’s results on the ITC test suite
The Toyota Infotechnology Center (ITC) test suite, created for C/C++ static analysis tools, was designed to quantitatively evaluate industry tools and their ability to discover vulnerabilities in the code as well as their efficiency in doing so.
Static analysis is used to ensure that software is of high quality and successful in reaching high-level software assurance. Static analysis tools should help identify errors like runtime exceptions and other dangerous bugs/undefined behaviors from the source code.
Previous testing methods such as vehicle-level testing have been around for years, but vehicle-level testing does not always ensure safety because it does not take into account the interactions between different systems and components, does not account for all possible scenarios, and does not provide sufficient detail. To ensure the safety of a vehicle, it is necessary to perform additional testing and verification at the component and system levels.
Prior to this report, in 2010 Toyota’s accelerator software cost them approximately $130k/LoC on 280,000 LoC due to litigation, fines, recall costs, dealership losses, and devaluation in the market.
To combat these losses and continue their innovation they took a plan of action to research static analysis as a solution for safe and secure automotive software.
This test suite was created to evaluate the effectiveness of static analysis tools in finding critical bugs in C and C++: more information is available in the research paper created by ITC.
As cited in the report, “The E/E (electrical and electronic) components are gaining dominance in automotive systems. For example, most contemporary vehicles have more than 50 ECUs (Electronic Control Units) [1]. This implies that the software part of existing automotive systems has already been large. Moreover, ADAS (Advanced Driving Assistance Systems) are turning into a competitive area among car makers and Tier 1 suppliers. Obviously, this kind of new system is based on large-scale software.”
With the growing usage of electronic and electrical components, automotive software is growing in complexity and vehicle suppliers need to follow strict safety rules such as ISO 26262.
Additionally, automotive components are safety critical and need to stay compliant and ensure safety to consumers. As a result of growing concerns for the safety of their vehicle components occurred, ITC launched research into static analysis as a solution.
This test suite was conceived by the ITC researchers and consisted of:
The TrustInSoft R&D team used the ITC test suite from Toyota to do the same tests from the aforementioned report’s test suite to determine the effectiveness of our exhaustive static analysis tool, TrustInSoft Analyzer, in both finding these code defects and reporting a minimum of false alarms.
The tests were run by TrustInSoft Analyzer and returned errors/no errors respectively for each corresponding test case in the test suite. It found every bug and did not mislabel any of the tests which did not contain bugs, meaning that every bug found was a real one.
With exhaustive detection of defects on fully supported tests, 100% of the defects were found within the 1008 tests that cover undefined behavior in C/C++ code. This was done with no false alarms(no false positives), meaning that only real defects were detected.
268 tests are excluded from our analysis because they have been deemed out of the scope of TrustInSoft Analyzer at this time (2023).
Some tests in the tests suite that were not fully supported do not relate to undefined behavior (difficult to detect), and others (also not related to undefined behavior) are on the tool roadmap.
TrustInSoft Analyzer found the following:
View the build on TIS-CI
This build shows the analyzer’s results for each defect type and describes whether an Undefined Behavior is expected in the test or if it was found or not. It also includes a link to the GitHub repository with the test suite used for the analysis.
Not only can TrustInSoft Analyzer guarantee coverage for the most critical test cases, but it does so with zero false positives and zero false negatives. This is significant because it allows testers to spend their time on the real issues without having to sort through false alarms or worry if there are still bugs left in the code that were not identified during the analysis.
For the automotive industry, this is especially important. Automotive manufacturers need to know that every software component will behave correctly and not put humans at any risk while using the vehicle. Undefined Behaviors are frequently the source of safety or security problems and safety and cybersecurity must be assured by exploring all possibilities with static analysis.
In conclusion, TrustInSoft Analyzer masters dangerous C/C++ defect detection due to its ability to perform thorough static and dynamic analysis with a mathematical guarantee. There are zero false negatives and it is able to satisfy the requirements of the ITC test suite. All undefined behavior is detected with TrustInSoft Analyzer without wasting time on detecting bugs that aren’t actually there (false alarms). This ensures efficiency during the testing process and mathematical proof that all of these dangerous vulnerabilities have been removed from the code.
Talk to our experts or ask for a demo here: https://trust-in-soft.com/contact/