I once heard someone say:
“It’s impossible to develop a software tool that isn’t a competitor to Excel”.
And that’s also true when it comes to the creation of test cases for testing embedded software. Over the past 20 years, Excel has been a very popular format and language to create, manage and store test cases in the embedded software domain. This is the case not only for in-house tools developed within a particular project context, but also for many commercial tools which support or rely on Excel as a test specification language.
And the advantages are clear. Excel is usually already available on most PCs and the table structure is not a bad way to represent test data consisting of signals (columns) and timesteps (rows).
However, in this blog article, I want to share 4 aspects about why Excel is not a complete test case language.
1. Analysis and Update of the Test Architecture
Before I can start to write a test case, I need information about the interface of the system under test. This includes at least information about variable names and roles (e.g., input, output, calibration parameter, local measurement point). Of course, Excel is not able to extract this information, so I either need to manually create a template document with the right interface names or use a script to generate this template from my (for instance) Simulink model or production C code. But even if this seems to work reliably, there are several limitations:
- What happens if the interface of the SUT changes? If signals are added or removed or even renamed, it breaks the test case and it’s usually not possible to automatically update all tests.
- What happens with large interfaces that include structures, vectors, or arrays? This quickly leads to an unreadable document, as there’s no special handling to efficiently display these complex data types.
- What happens with enumerated data types? Usually, they just show up as integer values, instead of having a dropdown menu with the enumeration strings.
- What happens when I enter invalid data? This could be, for example, a value outside of the value range. As Excel doesn’t know about datatypes and ranges, it won’t provide a warning and I won’t discover the issue until I run a simulation.
2. Simulation and Reporting
What else is missing in Excel? Exactly: The “play” button to directly run my test case. Of course, I can go to a different UI to trigger an import of the Excel file and then convert the values into something which can be used in the simulation (e.g. a .mat file).
But:
- It’s still an extra (and maybe error prone) step in the process.
- It usually won’t feed the results from the simulation back to Excel, so I again need a different interface to access the simulation results.
- And even if I find a way to bring the simulation results back into Excel, I won’t get much support to highlight and find the parts of the test case which made it fail.
3. Missing Features
For small projects and small test interfaces, a test case representation in Excel could definitely be sufficient. But for larger projects and larger components I would expect several features from a test authoring language which are usually not available in Excel:
- Requirements-traceability: Usually test cases are derived from requirements, so I need a test authoring environment which allows me to import and view requirements in the same interface in which I’m writing my test case. Usually the requirements are not visible within Excel test cases
- Plotting: For longer test cases, it is of course very useful if I can plot signals to visualize patterns
- Signal generators
- Automatic management of unused signals/steps
- Test macros
4. ISO 26262 Compliance
Last but not least, a lot of automotive embedded software projects need to consider the ISO 26262 standard, and even for lower ASIL ratings it’s usually necessary to perform a tool qualification for the test environment. (Please find out more about ISO 26262 tool qualification in this article). As Excel doesn’t come with an ISO 26262 certificate, it means the confidence in the tool needs to be produced by a tool qualification process performed on the user side, which is usually very time consuming.
Conclusion
Excel is a powerful and versatile tool with a flexible data model and because it’s available on most PCs around the globe it appears in many different use cases. But when it comes to testing safety critical embedded software (especially within a larger team) it might make sense to use a dedicated test authoring environment to benefit from advantages including:
- Integration with the development environment and system-under-test
- Ability to run simulations directly from the test case editor
- Advanced features for handling complex data types and data structures
- ISO 26262 certification
