loader image

Test Lösungen für Simulink Modelle und Seriencode

Virtual Validation for ADAS/AD

How can I create the needed millions of scenarios?

How can I avoid the test explosion problem?

How can I find out if my tests are passed or failed?

Blog

4 reasons to avoid writing test cases in Excel

Markus Gros

Berlin, Germany

I once heard someone say:

“It’s impossible to develop a software tool that isn’t a competitor to Excel”.

And that’s also true when it comes to the creation of test cases for testing embedded software. Over the past 20 years, Excel has been a very popular format and language to create, manage and store test cases in the embedded software domain. This is the case not only for in-house tools developed within a particular project context, but also for many commercial tools which support or rely on Excel as a test specification language.

And the advantages are clear. Excel is usually already available on most PCs and the table structure is not a bad way to represent test data consisting of signals (columns) and timesteps (rows).

However, in this blog article, I want to share 4 aspects about why Excel is not a complete test case language.

1. Analysis and Update of the Test Architecture

Before I can start to write a test case, I need information about the interface of the system under test. This includes at least information about variable names and roles (e.g., input, output, calibration parameter, local measurement point). Of course, Excel is not able to extract this information, so I either need to manually create a template document with the right interface names or use a script to generate this template from my (for instance) Simulink model or production C code. But even if this seems to work reliably, there are several limitations:

  • What happens if the interface of the SUT changes? If signals are added or removed or even renamed, it breaks the test case and it’s usually not possible to automatically update all tests.
  • What happens with large interfaces that include structures, vectors, or arrays? This quickly leads to an unreadable document, as there’s no special handling to efficiently display these complex data types.
  • What happens with enumerated data types? Usually, they just show up as integer values, instead of having a dropdown menu with the enumeration strings.
  • What happens when I enter invalid data? This could be, for example, a value outside of the value range. As Excel doesn’t know about datatypes and ranges, it won’t provide a warning and I won’t discover the issue until I run a simulation.

2. Simulation and Reporting

What else is missing in Excel? Exactly: The “play” button to directly run my test case. Of course, I can go to a different UI to trigger an import of the Excel file and then convert the values into something which can be used in the simulation (e.g. a .mat file).

But:

  • It’s still an extra (and maybe error prone) step in the process.
  • It usually won’t feed the results from the simulation back to Excel, so I again need a different interface to access the simulation results.
  • And even if I find a way to bring the simulation results back into Excel, I won’t get much support to highlight and find the parts of the test case which made it fail.

3. Missing Features

For small projects and small test interfaces, a test case representation in Excel could definitely be sufficient. But for larger projects and larger components I would expect several features from a test authoring language which are usually not available in Excel:

  1. Requirements-traceability: Usually test cases are derived from requirements, so I need a test authoring environment which allows me to import and view requirements in the same interface in which I’m writing my test case. Usually the requirements are not visible within Excel test cases
  2. Plotting: For longer test cases, it is of course very useful if I can plot signals to visualize patterns
  3. Signal generators
  4. Automatic management of unused signals/steps
  5. Test macros

4. ISO 26262 Compliance

Last but not least, a lot of automotive embedded software projects need to consider the ISO 26262 standard, and even for lower ASIL ratings it’s usually necessary to perform a tool qualification for the test environment. (Please find out more about ISO 26262 tool qualification in this article). As Excel doesn’t come with an ISO 26262 certificate, it means the confidence in the tool needs to be produced by a tool qualification process performed on the user side, which is usually very time consuming.

Conclusion

Excel is a powerful and versatile tool with a flexible data model and because it’s available on most PCs around the globe it appears in many different use cases. But when it comes to testing safety critical embedded software (especially within a larger team) it might make sense to use a dedicated test authoring environment to benefit from advantages including:

  1. Integration with the development environment and system-under-test
  2. Ability to run simulations directly from the test case editor
  3. Advanced features for handling complex data types and data structures
  4. ISO 26262 certification

Markus Gros

Berlin, Germany

Senior Vice President Marketing & Sales

Markus Gros studied Mechatronics at the University of Darmstadt and at the Universidad Politecnica de Catalunya in Barcelona. After his Diploma in 2007, he began working for dSPACE in Paris where he provided support, trainings and consulting to French customers in the Automotive and Aerospace Domain for topics including model-based development, automatic code generation, AUTOSAR and ISO 26262. In 2012, he joined BTC Embedded Systems AG where he is today responsible for global marketing & sales activities. Since the beginning of 2019, he also serves as President of the newly founded daughter company BTC Embedded Systems Inc. in Detroit.

Connect on LinkedIn

Popular Videos

Video abspielen
Video abspielen

Request Evaluation License

If you would like to try out our tools, we will gladly provide an evaluation license free of chargeEvaluations include a free launch workshop and also provide an opportunity for you to meet one-on-one with our support and engineering teams.

Schedule a Meeting

Do you have any questions or want to see our tools in action? If so, please use the link below to schedule a meeting, where a member of our engineering team will be happy to show you the features and use cases and directly answer any questions you might have.

Melden Sie sich für unseren Newsletter an!

Für den Newsletterversand wird Ihre Email Adresse vom DSGVO zertifizierten Dienstleister Newsletter2Go verarbeitet. Weitere Informationen finden Sie in unserer Datenschutzerklärung.

Videos

Discover some of the main features of our products in these short videos.