Debugging Complex Failures of Real-Time Multi-Core Systems

Taking advantage of multi-core processors for safety-critical real-time systems significantly increases their complexity and probability for complex transient faults, turning debugging into a time- and cost-intensive nightmare. While debuggers fail to reproduce intermittent bugs, tracing, especially hardware tracing, is the choice to not interfere with the program execution, permitting a deep insight into the system€™s behavior. However, current tools often fail to track down the bugs€™ root causes, due to limited trace buffer sizes and trigger mechanisms. In this presentation, a novel tool is presented, overcoming these issues by processing the hardware-trace in real-time, providing complex triggers and variable monitoring scopes, followed by post-mortem analysis. It is demonstrated on an example system, having tight end-to-end timing constraints. The approach permits preventive monitoring even before the program fails and allows mastering the evolving complexity in embedded development.