OSS + ELC EU 2026

Running PX4 on thousands of devices surfaces errors that don't appear in lab testing or flight tests. The kind that only show up at a customer site, that you cannot reproduce locally, that disappear when you add or remove code in a completely unrelated module.

This talk presents real incidents from running PX4 at commercial scale: problem descriptions, how debugging was done, and how the errors were fixed.

For each incident we share the heuristics we use to quickly identify which part of the system to investigate, and the tools we use to diagnose Heisenbugs.

We conclude with the hardware CI setup we built, including hardware tracing capabilities and describe how this can help to catch certain kinds of failures described in this talk. In addition we describe how this can be shared with upstream.