OSS + ELC EU 2026

Testing drivers reliably across several versions of Linux (mainline, linux-next, stable, linux-cip, etc.) and several SoCs can be a challenging task, and it gets even more challenging if the whole process needs to be automated.

In this talk we introduce a testing framework centered around pytest, that makes heavy use of pytest markers to describe the test SW and HW requirements, and we combine this with other open-source projects (like Yocto, LAVA, and GitLab CI/CD) to fully automate Linux kernel testing on embedded devices.

Kernel and embedded Linux engineers often need to reproduce a build or test from someone else. In practice this is still harder than it should be.

A kernel may be built with one toolchain. The rootfs may come from a local build. Firmware may have changed. A model may need specific variables. Continuous Integration (CI) and lab runs may hide setup steps in scripts or jobs.

Then someone asks:

Can I reproduce this?

Too often the answer is: not easily.

This talk shows a practical way to make this better with modern open source tooling, and to show useful workflows that kernel and embedded engineers may not know about yet.

We show how to build the kernel in a repeatable way, provide a known rootfs, boot the target, run tests, and collect logs.

Then we use two Arm examples. One is an OP-TEE flow, where TF-A, U-Boot, and OP-TEE are part of the setup. The other is a CCA flow on QEMU-arm64 and FVP-aemva, where the model setup and software stack also matter.

We use open source tools from the TuxSuite ecosystem, but the focus is the workflow. The same workflow can run on a laptop, in CI, in lab infrastructure, or at scale.

If you've ever built your own CI Lab, you've probably already faced some of these issues:
- Remote power cycling of the boards (particularly when it requires pressing a push-button)
- Managing remote debug console
- Handling inrush current/power outage and surges
- Handling boards with heterogeneous form factors
- Handling remote GPIOS/accessories
- Handling remote displays/ KVMs
- Handling non-disruptive lab/board maintenance
- Handling distributed CI labs

At Baylibre, we've been dealing with CI Lab maintenance for many years, starting from 'scratch' (boards laying on shelves with wires all over the place) to standardised labs in server racks. In this presentation, we will share all the solutions we've found and developed to tackle all these issues, with the objective of improving the quality of our FOSS projects by making our CI labs reliable and easy to maintain.

More than 20 years ago the idea of transforming Linux into a real-time operating system resulted in the birth of the PREEMPT_RT development. But how could the real-time properties of such a complex system be tested? Due to the complexity of modern operating systems, such as Linux, and due to the fact that modern processors operate in a performant but non-deterministic way, appropriate methods for evaluating the real-time behaviour of a given system had to be established: That's why the OSADL QA Farm was born. While the initial focus of the QA Farm was (and still is) the long-term evaluation of the real-time properties of Embedded Linux systems, 20 years of operation with more than 100 systems have also brought many other insights to light - expected and unexpected. This presentation provides an overview of the quality assurance methods used at the OSADL QA Farm, the insights gained from numerous long-term measurements and latest developments, such as the integration of Zephyr systems.

Defined as bugs which disappear or change behavior when observed, every embedded engineer has come across one at some point. Some classic examples include bugs that stop reproducing after adding printk() statements, introducing new variables, connecting a debugger or even simply recompiling the same source code. The root causes of Heisenbugs range all the way from race conditions, memory corruption, alignment issues, cache incoherency, incorrect linker configurations to even hardware bugs.

This talk walks through the debugging techniques which could be applied for debugging these Heisenbugs like static code analysis, memory analysis, diagnostic data collection, tracing and instrumentation, selective elimination of unrelated code paths, modifying assembly instructions at runtime; all without probing the observer’s effect. Beyond techniques, the talk aims to develop the intuition and thought process needed when confronting these bugs. The talk discusses how to distinguish between mere workarounds that mask the underlying issue vs actual fixes. Practical experiences in debugging real Heisenbugs will be shared along with their root causes, investigation approaches and the fixes.

Everyone can build a prototype, but shipping products is where the chaos begins. In this session, we explore embedded systems by discussing failures we have experienced as engineers, consumers, and industry observers. Navigating the vast open-source ecosystem of the embedded Linux galaxy and selecting the right tools for the job is a challenge.

Instead of idealized architectural theory, we will share war stories and costly mistakes, facing the grim reality of bricked devices, maintenance costs and e-waste generation. We will be talking about a coffee machine that needs a technician to cross the galaxy to replace a PCB, an OTA update strategy that lacks the ability to upgrade offline devices with a USB stick, a smart home turned into a wreck when a bankruptcy leaves the IoT fleet permanently bricked and a final story on what happens when immovable object like legacy incident management meets unstoppable force like AI and CRA.

This talk is for anyone working on an embedded Linux product, from junior developers to experienced engineers and managers. Since it is always better to learn from others' mistakes, you will walk away with insights that will help you improve your products.

Large companies have built board farms for decades, providing both solid automated QA capabilities and short developer feedback loops.
LAVA and Labgrid have emerged as standard tools.

Although, today most small companies have not adopted those tools and practices, and too many developers spend time, constantly, on hardware set-up.
Either standard tools are unknown, or two myths circulate: that LAVA is just for large-scale scenarios, and Labgrid is complex to set up.
While at this, many create, instead, new in-house tools (wheel reinvention).

This talk aims to reverse this, by showing two portable mini board farms: one using LAVA, and one using Labgrid.
These two examples will act as comparison and demystification of these standard open-source tools, while showing common concepts.
Both examples share a common open-source CI/CD (GitLab), and the physical layout: a portable 10-inch rack, which can sit on a desk (introduced only in recent years).
Network, serial, power and cable management is done assembling off-the-shelf hardware.

Board farm adoption, at any scale, can help teams to increase ergonomics, comfort, and determinism, for stress-free development, CI, and test workflows.

Embedded Linux projects already have powerful testing frameworks, but they often come with requirements about infrastructure, dependencies, or target environments. In many firmware projects, developers need something simpler: a lightweight way to run system-level validation directly on the device, with minimal setup and no additional runtime requirements.

Cukinia is an open-source test framework designed to make embedded Linux validation simple and accessible. Requiring only a POSIX shell, Cukinia can run directly on target devices even in production and integrates quickly into existing CI/CD workflows. With a collection of predefined test statements, developers can quickly create tests for system configuration, hardware interfaces, services, networking, storage, and more.

Labgrid is a tool for both board farming and embedded systems testing. This session will include an overview of current developments and also provides a gathering for people interested or already using Labgrid.