CERN Moves Away from Red Hat to Debian Following Hardware Configuration Upgrade of RHEL

The European Organization for Nuclear Research (CERN) is migrating its 2,200 computers overseeing its accelerator complex to Debian. This decision comes after an estimated budget of 5.4 million Swiss francs for maintaining operations within the Red Hat ecosystem, which was deemed to have a successful compliance rate of just 20%. The switch stems from a fundamental instruction set requirement that disqualified approximately 65% of the organization’s control systems. This pivotal decision was publicly shared by two CERN engineers at a presentation on August 30, 2026. It serves as a case study for CIOs, highlighting how alternatives remain viable when vendor decisions and obsolescence converge to close off options — provided the infrastructure is sufficiently decoupled.

Between the operator in the control room and the magnet steering a proton beam, commands traverse three tiers of machinery. From office consoles, the operator may, for instance, request an increase in magnet power or beam intensity using applications developed in Java and Python. Data center servers log the command, authenticate it, and ensure it remains within safety parameters. Finally, computers located up to 100 meters underground convert these instructions into electrical signals, directing a power supply to raise its amperage, thereby strengthening the magnet’s field.

This third tier is identified as front-end computers. Nikos Tsipinakis, a systems engineer on the control system infrastructure team, points out the irony in this designation: for an electronics expert, the front end is where the magnet connects, while everything else resembles back-end operations. These systems serve as the interface between the world of electrons and software, dictating their configuration. They have no disks and boot over the network whenever powered on. They utilize PCI and VME boards custom-designed at CERN for equipment unique to the organization, with a single-board VME card playing a crucial role in the control system.

This custom hardware relies on proprietary drivers compiled into the Linux kernel, where any defect in one can jeopardize overall system stability. CERN has been using Linux on these machines since 2008, coupled with real-time patches, following dedicated real-time operating systems. These boards, designed for a duration of ten to fifteen years, become dependent on a system where the vendor alone dictates the lifecycle; when the two timelines fall out of sync, it is the operator who bears the consequences.

Software engineer Federico Vaga and his team discussed how CERN has managed to mitigate the impact of this issue on August 30, 2026, at the MiniDebConf in Winterthur. Their focus was exclusively on CERN’s accelerator complex, which spans 43 square kilometers. Meanwhile, the organization’s data centers continue to operate under RHEL and AlmaLinux, with other experiments using different options.

A compilation flag, and 65% of the fleet sidelined

RHEL 9 is built on a microarchitecture baseline set at x86-64-v2, which requires SSE4.2 and SSSE3 instructions, while RHEL 10 raises this baseline to x86-64-v3, demanding AVX and AVX2. With their custom-designed industrial boards at CERN, Vaga succinctly captures the consequence: the machines simply display a message from the C library stating they cannot proceed. The systems fail to boot, and the trigger for this transition was that single compiler flag change.

CERN gauged the extent of this boot failure in a risk assessment conducted in Q2 2023. It found that 47% of the systems in the complex operated on original x86-64 processors, including models like the Core 2 Duo, which would no longer boot under v2. An additional 17% of systems did not meet the v3 requirement. Thus, around 65% of the control systems were effectively excluded from both the current version and the upcoming one.

This percentage also factors in the cascading effect of PCI connectors. Machines built before 2010 featured numerous PCI connectors, whereas modern models only offer one or two on the most expensive industrial configurations. Replacing one system often requires three, which increases rack space utilization, complicates already congested cabling paths, and sometimes necessitates excavation. CERN currently manages 70,000 cables connecting 17,000 devices to its 2,200 computers.

What CERN calculated before making the switch

The same analysis estimated the cost of maintaining the status quo at 5.4 million Swiss francs. This included a strict deadline of Q4 2026, redesigning eleven electronic boards, hiring two electronics engineers, two software engineers, and two technicians, as well as reorganizing and rewiring the racks, with a significant impact on system launch times. The estimated success rate, assuming defect-free solutions, stood at 20%, with the authors themselves deeming this figure overly optimistic.

Before drawing their final conclusion, CERN explored an obvious alternative: recompiling CentOS Stream without the deprecated flag. This option was presented to the AlmaLinux team. However, no one was willing to take on the burden, as Vaga noted, removing a flag is insufficient without subsequent validation. He added the broader point that CERN’s primary mission is particle physics, which had previously led the organization to abandon the maintenance of its own distribution years prior. At the time, AlmaLinux was just starting, and lacked the resources to commit to maintaining the baseline.

This exchange prompted a participant to voice the conclusion, “It suggests you trust the community more than yourselves.” The speaker’s acknowledgment is telling. This conversation captures the crux of the reasoning: the decision hinges on what an organization chooses to maintain independently, rather than the technical quality of a vendor. Vaga succinctly summarized the decision: a software problem calls for a software solution.

Red Hat’s arguments and the point at which validity breaks

The slides from CERN reference a series of justifications from Red Hat concerning its two version upgrades, both authored by engineer Florian Weimer. For v2 in January 2021, the justification is hardware-related and holds—affected machines can no longer operate effectively due to drivers withdrawn from the kernel and other resource constraints. There is a collective argument, too: microarchitecture levels were set in 2020 by AMD, Intel, Red Hat, and SUSE collaboratively, and the vendor align with this agreement. Red Hat explicitly ruled out v3 due to lack of emulation and because contemporaneous server processors did not support AVX.

This restriction on AVX was rescinded in January 2024. For v3, the rationale shifted, making CERN’s case universally significant. The excluded hardware now included Atom processors released after 2021, meaning newer hardware was also being left behind. Notably, Red Hat acknowledged in its own release that it could not demonstrate performance improvements in the software delivered with RHEL and claimed that the change would justify itself by allowing third-party vendors to optimize their developments. Red Hat further limited itself by excluding v4, which is not supported by Intel’s efficient cores.

This shift leads to two observations about the 2020 agreement and the RHEL 10 documentation, neither of which assume to convey intent. The 2020 agreement brought together two manufacturers and two vendors without a standardization body, open adoption process, or operator representation, making all future compilation chains binding on everyone. Moreover, the RHEL 10 adoption chapter outlines prerequisites without a word for clients whose hardware is left out, lacking recommendations, migration options, or reminders about the support duration of the previous version. This silence represents a failure on the publisher’s part toward a specific category of clients and leaves them empty-handed precisely when they seek answers. These are mechanisms of capture that no single decision produces; rather, it is their combination that solidifies the impact.

Decoupling the foundation to avoid paying twice

CERN did not replace its dependency with another, and this constitutes the most transferable lesson from their presentation. The organization breaks down the distribution rather than adopting it as a whole. Debian provides the user space and boot image, the kernel comes from kernel.org rather than Debian, real-time patches are applied there, custom drivers are compiled in, and GRUB serves as the bootloader. Vaga states this intention bluntly: this modularization would enable a return to Red Hat without modifying the kernel.

This decoupling has prompted a complete redesign of the boot server and the delivery chain. The once singular, high-security boot server, which Tsipinakis notes was untouchable and required strict protocol, has transitioned to a stateful service built on Kubernetes, which can be destroyed and recreated at will. Artifacts are stored in a GitLab registry, specifying which machine runs which kernel and distribution version. The build tool also generates a script that transforms a Red Hat environment into a Debian environment, allowing Debian compilation on pre-existing Alma machines. CERN’s RPM package construction system has been extended to also create Debian packages, as the organization opted to integrate Debian into its existing infrastructure rather than the other way around.

This infrastructure is deployed at the pace of the LHC and its injector chain, rather than the typical IT cycle. Once operational, the system runs continuously without updates. Tsipinakis emphasizes the scale: while a typical enterprise plans for five years, CERN operates with a minimum fifteen-year horizon and is aware of its system’s state until 2041. Debian Trixie is validated across all systems, with deployment expected by the end of 2026, operational start in early 2027, and a desired stability period extending until the end of 2033. Two fallback plans rely on extended support should a major version upgrade prove impossible during operation. CERN is now funding the company that maintains these long cycles to secure the trajectory on which it depends.

In addition to these timing considerations, there is the argument of version debt, exemplified by CentOS 7. Under CentOS 7, CERN remained on an outdated compilation chain, lacking the language standards that had emerged, to the extent that teams upheld their own compilers and brought their own libraries. Vaga describes this as an environment frozen in the past, while Tsipinakis notes that skipping four versions at once is far costlier than a series of incremental upgrades. Operators ultimately pay for version debt, sooner or later, and at a higher price.

Two design rules that CERN engineers take away from this exercise are: never assume a system running on Debian 12 will work on Debian 13, and treat portability between distributions as a standalone asset. Before the next vendor announcement, CIOs managing a long-lived fleet must consider two questions: What proportion of their fleet relies on a prerequisite the vendor can unilaterally raise, and what would the costs of decoupling the foundational layer look like, measured today rather than when the door closes.

Scroll to Top