The European AI Office entrusts the cybersecurity assessment of large models to Capgemini, Wavestone, and LNE

The European AI Office has tasked appointed experts with assessing the cyber capabilities of large AI models. A consortium led by Capgemini, alongside Wavestone and the National Laboratory of Metrology and Testing (LNE), will measure their offensive capabilities. This initiative provides the regulator with its own assessment tools in a domain where available metrics often come from model publishers.

Currently, the cyber capabilities of large AI models are evaluated by their vendors. For instance, on November 9th, Anthropic published scores for its Claude Mythos 5.2 based on its own vulnerability discovery assessments, limiting access to organizations it pre-vets. In contrast, the European AI Office, under the Directorate-General for Communications Networks, Content and Technology (CNECT) of the European Commission, opted for a different evaluation tool. On November 5th, Capgemini announced that it had been selected to lead a consortium with Wavestone and LNE to assess the offensive cyber risks associated with general-purpose AI models.

The contract falls under Lot 2 of the EC-CNECT-2025-OP-0032 procurement process, scheduled for a duration of 36 months, Wavestone clarified. The consortium will model emerging risks, develop assessment tools, design compliance methodologies, and conduct evaluations at the Commission’s request. Wavestone’s contribution includes risk modeling, monitoring, as well as simulated attack exercises (red teaming) and stress tests on the models’ safeguards.

Six Lots, Each Addressing Systemic Risk

The call for tenders dates back to July 10, 2025. Funded by the Digital Europe program, it has a budget of €9 million and is divided into six lots. The first five tackle distinct systemic risks, covering nuclear, radiological, biological, and chemical threats, offensive cyber capabilities, loss of control, manipulation, and socio-technical risks. The sixth lot supports transversal evaluations, particularly those observing the autonomous behavior of models in open tasks.

This segmentation aligns with the European AI regulation, the AI Act, which imposes obligations on general-purpose model providers as of August 2025. Models classified as systemic risk must undergo evaluations, including adversarial testing, and implement risk mitigations. Lot 2 empowers the AI Office to verify these evaluations in the cyber domain. According to Wavestone, such a model could help uncover vulnerabilities, craft targeted exploits, or amplify attacks on critical infrastructure. The European AI Safety Program, under which this contract falls, aims to equip the Commission with tools, methodologies, and test beds to evaluate model capabilities and classify those posing a systemic risk.

Metrics Previously Provided by Publishers

Until now, cyber score assessments have come from the publishers selling the models. For instance, Anthropic claims that Claude Mythos 5.2 boasts a score of 89.6% on its internal vulnerability reassessment against the public vulnerability registry (CVE, Common Vulnerabilities and Exposures). The publisher itself states that external reviewers consulted prior to the release did not review either the announcement or the model’s technical specifications. Additionally, laboratories publish their own incident reporting guidelines, as highlighted by IT Social in an article focused on these rules.

Thus, Lot 2 positions a Commission-appointed evaluator between the model provider and its customers. The compliance methodologies developed will indicate what a systemic risk model provider must demonstrate and which tests are necessary. By involving the LNE, a public institution specializing in measurement and testing, the exercise is framed within a metrological approach, emphasizing reproducible protocols rather than isolated scores. In 2025, Wavestone published a barometer of AI security within large organizations. In collaboration with Capgemini, a digital services company, they incorporate intrusion testing practices and cyber risk management for their clients.

Model Access Still Controlled by Publishers

To evaluate Claude Mythos 5.2 or its competitors, the consortium will need direct access to the models, which is still governed by the publishers. In September, Anthropic limited access to Mythos 5.1 to U.S. organizations. Their cyber verification program, revamped in October, now outlines three access levels, the broadest of which is reviewed “in collaboration with the U.S. government.” European companies are left to procure results from these models through approved intermediaries. Additionally, Anthropic’s program requires prior approval for any model usage involving third parties, which also regulates the evaluator’s operations.

Neither Capgemini nor Wavestone have specified which versions of the models the evaluations will focus on or whether the results will be publicly available. The budget of €9 million for the six lots remains modest compared to the evaluation capabilities of major laboratories. While the framework establishes a method for assessment, it doesn’t yet guarantee access to the most advanced versions.

A Framework of Evidence for Model Buyers

For Chief Information Security Officers (CISOs) and IT Managers purchasing a model or related service, the implications extend beyond vendor compliance. The methodologies produced by the consortium will define the security evidence that general-purpose model providers are required to present to regulators, and consequently, those that buyers can demand in their own procurement processes.

In this way, the AI Office transitions from regulation to measurement. As long as the only assessments of offensive capabilities come from their creators, buyers negotiate based on the seller’s word. A European framework, if published, could provide a common benchmark for comparing suppliers based on empirical evidence rather than their claims.

Scroll to Top