GPU instances now account for 14% of the computing costs for companies deploying AI at scale, as reported by Datadog. Without visibility into the actual consumption of these resources, IT teams are unable to allocate costs by project, forecast usage, or optimize their spending. To address this issue, Datadog has made its GPU Monitoring available to all clients, enhancing visibility and leading to better cost control.
Teams managing shared GPU pools across various projects are looking to identify resource-heavy workloads, anticipate capacity needs, and detect underutilized resources before resorting to over-provisioning. Previously available tools provided only general metrics on equipment status, lacking the connection to application workloads and the responsible teams. This disconnect between hardware telemetry and application context extended diagnostic cycles and made it impossible to accurately attribute consumption to specific teams or projects.
GPU Monitoring directly links GPU telemetry to the workloads consuming these resources. Platform engineering and machine learning teams can now share insights to conduct joint investigations. “Most companies are still in the experimental stage, struggling to achieve a comprehensive and cohesive view of their entire stack. GPU monitoring addresses this challenge, providing an unprecedented level of efficiency and reliability,” stated Yanbing Li, Chief Product Officer at Datadog.
Capacity Planning and Team-Based Attribution
The financial governance challenge of GPU usage is a structural issue for organizations deploying AI at scale. Without precise measurements of consumption by team and workload, financial management faces a murky cost line that is difficult to adjudicate. “Companies lack the means to allocate GPU expenses across their various units and do not have sufficient visibility into workload contexts or optimization levers. In such circumstances, budgeting and planning effectively become challenging,” explained Li. GPU Monitoring addresses these issues through two main mechanisms. The first is measuring consumption by team: assigning accountability for GPU usage and identifying instances of over-allocation and under-utilization to reclaim and reallocate resources. The second is capacity forecasting: the platform’s recommendations indicate whether to acquire new GPUs or optimize existing ones based on actual usage data.
This level of granularity in measurement enables informed financial decisions. Platform teams can avoid knee-jerk purchases triggered by saturation alerts without analytical context, machine learning teams can access the capacities they need more quickly, and management can rely on accurate data to guide GPU investments during budget planning cycles.
Faster Diagnostics and Prevention of Production Failures
Beyond the financial dimension, GPU Monitoring tackles a critical operational challenge for teams deploying inference models in production. Workloads in queue are directly linked to the GPUs, pods, and processes executing them, allowing teams to identify bottlenecks within minutes instead of hours. Failing GPUs can be proactively identified before failures ripple across the cluster, preventing delays in both learning and inference phases.
Hyperbolic, a multi-tenant GPU infrastructure provider, was one of the first adopters of the solution. “Without additional configuration, we immediately gain visibility, both by instance and device, into core usage, memory, power, and temperature. We can directly correlate a spike in model latency to the underlying GPU metrics without switching tools,” remarked Kai Huang, Head of Product at Hyperbolic. This ability to correlate application latency with hardware state within a single interface eliminates the silos between infrastructure and machine learning teams, a common factor prolonging incident resolution cycles in AI production environments.
A Tool for CFOs as Much as for CIOs
The precise measurement of GPU consumption by team, project, and workload generates actionable metrics for financial departments: cost attribution, actual usage rates, expenditure forecasting, and return on investment for GPU purchases. This depth of data transforms an opaque cost line into actionable insights, a necessary prerequisite for serious financial decision-making in large organizations where GPU budgets are now comparable to traditional major infrastructure projects.
GPU Monitoring integrates with Datadog’s SaaS platform, a NASDAQ-listed company subject to the Cloud Act. European organizations handling sensitive data through the platform need to consider this in their risk assessments, especially for workloads involving personal data or those subject to NIS2 obligations. The solution is now available to all Datadog clients.



