AI Compute Requirements

Here is a number that should get your attention: training GPT-4 reportedly required around 25,000 specialized chips running for approximately 100 days. At current prices, that represents roughly $100 million in compute alone. AI does not run on magic; it runs on silicon. Specifically, it runs on highly specialized hardware capable of performing trillions of mathematical operations per second.

Here is the key insight: To lead an organization through an AI transformation, you need to understand the hardware layer. It determines what is possible, how fast you can innovate, and most importantly, what it will cost. If data is the fuel for AI, compute is the engine, and right now, that engine is incredibly expensive and difficult to acquire.

You might wonder why we need specialized chips at all. Your computer has a CPU (Central Processing Unit), which is a brilliant generalist. It is designed to handle complex logic and branching tasks one after another. However, AI training requires doing the exact same simple math—mostly matrix multiplication—billions of times simultaneously.

This is why GPUs (Graphics Processing Units) rule the AI world. Originally built to render video game graphics, they use "parallel processing" to perform thousands of simple calculations at once. While a CPU might have 16 powerful cores, a modern GPU has thousands of smaller cores working in concert. This architecture is orders of magnitude faster for the specific math AI requires.

In the AI world, we distinguish between two distinct phases: Training and Inference. Training is the one-time process of teaching the model from data—it is like creating a recipe. Inference is every time someone uses that model to make a prediction—like cooking from that recipe.

Training is where the massive upfront checks are written. To understand the scale, consider the cost of a single high-end H100 GPUcost of a single high-end H100 GPU, which can range from $25,000 to $40,000. For a frontier model, you need thousands of them. However, for most organizations, the ongoing cost of inference—every query your employees or customers send—will eventually exceed the training cost. In fact, Meta reports that 70% of their AI energyMeta reports that 70% of their AI energy goes to inference, not training.

The supply chain for these chips is incredibly narrow. One company, NVIDIA, controls roughly 80-90% of the AI chip market80-90% of the AI chip market. This dominance stems not just from their hardware, but from CUDA, a software layer that makes programming these chips possible. This creates a significant "concentration risk" for your strategy.

Furthermore, the most advanced chips are fabricated almost exclusively by TSMC in Taiwan. This geopolitical bottleneck means that a disruption in one region or a change in US export controlsUS export controls can affect the entire global AI industry overnight. As a leader, you must recognize that your access to "intelligence" is tied to a fragile physical supply chain.

Because compute is a physical, measurable resource, it is becoming a major lever for governance. Regulators and organizations are looking at "compute thresholds" to determine which models need extra oversight. For your organization, this means you must manage your silicon as carefully as your data.

I recommend using an Organizational AI Compute Governance ChecklistOrganizational AI Compute Governance Checklist to track your GPU usage, establish budgets for "exploding" cloud costs, and assess your dependencies on single hardware vendors. By understanding these requirements today, you can build a more resilient strategy that accounts for the physical reality behind the digital magic.