Foundation Models

For decades, if you wanted an AI to translate languages, you built a translation model. If you wanted it to summarize text, you built a summarization model. It was like buying a separate tool for every single repair job in your house.

Here is the key insight: Around 2020, the paradigm shifted. Researchers discovered that if you train a single neural network on enough data, it becomes a "Swiss Army Knife"—capable of translation, summarization, coding, and reasoning all at once. Stanford researchers termed these systems Foundation Models because they serve as the base upon which countless other applications are built. The most famous example is the public release of ChatGPTpublic release of ChatGPT, which is an application built on top of the GPT foundation model.

To govern these systems, you must distinguish between two phases of development. This distinction determines your costs and your risks.

Phase 1: Pre-training. This is the heavy lifting. The model is exposed to vast amounts of data—often substantial portions of the public internet—and learns to predict patterns. This phase requires massive computational resources. To understand the scale, consider the training cost of GPT-4training cost of GPT-4, which is estimated to be over $100 million. Very few organizations can afford this; most buy access to models someone else pre-trained.

Phase 2: Fine-tuning. This is where you adapt the general model to your specific needs. Think of pre-training as teaching a student to read and write, and fine-tuning as teaching them the specific format of your company's weekly reports. This process is much faster and cheaper, allowing you to customize a powerful model without the massive upfront investment.

You might wonder how a model trained on Wikipedia articles can help you write a marketing email. This relies on a concept called Transfer Learning. It is the ability of an AI to take knowledge gained from one task and apply it to a different but related task.

Because the foundation model learned the structure of language and logic during pre-training, it can "transfer" that understanding to new problems it hasn't seen before. This enables what we call "emergent capabilities," where the model develops skills—like writing code or translating languages—that it wasn't explicitly trained to do.

While foundation models offer incredible efficiency, they introduce a systemic risk called homogenization. If every company uses the same foundation model (e.g., GPT-4) to power their customer service, then a single security vulnerability or bias in that model affects everyone simultaneously.

For governance professionals, this means you are often inheriting risks from your vendors. If the foundation model has a tendency toward hallucinationhallucination, your application will too. Therefore, you cannot simply trust the API provider. You must perform rigorous checks using tools like a Foundation Model Due Diligence ToolkitFoundation Model Due Diligence Toolkit to assess the provider's safety testing, data handling, and liability terms before you integrate their model into your business.

Now that we understand the text-based foundation models, we need to explore how AI is learning to see and hear. In the next article, we will discuss Multimodal AI.