Multimodal AI — How It Understands Images, Audio, and Video
How a text-only model learned to look at pictures
① Multimodal isn't a wholly new technology. The core trick is converting images and audio into coordinates the exact same way text gets turned into tokens and embeddings — covered in how LLMs work — and dropping them into the same coordinate space.
② That's why a single image costs far more than text — it gets sliced into small patches, and that explodes the token count.
③ "Seeing" and "understanding" aren't the same thing. The model is still picking the most probable next piece, so it's weak at counting things precisely or measuring exactly.
① What "multimodal" actually means
Marketing tends to frame it mystically — "an AI that can see pictures and hear voices" — but the technical reality is simpler. The kinds of input the model handles (its modalities) expanded from just text to images, audio, and video.
One boundary is worth drawing up front. This article covers models that read and understand photos, audio, and video (like answering a question about a picture). It does not cover models that generate images or video from text (image-generation AI, for instance) — those run on diffusion models, a completely different family of technology, and are out of scope here. Conflating the two makes everything that follows confusing.
② The core idea — a photo eventually becomes "tokens" too
As covered in how LLMs work, text goes through three steps: ① split into tokens, ② each token converted into a coordinate called an embedding, ③ tokens reference each other via attention. Multimodal simply adds one more path into step ①.
An image gets sliced into small square patches — say, 16×16 pixels each — so even an ordinary photo breaks into hundreds of patches. Each patch runs through its own encoder and comes out as a coordinate (embedding) in the exact same form as a text token. Audio works the same way: it's chopped into short time windows (say, 20 milliseconds each), and each window becomes a coordinate.
Attention computes how much one coordinate should reference another. That computation doesn't distinguish whether a coordinate originated from a word or an image patch. If training places the coordinate for the word "cat" near the coordinate for a patch of a cat photo, the model can effectively treat them as the same thing. The exact structure that already understood text keeps working once the input types simply expand.
③ How it's actually assembled
Break the full pipeline into three parts:
| Stage | What it does |
|---|---|
| ① Modality-specific encoder | Runs image patches or audio segments through a dedicated network to produce numeric vectors |
| ② Projection | Rescales vectors from different encoders so they land in the same dimensional space as text embeddings |
| ③ The same language model | Concatenates text-token embeddings with image/audio embeddings into one sequence and feeds it into the original attention and next-token-prediction machinery, unchanged |
From the model's point of view, "this sentence, then a photo, then a question" is just one continuous sequence of coordinates. It isn't a new kind of model built from scratch — it's closer to bolting an extra converter onto the front door of the existing architecture.
④ Why a single photo costs so much
As covered in tokens and the context window, both cost and processing speed scale with token count. And because images get sliced into patches, they generate far more tokens than text does.
| Input | Approximate token count |
|---|---|
| One line of text (about 20 English words) | 25–35 |
| A typical photo | Hundreds to around 1,000 |
| A high-resolution image or full screenshot | Thousands |
| 1 minute of video (sampled at 1 frame/sec) | Tens of thousands or more — each frame adds up like another image |
Video is especially expensive. Video is fundamentally a sequence of image frames laid out over time, so even a few seconds of it generates roughly the same token volume as stitching together that many separate images. That's why feeding in a long video can fill up a context window almost instantly.
⑤ The limits — "seeing" and "understanding" aren't the same
The model is still choosing the most probable next piece. Looking at an image isn't "perceiving" it the way a person does — it's probabilistically reconstructing the kind of description that tended to accompany similar patterns in training data. That produces a few characteristic weaknesses.
· Precise counting — struggles with fine-grained counts, like whether a photo shows 7 or 8 people. It's good at "roughly several," but an exact number is just a probabilistic guess.
· Precise position or measurement — confidently gives a wrong answer to "how long is this line, in cm." It didn't measure with a ruler; it picked a plausible-sounding number.
· Details not actually in the image — the same mechanism as hallucination. If a detail is the kind that's commonly present in similar photos, the model will add it even if it isn't there.
This isn't a bug — it's a direct consequence of the architecture. As our hallucination guide covers, there is no step anywhere in the pipeline that checks whether an answer is actually true. The same mechanism that invents a nonexistent paper title in text shows up here as inventing an object or a number that isn't in the image.
⑥ Practical use — where it's ready, and where it isn't
| Use case | Fit | Why |
|---|---|---|
| Summarizing/describing a photo or screen | ✅ Good fit | Rough context capture is a strength |
| Reading scanned documents/tables | ✅ Good fit | Text recognition happens inside the same pipeline without a separate OCR step — see reading documents with AI |
| Summarizing an audio meeting | ✅ Good fit | Strong at grasping context and gist |
| Exactly counting objects in a photo | ❌ Risky | Probabilistic estimate — always verify manually |
| Medical-image diagnosis or safety inspection | ❌ Risky | Misses or false positives can be severe; needs expert review |
| Precise dimensions or distance measurement | ⚠️ Caution | Treat as a reference only; verify with an actual measuring tool |
□ First decide whether the task needs rough understanding or an exact number/verdict.
□ When exact numbers matter, treat the model's answer as a draft, not a final value, and verify it separately.
□ Cross-check the answer against the original image for details that aren't actually there.
□ Route any safety-, medical-, or legally consequential image analysis through a human sign-off.
⑦ The investor angle — why multimodal amplifies chip demand
As section ④ showed, images and video generate dramatically more tokens than text does — meaning each query requires more compute. Part of the "surging inference demand" story covered in our semiconductor sector deep dive comes from exactly this: as services expand from text-only to multimodal handling of images, audio, and video, token consumption per query rises on its own. Even with a flat user base, inference-server demand can climb simply because more people develop the habit of attaching a photo to their prompt.
※ This article describes the general architecture of publicly documented multimodal models as of September 2026, and does not evaluate any specific company or product or constitute investment advice. Specific implementations vary by model.
※ This guide is provided for general educational purposes and simplifies technical details for readability.
New guides, when they land
We publish AI literacy guides twice a week. Subscribe and the next one comes to you — free, unsubscribe anytime.