How many steps does it take to make your phone "understand" a photo?
About 10 min readA Vision-Language Model (VLM) is a type of AI model that can process both images and text simultaneously. It does not just "read" text -- it can also "see" images, and then use natural language to describe, analyze, and answer questions about visual content.
The core architecture of a VLM consists of three components: a Vision Encoder that "sees" the image, a Language Model (LLM) that "speaks" in text, and a Connector (Projector) that acts as the bridge between them -- translating visual signals into representations the language model can understand.
A photo goes through these steps from input to text output:
Click any step to see its description
A VLM is like an assistant who can both look at photos and write about them -- first using its eyes (Vision Encoder) to see the image, then its brain (LLM) to form words that describe it.
Text-only LLMs are already large, and VLMs add an entire vision encoder on top -- meaning more parameters and more computation. But the real reason VLMs are "heavy" is the number of tokens each image produces.
Language models process images by splitting them into small patches, with each patch becoming a token. Higher resolution means quadratic token growth:
224 x 224 → ~196 tokens
336 x 336 → ~441 tokens
768 x 768 → ~2,304 tokens
A single photo can produce as many tokens as an entire long article. That is why VLMs demand so much memory and compute.
Drag the slider to adjust image resolution and watch the token count change:
Imagine building a mosaic -- the more pixels you want, the more tiles you need. A high-resolution image is like asking the VLM to read an entire book.
AtomGradient's Gemma-Prune research introduces a multi-stage compression pipeline specifically designed for the Gemma 3 4B VLM. Through three progressive compression stages, the model shrinks from 2.8 GB to 2.1 GB -- a 25% size reduction while maintaining excellent performance.
Click each stage for details and watch the model size shrink:
Multi-stage compression is like packing a suitcase -- first you remove what you don't need (pruning), then roll your clothes tightly to save space (quantization), and everything fits into a much smaller bag.
When a VLM is small enough and fast enough to run directly on a phone or tablet, it unlocks a range of fully offline visual understanding capabilities. Most importantly -- your photos never leave your device.
Take a photo and let AI automatically identify contents and generate descriptions. "This is an orange cat lying on a blue couch."
Point your camera at a document to extract and understand text. Supports handwriting, tables, and multilingual documents.
"How many calories in this dish?" "What model is this part?" Point at an image, ask a question, get an instant answer.
Describe surroundings for visually impaired users, read signs, identify products. AI becomes a personal pair of "eyes."
An on-device VLM is like giving your phone a pair of "AI eyes" -- it can understand everything you photograph, runs entirely on your device, needs no internet, and never leaks your privacy.
Vision encoder sees, language model speaks, connector translates -- three components working together so AI can both see and talk.
A single image produces 256-2,304 tokens. Doubling resolution quadruples the token count -- this is the key reason VLMs are large.
Through vision encoder optimization, structured pruning, and quantization, Gemma-Prune achieves 25% compression and 3.4x speedup.
Photos never leave the device, all visual understanding runs locally. AI capability and privacy protection are no longer at odds.
When vision models are small enough to fit in your pocket, AI truly gains a pair of eyes to see the world.