Vision Banana: Why Image Generators Are Secretly Master Vision Learners
Image Generators are Generalist Vision Learners
The paper introduces Vision Banana, a generalist vision model created by instruction-tuning the Nano Banana Pro image generator. It demonstrates that generative pretraining serves as a foundation for visual understanding, achieving SOTA results across 2D tasks (segmentation) and 3D tasks (metric depth and surface normals) while maintaining high-fidelity image generation and editing capabilities.
TL;DR
Is the ability to create art the same as the ability to understand the world? The researchers behind Vision Banana say yes. By taking a SOTA image generator (Nano Banana Pro) and giving it a light "nudge" via instruction-tuning, they’ve created a model that doesn't just draw cats—it understands depth, surface normals, and complex segmentation better than the most specialized "expert" models in the field.
The Core Insight: Generation is Understanding
For years, the AI community has treated Visual Perception (understanding what's in a photo) and Visual Generation (creating an image) as two separate paths. Perception used discriminative models (like ResNet or ViT), while Generation used Diffusion or GANs.
This paper challenges that wall. Much like how LLMs (GPT-4) learned to reason simply by learning to predict the next word, the authors argue that image generators learn a "hidden" physical model of the world just to make their generations look realistic. To unlock this, they used Lightweight Instruction-Tuning, treating tasks like depth estimation as just another "style" of image to be generated.
Methodology: The RGB Interface
The genius of Vision Banana lies in its simplicity. Instead of changing the model's architecture or adding "heads" for different tasks, it uses RGB as a Universal Interface.
1. The Invertible Depth Map
To teach the model metric depth (actual distance in meters), they mapped scalar depth values to a 3D Hilbert curve within the RGB color cube. This creates a "false-color" image that the model can generate, which can then be mathematically reversed back into precise metric distances.
Figure 1: Reframing perception as image generation through instruction-tuning.
2. Semantic and Referring Segmentation
The model is prompted with natural language: "Segment the cat in red and the background in yellow." Because the base model already understands "cat" and "red" from its generative training, it aligns these concepts to the pixel-level mask with startling precision.
Results: Beating the Specialists
Vision Banana was tested against "domain specialists"—models built for only one specific task. The results suggest we are at a turning point:
- 2D Segmentation: It outperformed SAM 3 (Segment Anything Model 3) on the Cityscapes benchmark (0.699 mIoU vs 0.652).
- 3D Geometry: In metric depth estimation, it surpassed Depth Anything 3 across four major datasets.
- Surface Normals: It defeated Lotus-2, providing much finer-grained detail in local geometry.
Table 1: Vision Banana vs. SOTA specialist models across various benchmarks.
The "Vibe Check" is perhaps most impressive. When fed a casual smartphone photo, Vision Banana produced a depth map that perfectly matched Google Maps' real-world measurements, despite never seeing such data during training.
Crucial Findings: No Sacrifice Required
A common fear with fine-tuning is "catastrophic forgetting"—the model becoming a good depth estimator but losing its soul as an artist. Vision Banana maintains its generative prowess. In human evaluations, it was preferred over its base model 53.5% of the time for general image generation tasks.
The Future: A Paradigm Shift
The success of Vision Banana suggests that Computer Vision is following the path of NLP. In the future, we may no longer build "segmentation models" or "depth models." Instead, we will build massive Foundational Vision Models through generative pretraining, then simply "ask" them to perform any visual task via an image-based interface.
Limitations
- Compute Cost: Running a full-scale image generator for a "simple" task like segmentation is currently much slower than using a tiny specialist model.
- Monocular Bias: The current work is focused on single images; extending this to temporally consistent video or multi-view 3D is the next frontier.
Conclusion
Vision Banana proves that the "internal representations" of generative models are far more powerful than we realized. By treating image generation as a universal language for vision, the authors have paved a clear path toward Artificial General Intelligence for Vision (AGI-V).
Keep an eye on this space—we are likely witnessing the end of the 'Specialist Model' era in Computer Vision.