[ACL 2024] VisAidMath: Cracking the "Reasoning Illusion" in Multi-modal Math

TL;DR

While Large Multi-modal Models (LMMs) like GPT-4V and Doubao-seed-1.6 seem to solve complex math problems, they are often just "guessing" the right answer via text shortcuts. VisAidMath is a new benchmark of 1,200 problems that forces models to generate visual aids (like auxiliary lines). The result? A massive performance collapse, revealing that models barely use visual logic to reach their conclusions.

The Problem: The "Lucky Guess" Phenomenon

In traditional benchmarks like MathVista, models are judged purely on their final answer. However, the authors of VisAidMath argue that mathematical problem-solving (MPS) is inherently a multi-step, multi-modal process. In geometry or calculus, humans don't just "see" the answer; we draw auxiliary lines, establish coordinate systems, and perform spatial deductions.

Current SOTA models are prone to what the authors call the "Reasoning Illusion": achieving high accuracy while their internal reasoning steps are either missing, hallucinated, or entirely irrelevant to the visual context provided.

Methodology: The Three-Layered Funnel

To expose this illusion, the paper introduces a rigorous evaluation framework that acts like a sieve:

1. ACCU (Standard Accuracy): Did the model get the right answer?
2. PVA (Process-Verified Accuracy): Is the answer correct and the reasoning free of fatal logical flaws?
3. SPRS (Solution Process Robustness Score): A fine-grained metric that penalizes even minor systemic glitches in the reasoning chain.

Task Definitions

The benchmark defines three specific scenarios to probe model depth:

• General Reasoning (GR): The baseline "solve this" prompt.
• Direct Visual-Aided Reasoning (D-VAR): The model must first describe a visual aid it needs, then solve.
• Indirect Visual-Aided Reasoning (I-VAR): The model is given a "Golden" visual aid and must use it effectively.

Radical Findings: Models Prefer to "Evade"

The most shocking discovery was the Evasion of Visual Reasoning. After analyzing 200 correct samples in the D-VAR task, the authors found that only 3% of successful attempts actually used the visual aid as intended.

• 41.1% of models defaulted to pure arithmetic logic.
• 19.3% ignored the visual context entirely.

When models did try to generate visual aids, they failed miserably. For instance, GPT-4V only achieved 45.33% accuracy in the D-VAR task, and providing it with a "Golden" aid (I-VAR) actually decreased its performance, suggesting the model struggled to integrate external visual logic with its internal textual patterns.

Diagnostic Insights: Catastrophic vs. Systemic Failures

The paper categorizes model failures into two personas:

• The "Catastrophic" Model (e.g., InternVL3.5): High ACCU but very low PVA. These models essentially "hallucinate" their way to a correct answer; if they make a mistake, it’s a total logic collapse.
• The "Sloppy" Model (e.g., Doubao-seed-1.6): Higher PVA but lower SPRS. These models understand the logic but are rife with minor inaccuracies that would fail a human-level rigorous check.

Conclusion and Future Outlook

VisAidMath proves that we are far from "solved" when it comes to multi-modal mathematical reasoning. The benchmark establishes that:

1. Visual Planning is the Missing Link: Models can "see" and "calculate," but they cannot "construct" the necessary visual steps to simplify a problem.
2. Metric Reform is Necessary: Standard accuracy is a deceptive metric for reasoning. Success in the future requires models to bridge the gap between spatial imagination and formal logic.

The authors have opened the door for future research in Mathematical Image Generation and Spatial Capability—moving beyond identifying what is in a picture to understanding how to manipulate that space to solve a problem.