Towards Reliable VLM Judges:

State-Conditional Invariance and
Presentation-Aware Diagnostics

Qinan Zhang*Xinyu Wang*Qihang Jin*Tianyuan WangHongbing LuoPengfei LiWenjun LvYu KangJian Di†

* Equal contribution · † Corresponding author

University of Science and Technology of China

NeurIPS 2026 Evaluations and Datasets Track

Code Hugging Face

THE RELIABILITY PRINCIPLE

Stable when it should be.
Sensitive when it matters.

Presentation can change without changing the task outcome. A causal intervention cannot. A reliable judge must distinguish the two.

Towards Reliable VLM Judges overview: presentation invariance, action-conditioned dynamics, representation exchangeability and causal sensitivity

A real Full / NoCue / CF example

Loading recorded examples…

Same source group · simulator-grounded answers

INSIDE THE BENCHMARK

One benchmark. Five diagnostics.

From local observations to trajectory-level decisions, inspect the evidence behind each question.

QUESTION · ORIGINAL PROMPT

Task goal and response contract
SIMULATOR GROUND TRUTH

        
Original image 

      

      
Curated examples from a freshly generated base exam: A 103 · B 144 · C 486 · D 144 · E 325. Answers are simulator labels, not model predictions.

    


    
BEYOND A SINGLE SCORE
Correctness is only one axis.
01
Correctness
Does the verdict agree with the simulator-grounded task outcome?
02
Presentation invariance
Does the judgment survive semantics-preserving changes to the evidence interface?
03
Causal sensitivity
Does the judgment respond when an intervention changes the task outcome?


    
REPRODUCIBLE BY CONSTRUCTION
From trajectories to verdicts.
Generate images, build exams, collect responses and score them with the same pipeline.
Explore the code 

    
  

  
Original evidence
Original exam evidence