Outcomes
Win rates and game results remain visible, but they are no longer the only score.
Evaluation · Grounded social agents
QUACK
An open-source environment and evaluation framework that audits whether agent language is grounded in what the agent actually perceived and did.
Trajectories
Every episode is replayable from engine logs, tick by tick, for each agent.
Utterances
Each discussion claim is checked against the reconstructed ground-truth path.
What the audit flags
- Spatial hallucinationClaims about rooms or movement that the trajectory does not support.
- Unsupported accusationNaming another agent without grounded evidence.
- Deception collapseA hidden-role strategy that fails to stay consistent over time.
- Language–action inconsistencySaying one thing while the logged actions show another.
Abstract
Social deduction games test reasoning, deception, and coordination, but most agent environments still score only win rates and stay text-only. QUACK is a multimodal environment in which agents navigate partial views, talk in free-form discussion, and vote under hidden roles.
Its Statement Verification Pipeline reconstructs each agent’s trajectory and checks every claim against it. Across three frontier VLMs, even the strongest agent hallucinates 15.1% of verifiable spatial claims and makes more than half of its accusations without grounded evidence. The engine, evaluation framework, toolkit, and logs are public.