Third-person
Scene dynamics, material interactions, motion, and state evolution across diverse physical scenes.
PHYSICALLY GROUNDED VIDEO CAPTIONING
A Cross-Domain Benchmark for Physically Grounded Video Captioning
1 Beijing Academy of Artificial Intelligence2 Peking University3 Tsinghua University4 Beihang University5 Physis
WHY PHYSCAPBENCH
A physical data yardstick
for world models.
To our knowledge, the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations.
Videos contain contact, motion, deformation, and recovery. A caption can name the activity and still leave these processes out. What physical information survives when video becomes language?
Born from world-model pretraining practice, PhysCapBench provides a common yardstick: human-verified physical references, seven evaluation dimensions, and downstream tests in generation and manipulation.
The benchmark in motion
01 / THE BENCHMARKOne benchmark.
One benchmark.
Three views of the physical world.
Task-level descriptions can name an activity while missing the physical process. PhysCapBench evaluates both accuracy and coverage of observable interactions.
THE DATASETThree domains.
Three domains.
A shared physical perspective.
From scene-level dynamics to human and gripper manipulation, PhysCapBench brings 500 examples from each domain into one physical-description framework.
1,500videos & demonstration episodes
3 domainsone physical-description framework
7 dimensionsmeasure what captions retain
Three domains of reference data and annotation; current caption-strategy results cover third-person and egocentric videos. Handheld-gripper rankings will follow.
Egocentric
Human manipulation with explicit hand identities, object roles, and visible changes.
Handheld grippers
Human-operated demonstrations with synchronized viewpoints and trajectory cues for annotation.
Each domain contains 500 videos or demonstration episodes. Egocentric videos include paired hand tracking (approximately 2.21 hours in total); handheld-gripper episodes include synchronized high, left-wrist, and right-wrist views with gripper trajectories. Multiple camera views count as one episode.
02 / BUILDING THE YARDSTICK
What is a physical event?
A time-localized interaction: who acts on what, how contact or motion changes, and what state remains afterward.
Find the evidence
Inspect RGB frames. Hand motion, interaction cues, synchronized views, and gripper trajectories help identify changes worth revisiting.
Structure the process
Assign entity identities and roles. Record event intervals, motion, contact transitions, before-and-after states, and supported relations.
Verify every claim
Revisit the source frames, correct unsupported details, and check identities, timing, and coverage. Seven professional annotators verify and cross-check the records.
THREE DOMAINS / LIVE PHYSICAL EVENTS
Different worlds. Physics in motion.
Three perspectives. Every event in focus.
All three domains play togetherSelect any clip to inspect its event record ↗
How the seven-dimensional score works ＋
01 / Event coverage
Are the important events retained?
02 / Objects and roles
Who acts, with which instrument, on what?
03 / Actions and motion
How do the participants move?
04 / Contact and interaction
What touches, supports, grasps, or separates?
05 / States and outcomes
What changes, and what remains afterward?
06 / Physical mechanisms and causality
Which mechanisms and causal links are supported?
07 / Temporal order
Are sequence and overlapping events preserved?
Doubao Seed 2.1 Pro compares captions against fixed reference content at temperature 0, with thinking disabled. Scores measure semantic agreement and coverage on a 0–100 scale. Applicable dimensions are averaged per video; overall results average those per-video scores.
03 / MEASURING PHYSICAL INFORMATION
Reported strategies: 7PhysCapBench Leaderboard
Captioning strategies evaluated across seven physical dimensions.
Scores: 0–100 · Higher is better · Scroll horizontally for all dimensions
Egocentric captioning strategies evaluated against shared references. Overall scores are averaged per video over applicable dimensions; they are not the mean of the displayed dimension averages. Equal scores share the same rank.
03
Data ready. Results to come.
The benchmark includes 500 handheld-gripper episodes.
The supplied report does not publish a caption-strategy leaderboard for this domain.
04 / FROM MEASUREMENT TO MODEL BEHAVIOR
The details matter.
Here is the evidence.
Caption scores locate missing information. Reconstruction and policy experiments test whether richer physical descriptions are useful downstream.
Physis vs. Meta overall score on egocentric videos: 79.36 vs. 42.41.
Lower LPIPS for H3 on egocentric videos with physical captions.
Single-task success: 37.5% → 45.0%, across 40 shared held-out scenes.
01 / WHAT THE YARDSTICK REVEALS
Better ordering can still lose the outcome.
Compressing EgoVid Dense into EgoVid Action raises temporal-order scores from 63.75 to 67.50, while states and outcomes fall from 70.88 to 60.62. A single overall score would hide that trade-off.
Third-person
Physis leads overall at 74.07 and tops five dimensions. LingBot Temporal leads temporal order (72.84), while Cosmos3 Narrative leads mechanisms and causality (71.00).
Egocentric
Physis leads overall at 79.36. Cosmos Ego leads contact and interaction (83.57) and temporal order (77.50). Different strategies retain different physical information.
02 / VIDEO RECONSTRUCTION
A missing recovery changes what gets generated.
The source shows soft toys compressed under a board, then recovering their volume. The physical caption includes recovery after unloading; the metadata caption ends with revealing deformed toys.
METADATA CAPTION · EXCERPT“then lifts the board to reveal the deformed toys”
PHYSICAL CAPTION · EXCERPT“The group of soft rubber toys slowly returns to its original shape.”
Watch the state after each board is lifted: the physical-caption result retains volumetric toy shapes; Meta leaves flattened, sheet-like forms. The recovery speed and some later interactions remain imperfect. This is a selected qualitative example, not a measure of overall physical correctness.
Matched source first frame and target duration; only caption conditioning changes. H3: 33B; LingBot-World 2.0: 14B. Reductions are relative to Meta. LPIPS and PVCS measure perceptual distance to the source, not physical accuracy.
More visual evidence: hand roles, tool contact, and material evolution ＋
Drilling and contact termination
Physical conditioning better retains localized debris and the drill bit separating from the workpiece.
Plume structure and barrel orientation
Physical conditioning better retains the persistent plume and the barrel tipping onto its side.
03 / MANIPULATION POLICIES
Training Instruction Upsampler
Physical instruction enrichment specifies the demonstrated sequence, acting gripper, target placement, and post-release state. The report tests both a controlled single-task comparison and four policy families across selected tasks.
| Benchmark | Original | Enriched | Gain |
|---|---|---|---|
| RoboTwin 2.0 | 27.27 | 69.44 | +42.17 |
| RoboDojo Sim | 34.68 | 73.26 | +38.58 |
| Held-out scenes | Raw | Enhanced | Gain (pp) |
|---|---|---|---|
| Standard · 20 scenes | 50.0 | 55.0 | +5.0 |
| Randomized · 20 scenes | 25.0 | 35.0 | +10.0 |
| Overall · 40 scenes | 37.5 | 45.0 | +7.5 |
Same 50 demonstrations, 23,098 frames, action targets, sampling indices, and training settings. 5,000 LoRA updates; batch size 32; training seed 0. Half of training indices receive enriched instructions. Both policies use the same original short instruction at execution time.
| Policy | RoboTwin baseline | Enriched | Gain (pp) | RoboDojo baseline | Enriched | Gain (pp) |
|---|---|---|---|---|---|---|
| π₀.₅ | 50.4 | 51.9 | +1.5 | 30.6 | 32.6 | +2.0 |
| X-VLA | 44.4 | 45.4 | +1.0 | 30.9 | 32.2 | +1.3 |
| GalaxeaVLA | 29.1 | 30.1 | +1.0 | 13.6 | 14.9 | +1.3 |
| starVLA | 23.5 | 24.0 | +0.5 | 13.7 | 14.3 | +0.6 |
Official baselines versus enriched-instruction policies in the report’s selected-task evaluation. These are task-subset means, not full-benchmark rankings. Gains are percentage-point differences between displayed values.
Physical interaction examples
FailureSuccess
A YARDSTICK FOR BETTER DATAFind what is missing.
Find what is missing.
Improve the description.
Test what changes.
Use PhysCapBench to compare captioning strategies, diagnose omitted physical information, refine data construction, and connect description quality to downstream behavior.
Start with the evaluation code ↗EXPLORE PHYSCAPBENCH
Build on physical understanding.
Read the tech report and explore the code and dataset.