Scene clips rebuilt from the gallery's own footage at 1920×1080,
with the trajectory, per-gate scores and failure captions drawn in — scrub to any frame and
download it. Below them, five analysis figures, each paired with the published precedent it
copies, or with a note that no precedent exists.
Scene clips
The same scenes as video. These are rebuilt from the gallery's own clips
— the camera frames are decoded out of the existing MP4s, so no dataset access is needed
— then re-composited against the corrected BEV.
1920×1080 at 20 fps, ~8.6 s, against the gallery's
1500×540 at 2 fps for 3.5 s.
What the clip adds over the gallery version
The trajectory is in the video. The gallery clips contain no
trajectory lines at all — the viewer draws them in the browser, so the MP4 alone is
nearly empty. Here the prediction is baked in, and projected into the camera image too.
Two phases. ~2 s of history, cross-faded to 20 fps instead of
stuttering at 2, then a 4 s future rollout where the camera holds and the BEV animates
at 10 Hz, with agent boxes interpolated between the nine stored timesteps.
Live state. Per-gate chips, a running clock, a scale bar, and the
failure caption surfaced when it applies.
frame 0 / 171 · t = 0.00 s
Download this frame
00016f8b45c25a1d · PDMS 0.000 · fails DAC, DDC
frame 0 / 171 · t = 0.00 s
Download this frame
000bf168dd5159af · PDMS 0.000 · fails DAC
frame 0 / 171 · t = 0.00 s
Download this frame
037b976caae85af5 · PDMS 0.000 · fails DAC
frame 0 / 171 · t = 0.00 s
Download this frame
04f548a98d695310 · PDMS 0.000 · fails NC, TTC
frame 0 / 171 · t = 0.00 s
Download this frame
00f925ab5406520f · PDMS 0.583 · fails TTC
frame 0 / 171 · t = 0.00 s
Download this frame
011376b151f2526c · PDMS 0.583 · fails TTC
frame 0 / 171 · t = 0.00 s
Download this frame
0436604d25145231 · PDMS 1.000 · all gates pass
frame 0 / 171 · t = 0.00 s
Download this frame
0dd23f4ea286544f · PDMS 1.000 · all gates pass
Downloading a frame
Press play, pause where you want, then use the buttons under the video.
The arrows step one frame (1/20 s) at a time, and the readout shows the exact timestamp
and frame index so a figure caption can cite it.
Camera (left) and BEV (right) save the two halves as
separate PNGs, cropped at the renderer's own panel rectangles — the crop boxes come
from scenes_v2/clip_meta.json, written by the render job, so they
cannot drift out of sync with the layout.
Full frame saves the whole 1920×1080 composite.
The chips under the slider jump straight to a time point — dashed ones are in
the replayed history (t < 0), solid ones in the future rollout. The chip for
the segment you are in highlights as you scrub.
Each clip is 172 frames — 40 of replayed history
(t = −1.5…0.0 s) then 132 of future rollout
(t = 0.0…4.0 s), at 20 fps for 8.6 s. The slider steps one
frame at a time.
Metadata (JSON) saves everything known about that scene: all per-gate
sub-scores, PDMS for every method evaluated on it, ego speed, driving command, heading
change and curvature, agent count, the failure captions, the BEV bounds, and the timestamp
you were paused at.
Ordered worst PDMS first. Posters are the t = +0.6 s frame;
nothing autoplays and nothing loads until you press play.
Analysis figures
What a survey of the published figures actually shows
Seven primary sources were enumerated figure-by-figure — the NAVSIM
benchmark paper (arXiv:2406.15349), Hydra-MDP (2406.06978), DiffusionDrive (2411.15139),
DriveSuprim (2506.06659), GTRS (2506.06664), TransFuser (2205.15997) and Bench2Drive
(2406.03877). Three results shaped the choices below.
The multi-component score is never charted. Sub-score decomposition
(NC / DAC / EP / TTC / Comfort) appears only as table columns, in all seven. No stacked bars,
no radar plots, no parallel coordinates anywhere. The figure slot is open, not forbidden
— the absence is plausibly page-limit economics.
Seed variance is reported, but never plotted. NAVSIM quotes
±0.56 PDMS over three seeds in ablation prose; TransFuser gives Table VI to a
3-seed × 3-run study. Always a table or an inline ±. A figure that draws the
noise band would be a departure from convention, and arguably an improvement.
The correlation figure already has a canonical form. NAVSIM's own
Fig. 3(a) is a per-planner scatter of open-loop against closed-loop score with fitted
trendlines, and Fig. 4(a) is a grouped bar chart of Spearman and Pearson coefficients by
planner type. A loss-versus-metric figure should look like those.
One hard constraint: never stack PDMS
PDMS is a gated composite, not a sum. From the benchmark paper's Eq. 1:
PDMS = NC × DAC ×
(5·EP + 5·TTC + 2·C) / 12
NC and DAC enter multiplicatively as hard gates; the rest is a
weighted average normalised by 12. No assignment of the five sub-scores to stacked segments
sums to PDMS, so a stacked bar is not merely unconventional here — it is wrong, and a
reviewer who knows the metric will see it immediately. Grouped bars and small multiples are
unaffected. The same gated structure holds for v2 EPDMS, so the argument survives the metric
change.
1 · Rank-correlation bars
Does the quantity we optimise rank scenes the way the benchmark does? One bar
per model per predictor, on a diverging axis so the sign is unmissable.
Training loss and ADE as rank predictors of PDMS
further left = better
training lossADE
Spearman ρ against per-scene PDMS, interior scenes only
(0 < score < 1). A bar at −1 would mean the predictor ranks scenes exactly as the
metric does; zero means no relationship. Everything sits between −0.15 and 0.
Constant-velocity is excluded: it scores exactly zero on 8,130 of 12,146 scenes, so its
interior is 4,016 unrepresentative scenes and its ρ of +0.378 would dominate the axis
without meaning much.
PrecedentNAVSIM Fig. 4(a) — grouped Spearman/Pearson bars by planner type
TrapQuoting one pooled ρ over all scenes — the 26.8% of scenes tied at
exactly 1.000 inflate it. Report full and interior separately.
2 · Ablation deltas against a seed-noise band
The figure that kills over-reading. Each model's distance from the reference,
with the measured run-to-run spread drawn behind it: anything inside the band is not separated
from luck.
PDMS points relative to cmd_tokens_filmgrey band = run-to-run spread
The band is ±2.33 PDMS points, the spread between our model and its
seed-43 replication. That pair differs in seed and objective, so it is an upper bound
on seed variance, not a measurement of it — which makes it the conservative choice for this
figure. Three of five deltas fall inside it. Only cls_tokens_film
clears it decisively.
State the scope, or this figure overclaims
This band governs separate training runs. It does not govern paired
ablations evaluated on a frozen policy over the same scenes, where the pipeline reproduces to
~1e−5 and a 0.02 delta is real. A figure like this without that caption invites the
reader to dismiss every paired result in the paper.
PrecedentNone as a figure. NAVSIM and TransFuser both report seed spread as a
table or inline ±.
AnswersWhich of these differences are real?
NeedsAt least two runs of one config. Three to five would let you draw a real
interval instead of a range.
TrapDrawing a range from two runs and calling it a standard deviation. It bounds
nothing formally; say so.
3 · Per-bin failure profile
Where the score is lost, and what those scenes look like. Scenes binned by
score, then characterised by scene properties rather than by model.
Scene properties across the score range
bins run worst → best
drivable-area margin (m)agents in scenescenes per bin
Margin rises almost monotonically with score, 2.84 m in the failure bin against
4.69 m at the top — scenes fail where the road is tight. Agent count rises
too, which is the opposite of the intuition that traffic makes scenes hard; plausibly
dense traffic implies wide multi-lane roads. Bin counts are drawn behind because the extremes
are thin: 180 scenes in the failure bin against 3,839 at the top.
PrecedentBEV-Planner's command-conditioned straight-vs-turning split — the
analysis is established, but published as a table.
AnswersWhat kind of scene do we fail on?
NeedsPer-scene metadata joined to per-scene scores. We have speed, command, agent
count, margin and curvature.
TrapOmitting bin counts. A dramatic trend across bins holding 180 scenes and 3,839
scenes reads very differently once the reader can see the n.
4 · Oracle-versus-K curve
How much of the headroom is reachable by choosing rather than by
regressing better. Plotted as a curve, which is the part nobody publishes.
Best-of-K over the candidate pool
↑ better
Best-of-K has not plateaued at K=63. The reference lines are what
make the curve legible: a perfect chooser over the 40-entry vocabulary reaches 0.9456 against
our 0.7758, so the prize is +0.170 PDMS — but the honest floor is the best single constant
vocabulary entry at 0.645, not zero, and our policy already sits above it.
The privileged-pool trap
The wider pool behind this curve contains ground-truth-derived perturbations, and quoting
best-of-63 from it gives +0.194. The 40-entry vocabulary is the only unprivileged pool and it
gives +0.170. Worse, the human trajectory sits at a fixed column index in
every scene, so a naive “best constant column” baseline is just always pick the
human. Any figure like this must say which pool it used.
PrecedentDriveSuprim Table 1 gives PDMS at K = 1, 4, 16, 256 — as a table. The
curve is an unclaimed slot.
AnswersIs candidate quality or candidate selection the binding constraint?
NeedsEvery candidate scored by the real scorer, not a proxy.
TrapA linear K axis. The interesting structure is all at small K; use log.
5 · Sub-score small multiples, and the gates behind them
The decomposition figure the field does not print. Six small panels, one per
component, on a shared axis — never stacked, and never a radar.
PDMS components by model
shared axis per panel · ↑ better
Two components are saturated — comfort sits at 0.999–1.000 for every model
and DDC above 0.96 — which is exactly why a radar plot misleads here: it would spend two of
six axes on components that never move, and its enclosed area has no meaning under a
multiplicative metric. Ego progress is where the models actually differ, and where the human
reference is furthest from 1.0.
The same models by hard-gate failure rate
↓ better
Percentage of the 12,146 scenes failing each gate. This separates the models far
better than the means above: drivable-area failures span 7.0% to 13.1% here, while the
corresponding sub-score means compress into 0.93–0.87. If a paper prints only one of
these two panels, print this one. Note the collision gate is three-valued
(1.0 / 0.5 / 0.0), so “failure” means < 1.0, not
= 0.
PrecedentNone. Zero sub-score charts across seven papers — tables only.
AnswersWhich component is the headroom actually in?
NeedsPer-scene sub-score columns, already in the eval CSVs.
TrapStacking (invalid — see above), or a radar plot whose area is meaningless
and whose saturated axes waste half the figure.