Planning-Aligned Pretraining of BEV Representations with Sparse Action-Conditioned Targets for End-to-End Autonomous Driving

Jaeha Song  ·  Soonmin Hwang Hanyang University

PAVER aligns BEV pretraining with candidate ego motions: a single LiDAR sweep provides sparse targets for occupied and unobserved regions along each path. A 10K-parameter head learns to predict these targets without driving-task labels or dense scene reconstruction. Transferring only the BEV encoder improves end-to-end driving under shorter training schedules, while preserving the downstream architecture and camera-only inference.

PaperCode
Planning safety
0.51%→0.19%
−63% VAD-Tiny at epoch 30, averaged over the 1, 2 and 3 second horizons.
Training cost
21.3h→13.6h
1.57× faster VAD-Tiny on four RTX 5090 GPUs, including the 20 pretraining epochs.
Closed-loop
48.5→58.8
+10.3 driving score UniAD-Tiny on Bench2Drive Town05 Long, nine routes driven once each.

Teaser

Predictions and Grad-CAM

Six surround cameras with predicted boxes and futures, over the Grad-CAM of the BEV features behind them. PAVER captures the objects and the surrounding scene more completely, and plans better for it.

0:00/0:00

Abstract

End-to-end autonomous driving relies on a shared bird’s-eye-view (BEV) representation for perception, prediction, and planning, yet the BEV encoder is typically optimized only through heterogeneous downstream tasks. This leaves no dedicated stage for organizing the representation around geometry relevant to candidate actions. We introduce PAVER, Planning-Aligned BEV Encoder Pretraining, which rasterizes a LiDAR sweep into sparse free, occupied, and unknown cells. Rule-based candidate ego motions are rolled out and queried laterally across the vehicle width; the fractions of queries falling on measured returns and on unobserved cells form sparse action targets.

To limit direct copying from the supervised regions, PAVER replaces action-corridor features with a shared learnable mask token and predicts these targets from masked camera-derived BEV features conditioned on the corresponding action state. The framework requires no driving-task annotations or learned teacher and introduces only 10K trainable auxiliary parameters. After pretraining, the target builder and prediction components are discarded and only the BEV encoder is transferred, leaving the downstream architecture and inference pipeline unchanged.

On nuScenes, PAVER lowers VAD-Tiny’s average collision rate from 0.51% to 0.19% and its planning L2 from 0.66 to 0.60, while improving motion ADE from 0.91 to 0.80 and map mAP from 0.42 to 0.44. Planning L2 improves on VAD-Base and GenAD as well, although the collision gain does not carry over to VAD-Base.

Results overview

Planning performance and pretraining head size

Arrows run from each baseline to the same architecture with PAVER. Hollow markers are reconstruction-based pretraining, as its authors report it.

Table 1

Multi-task performance

Planning, motion prediction, detection, and mapping results on nuScenes for VAD-Tiny, VAD-Base, and GenAD, with and without PAVER pretraining.

Interactive 3D visualization

Both models decode the same clip: 3D boxes with object futures, the vector map, and the planned trajectory at one instant.

Model comparison

0:00/0:00
Layers
0.30

Pretraining objectives

Supervision sources and auxiliary-head requirements of driving pretraining methods.

Figure 2

Comparison of pretraining objectives

(a) supervises with perception labels, (b) reconstructs scene content and transfers the image backbone, and (c) PAVER supervises the BEV encoder with sparse action targets read off a single LiDAR sweep.

Training efficiency

Estimated total training time, including pretraining, compared with scratch training.

Figure 1

Total training time

Measured end to end on four NVIDIA RTX 5090 GPUs. The PAVER rows carry their own 20 pretraining epochs and still finish sooner than the scratch baselines.

Method

Perception supervision needs dense annotation; reconstruction needs a dense 3D objective. PAVER needs neither.

nuScenes results

Planning, mapping, and pretraining comparisons on the nuScenes validation split.

Table 5

Planning by horizon

Planning L2 error and collision rate at 1, 2, and 3 seconds for VAD-Tiny, VAD-Base, and GenAD, with and without PAVER pretraining.

Table 6

Comparison with reconstruction-based pretraining

Planning performance and temporary auxiliary-parameter counts for PAVER and reconstruction-based pretraining with UniPAD and MIM4D.

Table 3

Map prediction results

Average precision for dividers, pedestrian crossings, and road boundaries on nuScenes, with and without PAVER pretraining.

Table 2

Pseudo-LiDAR targets

VAD-Tiny downstream performance with no BEV pretraining, pseudo-LiDAR targets from monocular depth, or targets from measured LiDAR.

Closed-loop evaluation

Bench2Drive Town05 Long: nine routes, one repetition, identical protocol.

Table 7

Closed-loop metrics

Driving Score, Route Completion, and Infraction Score on Bench2Drive Town05 Long for UniAD-Tiny and GenAD, with and without PAVER pretraining.

Ablation studies

Effects of pretraining targets, feature masking, action conditioning, and training schedules.

Table 8

Pretraining targets and head size

VAD-Tiny downstream performance across pretraining targets and auxiliary-head sizes, using 20 pretraining epochs for every pretrained variant.

Table 9

Feature masking and action conditioning

VAD-Tiny downstream performance with action-corridor feature masking and action-state conditioning enabled separately or together.

Table 10

Pretraining and fine-tuning schedules

Downstream performance and estimated total training time for VAD-Tiny and VAD-Base across pretraining and fine-tuning schedules.

Representation analysis

Analyses of BEV representations, pretraining targets, and downstream learning dynamics.

Downstream learning curves

Supplementary results

Full metrics

Complete planning, motion prediction, detection, and mapping results, followed by class-wise detection and map metrics.

Citation

BibTeX

@misc{song2026paver,
  title  = {Planning-Aligned Pretraining of BEV Representations with Sparse Action-Conditioned Targets for End-to-End Autonomous Driving},
  author = {Jaeha Song and Soonmin Hwang},
  year   = {2026},
  eprint = {2609.22868},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  doi    = {10.48550/arXiv.2609.22868},
  url    = {https://arxiv.org/abs/2609.22868}
}