ELA: End-to-End Latent Action Policy Learning from Video Demonstrations

Cheng-Chun Hsu Peter Stone

University of Texas at Austin

CoRL 2026

Paper arXiv Code BibTeX

TL;DR ELA learns robot policies from abundant action-free video plus only a handful of action-labeled demonstrations. It trains latent actions end-to-end so they are both modelable (easy for a policy to predict) and decodable (easy to map back to robot actions).

Left: standard behavior cloning trains a policy on action-labeled demonstrations only; video demonstrations provide no supervision. Right: ELA has the policy predict latent actions, supervised on both action-labeled and video demonstrations, and an action decoder maps them to robot actions. — **Figure 1.** ELA overview. ELA extends existing behavior cloning methods, *e.g.*, diffusion policies, with latent actions jointly optimized for policy learning and action decoding. The approach enables training on mixtures of action-labeled and video demonstrations with little overhead, requiring only a lightweight Action Decoder.

Why latent actions fall short today

Robot action labels are expensive to collect, while video demonstrations, including human videos and unlabeled robot trajectories, are far easier to obtain at scale. Latent action modeling turns video into policy supervision: an inverse dynamics model (IDM) infers a latent action from a pair of frames, and a policy learns to predict it.

Existing methods, however, follow a two-stage pipeline. Latent actions are learned first, then frozen as targets for a downstream policy. Nothing shapes the latent space for control, so the latents can end up hard to predict, hard to decode, or both.

We identify two properties a latent action space needs for effective policy learning:

Modelability: latent actions can be reliably inferred from observations by a policy.
Decodability: latent actions can be faithfully mapped into executable continuous robot actions.

Scatter plot of modelability (x-axis) versus decodability (y-axis), lower is better on both. UniVLA, LAPA, LAPA-DINO and Villa-X sit in the upper region; ELA-DP without alignment is decodable but less modelable; full ELA-DP is in the lower-left corner. — **Figure 2.** Modelability vs. decodability of each latent action space. Both axes are action MSE on 20 held-out real-world robot demonstrations (lower is better). *Decodability* (MSE_idm) measures how well actions decode from latents the IDM extracts from frame transitions. *Modelability* (MSE_dp) measures the error when those latents are instead predicted by a diffusion policy from observations. Latent action models trained without an action decoder (UniVLA, LAPA, LAPA-DINO) are weak on both axes. ELA_DP (w/o align) is decodable but less modelable. Full ELA_DP is best on both.

Method

Policy learning through latent actions

ELA splits a behavior cloning policy into two parts: a Latent Action Policy π that predicts a latent action z from the current observation, and an Action Decoder ψ that maps z to a k-step chunk of robot actions.

The latent actions come from a Latent Action Model (LAM), made of an inverse dynamics model (IDM) that infers z_t from an observation pair (o_t, o_t+k) and a forward dynamics model (FDM) that reconstructs o_t+k from o_t and z_t.

Figure 3. ELA training and deployment. During training, the IDM infers latent actions from image pairs. These latents supervise the FDM (future-frame reconstruction), the Action Decoder (robot-action prediction), and the Latent Action Policy. During deployment, the policy predicts latent actions from observations and the Action Decoder executes them.

Because the IDM can label any video, the policy trains on both action-free and action-labeled data. Only the Action Decoder needs action labels.

End-to-end joint training

Training the LAM, policy, and Action Decoder in a single combined objective causes latent action collapse: policy gradients push the latent space toward whatever is easiest to predict, even when that hurts decoding. ELA avoids this with two ideas.

Representation alignment. Instead of sending policy gradients into the LAM, ELA aligns both the LAM and the policy to a shared reference representation from a frozen perceptual encoder (DINOv3).
Two alternating passes that improve modelability from complementary directions.

LAM-update pass Update Latent Action Model and Action Decoder

Policy-update pass Update Latent Action Policy

Figure 4. End-to-end joint training. ELA alternates two passes over the same modules, coordinated through a shared Reference Representation from a frozen Perceptual Encoder. The LAM-update pass freezes the policy and reshapes latent actions so they are easier for the policy to predict; the Policy-update pass freezes the LAM and improves the policy's ability to predict them.

Real-world results: learning from human video

For each task, ELA uses 10 action-labeled robot demonstrations and 40 human video demonstrations. Tasks range from pick-and-place to articulated-object manipulation, and each is evaluated over 20 trials with randomized object positions.

Figure 5. Training data. A robot demonstration with action labels (left) and a human video demonstration without action labels (right) of the same task.

Figure 6. Real-world tasks: stack-bowl, open-oven, pour-bowl, place-food, and place-food-cabinet.

Comparison with latent action baselines

We compare against latent action models pre-trained on large-scale cross-embodiment data (LAPA, Villa-X, UniVLA), an in-domain variant (LAPA-DINO), and plain Diffusion Policy (DP).

**Table 1.** IDM size and training budget.
Method	IDM params	Training trajectories
LAPA	206M	1M+
Villa-X	124M	1M+
UniVLA	34M^†	1M+
LAPA-DINO	102M^†	250
ELA_DP (ours)	6M	250

^† DINO encoder parameters excluded. The IDM is used only during training. At test time, ELA adds just the 4M-parameter Action Decoder.

Bar chart of real-world success rate: UniVLA 0.08, LAPA 0.26, Villa-X 0.27, LAPA-DINO 0.33, DP 0.48, ELA-DP 0.70. — **Figure 7.** Real-world success rate, averaged over tasks.

ELA_DP reaches 0.70 success, against 0.48 for plain DP, the strongest baseline. The large pre-trained models do not turn their capacity into better downstream control, and LAPA-DINO trails both ELA_DP and DP despite in-domain training. In the few-shot setting, the bottleneck is not data scale or diversity but how the latent action space is learned. Figure 2 shows why: the baselines' latents are hard to decode and hard to predict.

Simulation: few-shot learning on LIBERO-10

LIBERO-10 is the most challenging subset of LIBERO: 10 long-horizon tabletop manipulation tasks. Each task has 50 robot demonstrations, of which only N ∈ {5, 10, 20} are action-labeled and the rest are action-free. Results are averaged over 3 seeds. We apply ELA to two representative backbones, Diffusion Policy (DP) and Unified World Model (UWM), giving ELA_DP and ELA_UWM. We compare them with the video-based methods ATM and AMPLIFY, which require external point-tracking models. ELA needs no external models or preprocessing.

Grouped bar chart of LIBERO-10 success rates for N = 5, 10, and 20 labeled demonstrations, comparing ATM, AMPLIFY, DP, ELA-DP, UWM, and ELA-UWM, with dashed lines for DP and UWM trained on all 50 labeled demonstrations. — **Figure 8.** Few-shot policy learning on LIBERO-10. Success rate with N action-labeled demonstrations per task. Dashed lines show DP and UWM trained on all 50 action-labeled demonstrations (upper bound).

ELA consistently improves both backbones by supervising them with latent actions inferred from all demonstrations. With only 5 labeled demonstrations, ELA raises UWM from 19% to 59% and DP from 31% to 54%. In the low-data regime (N = 5, 10), both ELA variants outperform ATM and AMPLIFY. UWM beats DP only when all 50 demonstrations are labeled. With scarce labels it falls behind, and ELA recovers its performance.

Limitations

Scale. ELA has not yet been validated at large scale. Scaling to cross-embodiment datasets and more diverse scenarios is future work.
RGB-only input. ELA operates on raw RGB pixels and does not use depth or 3D priors. Monocular depth models or 3D supervision could help.
Task scope. Our evaluation excludes highly dexterous and precision-intensive manipulation.

Citation

@inproceedings{hsu2026ela,
  title     = {ELA: End-to-End Latent Action Policy Learning from Video Demonstrations},
  author    = {Hsu, Cheng-Chun and Stone, Peter},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}