VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments

**Figure 1. Evolution of navigation instruction generation paradigms.** (a)–(c) Prior speakers rely on discrete viewpoint graphs with panoramic images, optionally augmented by external knowledge or 3D representations. (d) **VTInstructor (Ours)** generates instructions directly from ego-centric RGB video in continuous environments, using *Visual Trajectory Prompts* for spatial grounding without any navigation graph or 3D reconstruction.

Abstract

Navigation instruction generation from ego-centric RGB video in continuous environments is an important yet challenging task for human–robot interaction and scalable dataset construction. Prior instruction generators assume discrete viewpoint graphs with panoramic observations, where trajectory structure is explicit; in continuous environments, however, the agent receives only a dense RGB stream, making trajectory cues difficult to recover. We propose VTInstructor, the first VLN instruction generation framework for continuous environments. Our key idea is to convert implicit trajectory geometry into explicit visual trajectory prompts: EDTC condenses long RGB trajectories into navigation-critical keyframes, VTP overlays path, turn, and goal cues onto these anchors, VTMod injects the resulting trajectory signals into the visual encoder, and VT-GRPO further calibrates this spatial injection during training — all without requiring a navigation graph, pre-built map, or scene reconstruction. On the challenging R2R-CE and RxR-CE Val Unseen benchmarks, VTInstructor sets a new state of the art across all standard NLG metrics, surpassing the strongest baseline by +0.357 and +0.109 CIDEr, respectively. Beyond automatic metrics, VTInstructor-generated instructions raise a frozen follower's success rate to 63.3%, a +14.7 percentage-point gain over the best competing instruction source.

Contributions

The first navigation instruction generator for continuous environments. VTInstructor produces instructions directly from ego-centric RGB video, without any navigation graph, pre-built map, or scene reconstruction.
Visual Trajectory Prompting. We turn implicit trajectory geometry into explicit visual cues — a path ribbon, rotation arrow, and endpoint marker rendered back onto keyframes and encoded as a 3-channel binary mask — and inject them into the visual encoder through the lightweight VTMod, so spatial structure is available where the model actually needs it.
State of the art with real utility. VTInstructor sets a new SoTA on R2R-CE and RxR-CE Val Unseen across all NLG metrics, and its instructions raise a frozen follower's success rate to 63.3% — surpassing human annotations (61.6%).

Method Overview

Overview of the VTInstructor framework — **Figure 2. Overview of VTInstructor.** Given a raw ego-centric RGB trajectory, event-driven compression produces a compact keyframe set. Each keyframe is annotated with a Visual Trajectory Prompt (path ribbon / rotation arrow / endpoint marker), encoded as a 3-channel binary mask, and fed to the VTMod-augmented Qwen3-VL backbone. The model is trained in two stages: SFT, followed by VT-GRPO.

1

EDTC — Event-Driven Trajectory Compression

A three-level pipeline condenses a long RGB trajectory into navigation-critical keyframes (\(n \le 32\)): action run-length encoding, small-step combo grouping, and dynamic priority-based selection using a joint visual–geometric score.

2

VTP — Visual Trajectory Prompt

Each keyframe is overlaid with a path ribbon, rotation arrow and endpoint marker, stored as a 3-channel binary mask \(M \in \{0,1\}^{3\times H\times W}\). GPT-based quality filtering keeps only spatially reliable instruction–keyframe pairs.

3

VTMod — Visual Trajectory Modulator

A lightweight encoder maps the mask to patch-aligned features injected into the ViT at layer 7 via per-token spatial modulation \(\mathbf{h}'_i = \mathbf{h}_i + \mathbf{g}\odot \mathrm{LN}(\mathbf{W}\mathbf{v}_i)\), where the gate \(\mathbf{g}\) starts near zero for lazy activation.

4

VT-GRPO — Reward Calibration

Reinforcement learning refines the gate and LLM using a weighted NLG reward over \(G{=}8\) sampled responses, plus a gate contrastive loss on the best/worst completions to sharpen which trajectory channels are amplified.

See the Visual Trajectory Prompt

The Visual Trajectory Prompt renders the route back onto the ego-centric view. Watch how the same trajectory becomes far easier to read once the path ribbon, turn arrows and goal marker are overlaid.

Generated instruction: Exit the bedroom and turn right. Walk past the kitchen and turn right. Wait at the entrance to the hallway, close to the sculpture.

Generated instruction: Walk through the kitchen and turn right. Walk past the dining room table and chairs and into the living room. Turn right and walk into the pool table room. Stop in front of the pool table.

Generated instruction: Exit the room by the smaller of the two doorways. Turn right and go down the stairs. At the landing, turn right, and go half way down the second flight of stairs. Stop there.

Three R2R-CE trajectories, keyframe by keyframe. Left: raw ego-centric RGB. Right: the same frames with the Visual Trajectory Prompt — direction, turns and goal progress that were implicit in the pixels are now explicit.

Drag the bar to compare the Visual Trajectory Prompt on / off for the current keyframe, and click Next keyframe to move one step along the route.

↔

+ VTP no VTP

Keyframe 1 / 22

Generated instruction: Walk out of the living room and into the kitchen. Turn left and walk into the bathroom. Stop in the doorway to the bathroom.

Main Results

Navigation instruction generation on R2R-CE and RxR-CE Val Unseen. VTInstructor sets a new state of the art across all standard NLG metrics.

Table 2: cross-setting comparison with prior instruction generation methods

Qualitative Comparison

Compressed keyframes with VTP overlays (top) and instructions from the ground truth, the strongest baseline, and VTInstructor. Yellow marks factual errors (wrong turns, hallucinated landmarks).

Practical Utility

Beyond NLG metrics, VTInstructor-generated instructions improve real navigation: they beat human annotations as a frozen follower's guidance, augment training data, and win human preference on real-world videos. Adding VTInstructor-generated instructions to the training set of a frozen CorrectNav follower (LLaVA-Video-7B backbone) improves navigation on both benchmarks — a consistent gain on top of human data alone.

Table 5: downstream navigation performance with a frozen CorrectNav follower

Table 8: human evaluation on real-world navigation videos

Table 6: augmentation on R2R-CE Val Unseen

Table 7: augmentation on RxR-CE Val Unseen

BibTeX

@misc{yang2026vtinstructorvisualtrajectoryprompting,
      title={VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments},
      author={Haolin Yang and Yuxing Long and Zihan Yang and Hao Dong},
      year={2026},
      eprint={2608.15284},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2608.15284},
}