#### NOTE This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs. ![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow) # Dataset Card for KITScenes-LongTail Videos ![image/png](https://huggingface.co/datasets/Voxel51/KITScenes-LongTail/resolve/main/kitscenes_longtail.gif) A [FiftyOne](https://github.com/voxel51/fiftyone) **grouped video** dataset built from [KIT-MRT/KITScenes-LongTail](https://huggingface.co/datasets/KIT-MRT/KITScenes-LongTail), a long-tail autonomous-driving benchmark for VLMs/VLAs. Each scenario is a ~9 s, six-camera surround-view event with a high-level driving instruction, multiple candidate future trajectories, and multilingual expert reasoning traces. This is a [FiftyOne](https://github.com/voxel51/fiftyone) dataset with 103 samples. ## Installation If you haven’t already, install FiftyOne: ```bash pip install -U fiftyone ``` ## Usage ```python import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub from huggingface_hub import snapshot_download # Download the dataset snapshot to the current working directory snapshot_download( repo_id="Voxel51/KITScenes-LongTail", local_dir=".", repo_type="dataset" ) # Load dataset from current directory using FiftyOne's native format dataset = fo.Dataset.from_dir( dataset_dir=".", # Current directory contains the dataset files dataset_type=fo.types.FiftyOneDataset, # Specify FiftyOne dataset format name="KITScenes-LongTail" # Assign a name to the dataset for identification ) # Launch the App session = fo.launch_app(dataset) ``` ## Dataset Details ### Dataset Description KITScenes-LongTail targets generalization to rare (“long-tail”) driving events for end-to-end driving. It provides synchronized multi-view video, ego trajectories, high-level instructions, and detailed multilingual reasoning traces, supporting in-context learning and few-shot generalization for multimodal models (VLMs and VLAs). Beyond safety/comfort metrics, it is designed to evaluate instruction following and the semantic coherence between a model’s reasoning and its predicted trajectory. The reasoning traces are authored by domain experts (self-driving researchers) with diverse cultural backgrounds in English, Spanish, and Chinese. This FiftyOne version models each scenario as one **group** with six camera video slices, a composite surround slice, and a 3D point-cloud slice reconstructed from the front camera. The observed clip (~4 s) is real footage; where labels exist (train split), the 5 s prediction horizon is represented and the expert reasoning’s action windows are stored as temporal detections (see **Dataset Structure** for the exact, faithful breakdown of what is verbatim vs. derived vs. synthetic). It also ships with ready-made saved views and Qwen3-VL clip embeddings (with similarity, 2D visualization, uniqueness, and representativeness indexes) for exploration. - **Curated by:** Institute of Measurement and Control Systems (MRT), Karlsruhe Institute of Technology (KIT), with collaborators from FZI Research Center for Information Technology, University Charles III of Madrid, Technical University of Madrid, University of Toronto, and Delft University of Technology. - **Funded by:** German Federal Ministry for Economic Affairs and Energy, project “NXT GEN AI METHODS”; compute provided by NHR@KIT (HoreKa). - **Shared by:** Original dataset by KIT-MRT; FiftyOne conversion published at `harpreetsahota/KITScenes-LongTail`. - **Language(s):** English, Spanish, Chinese (reasoning traces). - **License:** CC BY-NC 4.0, with additional dataset terms (non-commercial; cite the publication; anonymized faces/plates; data provided “as is”). Where the dataset terms conflict with CC BY-NC 4.0, the dataset terms prevail. ### Dataset Sources - **Repository:** https://huggingface.co/datasets/KIT-MRT/KITScenes-LongTail (original) - **Paper:** Wagner et al., “LongTail Driving Scenarios with Reasoning Traces: The KITScenes LongTail Dataset,” arXiv:2603.23607 (2026) ## Uses ### Direct Use - Benchmarking VLMs/VLAs for end-to-end driving in long-tail scenarios. - Trajectory prediction: forecasting a 5 s future trajectory (25 waypoints @ 5 Hz) from the past 4 s, the surround video, and a high-level instruction. - Instruction-following evaluation with fine-grained commands (e.g. “overtake truck driving on the right”). - Studying semantic coherence between reasoning traces and predicted trajectories. - Zero-shot, few-shot, and few-shot chain-of-thought (CoT) prompting; multilingual reasoning research across English/Spanish/Chinese. - Multi-camera (360°) video understanding and qualitative review via the synchronized `surround` montage slice. - Dataset exploration with the bundled Qwen3-VL clip embeddings: similarity / nearest-neighbour search, 2D embedding plots, and uniqueness / representativeness ranking to surface the rarest or most prototypical scenarios. - Inspecting per-frame monocular **depth maps** and the merged front-camera **3D point cloud** (the `threed` slice) in the FiftyOne 3D viewer. ### Out-of-Scope Use - Commercial use (the license is non-commercial). - Perception tasks needing ground-truth object annotations (2D/3D boxes, masks, tracks, depth, HD maps): none are provided. The `threed` point cloud and per-frame depth maps are **model-estimated** (VGGT-Omega), not sensor-measured ground truth, so they should not be used as depth/geometry labels. - Photometric or geometric tasks on the **synthetic future-horizon frames** of train videos (these freeze the last observed frame and are not real imagery). - Treating the held-out `test` split as labeled: its future trajectories and reasoning are withheld. ## Dataset Structure **Topology.** `media_type = group`. **103 groups** (3 `train` + 100 `test`), each with **8 group slices** (seven video, one 3D): - `front_left`, `front`, `front_right`, `rear_left`, `rear`, `rear_right` — the six surround cameras (one H.264 mp4 each). - `surround` — a derived 2×3 montage of the six cameras playing in sync (top row: front-left/front/front-right; bottom row: rear-left/rear/rear-right). - `threed` — a 3D point-cloud scene (`.fo3d`, `media_type = 3d`) reconstructed from the `front` clip, viewable in the FiftyOne 3D viewer (see **3D reconstruction** below). Each sample is also tagged with its split (`train` / `test`), and a `split` string field is stored for convenience. **Timeline.** Videos are 5 Hz. Frames `1..n_observed` (n_observed = 21, ~4 s) are the real observed clip. On `train`, frames after `n_observed` **freeze the last observed frame** to represent the 5 s prediction horizon, so the reasoning action windows can be stored as `TemporalDetections` with valid frame supports (train clips are 46 frames; test clips are 21 frames, observation-only). **Sample fields** | Field | FiftyOne type | Description | |----------------------------------------------------------------------------------------|----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| | `scenario_id` | `StringField` | Scenario identifier, e.g. `"0000400"` (verbatim) | | `split` | `StringField` | `"train"` or `"test"` (also a sample tag) | | `camera` | `StringField` | Slice name for this sample (a camera, or `surround`) | | `fps` | `IntField` | Clip frame rate (5 Hz; from the paper) | | `n_observed_frames` | `IntField` | Number of real observed frames (21) | | `n_future_frames` | `IntField` | Synthetic future-horizon frames (25 on train, 0 on test) | | `now_frame` | `IntField` | 1-indexed frame marking “now” (the last observed frame) | | `driving_instruction` | `Classification` | High-level maneuver instruction (verbatim) | | `scenario_type` | `Classification` | Long-tail scenario category (verbatim) | | `trajectory_bev` | `DictField` | Ego BEV waypoints `[x, y]` (x forward, y left, meters): `past` (21 pts) plus populated future variants (verbatim values; `[[-100,-100]]` placeholders dropped) | | `reasoning_english` / `reasoning_spanish` / `reasoning_chinese` | `DictField` | The 9 expert reasoning fields per language, verbatim (train only) | | `driving_actions_english` / `_spanish` / `_chinese` | `TemporalDetections` | Acceleration/steering action windows (`0–3 s`, `3–5 s`); labels/reasons verbatim, frame supports derived (train only) | | `qa_english_reconstructed` | `ListField` | Reconstructed 5 Q&A pairs (paper’s English question templates + verbatim answers) (train only) | | `reasoning_cot_english_reconstructed` | `StringField` | Reconstructed chain-of-thought block (train only) | | `scene_3d` | `StringField` | Path to the front-camera `.fo3d` 3D scene (also materialized as the `threed` slice) | | `qwen_emb_surround` / `qwen_emb_front` / `qwen_emb_rear` | `VectorField` | 2048-d Qwen3-VL clip embedding (one per slice; on the `surround` / `front` / `rear` slices respectively) | | `uniqueness_surround` / `uniqueness_front` / `uniqueness_rear` | `FloatField` | Per-clip uniqueness score in `[0, 1]` (higher = more unusual within its slice) | | `representativeness_surround` / `representativeness_front` / `representativeness_rear` | `FloatField` | Per-clip representativeness score in `[0, 1]` (higher = more prototypical within its slice) | **Frame fields** (attached to the six camera slices only; not the `surround` montage) | Field | FiftyOne type | Description | |--------------------|-----------------|------------------------------------------------------------------------------------------------------------------| | `past_trail` | `Polylines` | History breadcrumb; grows over observed frames (per-frame ego-pose transform), then persists through the horizon | | `traj_expert_like` | `Polylines` | Expert (GT) future plan, progressively built out across the horizon | | `traj_wrong_speed` | `Polylines` | Adversarial future (wrong speed), built out across the horizon | | `traj_off_road` | `Polylines` | Adversarial future (off road), built out across the horizon | | `traj_crash` | `Polylines` | Adversarial future (crash), where present | | `plan_cursor` | `Keypoints` | Single marker advancing along the expert plan | Trajectories are rendered as `Polylines` (connected paths) rather than `Keypoints` because a driving trajectory is an ordered, connected curve; the advancing `plan_cursor` remains a single `Keypoint`. The **`front`** slice additionally carries per-frame outputs from the 3D reconstruction (sampled at 5 Hz): | Field | FiftyOne type | Description | |---------------------------|-----------------|----------------------------------------------------------------------| | `depth_map` | `Heatmap` | Monocular depth map for the frame, rendered as an overlay in the App | | `camera_translation` | `ListField` | Camera position (3-vector) in the scene’s OpenCV world frame | | `camera_rotation_matrix` | `ListField` | Camera rotation as a 3×3 matrix | | `camera_quaternion` | `ListField` | Camera rotation as a quaternion `[qw, qx, qy, qz]` | | `intrinsic_matrix` | `ListField` | Recovered 3×3 pinhole intrinsics for the frame | | `fov_h_deg` / `fov_w_deg` | `FloatField` | Estimated horizontal / vertical field of view, in degrees | **`dataset.info`** - `camera_parameters` — per-camera pinhole `K` and extrinsics `R`, `t` (ego→camera optical frame), copied verbatim from the dataset card; the rig is fixed so they apply to every sample. - `camera_parameters_note` — explanation of the calibration convention. - `timeline_note` — describes the 5 Hz timeline and the synthetic freeze-frame horizon. - `provenance` — explicit classification of every field as `verbatim_from_dataset`, `derived`, or `constructed_or_synthetic`, plus assumptions (`ground_z = -1.6 m`, `fps = 5 Hz`, synthetic horizon). **Parsing decisions** - **Verbatim:** `scenario_id`, `driving_instruction`, `scenario_type`, the per-language `reasoning_*` dicts, `trajectory_bev` waypoint values, and the real observed frames. - **Derived:** trajectory overlays are projected with the card’s camera matrices plus a road-plane estimate `ground_z = -1.6 m` (and estimated per-frame headings for the past trail); `fps`/timing are taken from the paper (the parquet has no timestamps); temporal-detection frame supports map the paper’s `0–3 s`/`3–5 s` windows onto the horizon. - **Constructed/synthetic:** the train future-horizon frames (a freeze of the last real frame); `qa_english_reconstructed` / `reasoning_cot_english_reconstructed` (assembled from paper question templates + verbatim answers). Spanish/Chinese Q&A are intentionally **not** fabricated. - **Held-out:** on `test`, future trajectories and all reasoning are withheld (empty), so those samples are inputs-only (video + instruction + scenario_type + past trajectory), have no future horizon, and no temporal detections. - Scenario-level labels are attached to every slice (including `surround`); per-pixel overlays are attached only to the camera slices (their coordinates are per-view, not montage-space). **Saved views** The dataset ships with these saved views (available from the view dropdown in the App or via `dataset.load_saved_view(...)`): | Saved view | Description | |-------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------| | `slice__front_left`, `slice__front`, `slice__front_right`, `slice__rear_left`, `slice__rear`, `slice__rear_right` | One per camera — flattens the group to a single camera so you can scan all 103 clips from that viewpoint | | `by_scenario_type_and_instruction` | The `surround` clips dynamically grouped by `scenario_type`, ordered by `driving_instruction` within each group | **Embeddings, similarity & quality scores** Each of the `surround`, `front`, and `rear` slices carries a **2048-d Qwen3-VL clip embedding** (`qwen_emb_`) summarizing the whole video clip, along with FiftyOne Brain indexes computed from those embeddings: | Brain key | Type | Use | |--------------------------------------------------------------------------------------|----------------------|-----------------------------------------------------------------------| | `sim_surround`, `sim_front`, `sim_rear` | similarity | Nearest-neighbour / sort-by-similarity search over clips | | `viz_surround`, `viz_front`, `viz_rear` | visualization (UMAP) | 2D embedding scatter plot for interactive exploration | | `uniqueness_surround`, `uniqueness_front`, `uniqueness_rear` | uniqueness | Per-clip `uniqueness_` score (find rare / outlier clips) | | `representativeness_surround`, `representativeness_front`, `representativeness_rear` | representativeness | Per-clip `representativeness_` score (find prototypical clips) | **3D reconstruction** The `threed` slice contains one `.fo3d` scene per scenario, reconstructed from the `front` clip with VGGT-Omega. Each scene holds a merged, colored point cloud plus camera frustums for the sampled frames and opens in the FiftyOne 3D viewer. The matching per-frame depth maps and camera poses live on the `front` slice’s frame fields (see the frame-fields table above). The point cloud and depth are model-estimated (not sensor LiDAR), and are derived from the real observed frames. ## Dataset Creation ### Curation Rationale Generalization in perception has advanced, but decision-making in long-tail scenarios remains a major challenge. The dataset couples driving with high-level instructions and multilingual, step-by-step reasoning traces to accelerate progress on decision-making in rare events, and to enable studying how reasoning style and language affect driving competence. Multiple plausible maneuvers are evaluated rather than a single expert trajectory (the paper introduces a multi-maneuver score, MMS). ### Source Data #### Data Collection and Processing Recorded over two years beginning in late 2023 across urban, suburban, and highway environments (main locations: Karlsruhe, Heidelberg, Mannheim, and the Black Forest). Routes were adjusted to include construction zones and intersections, and filtered for rare events such as adverse weather (heavy rain, snow, fog), road closures, and accidents. Long-tail data was further selected using the Pareto principle with nuScenes as a reference distribution (80% cumulative-frequency threshold on rank-frequency plots). The full collection is 1,000 nine-second scenarios split train (500) / test (400) / validation (100); the released subset used here includes the test split and a small set of train samples. Each scenario provides synchronized six-view video over a 360° horizontal field of view at 5 Hz, ~4 s observed, in raw, pinhole, and stitched formats (this FiftyOne build uses the pinhole frames, downscaled for encoding). The future prediction horizon is 5 s (25 waypoints at 5 Hz). #### Who are the source data producers? The KIT-MRT team and collaborators, recording with a research vehicle sensor suite. ### Annotations #### Annotation process High-level driving instructions were manually annotated by domain experts. Reasoning traces were collected by asking experts five questions per scenario: one open-ended situational-awareness question grounded in the instruction, and four questions about the acceleration and steering decisions for the next 0–3 s and the final 3–5 s of the expert trajectory. Action labels follow heuristics (acceleration: slight/strong accelerate, decelerate, maintain; steering: slight/sharp left/right, straight). Experts answered in their mother tongue (or a fluent language); verbal responses were transcribed with Whisper. For evaluation, reference trajectories are provided per scenario across categories (expert-like, wrong speed, neglect instruction, off road, crash); expert trajectories are driven, wrong-speed variants are augmented via state estimation and spline modification, and the remaining categories are manually labeled. #### Who are the annotators? Domain experts (researchers working on self-driving) with diverse linguistic and cultural backgrounds (English, Spanish, Chinese). #### Personal and Sensitive Information Faces and license plates are anonymized using state-of-the-art anonymization software (BrighterAI). Requests for removal of specific frames can be directed to info@mrt.kit.edu. ## Citation **BibTeX:** ```bibtex @misc{wagner2026longtaildrivingscenariosreasoning, title={LongTail Driving Scenarios with Reasoning Traces: The KITScenes LongTail Dataset}, author={Royden Wagner and Omer Sahin Tas and Jaime Villa and Felix Hauser and Yinzhe Shen and Marlon Steiner and Dominik Strutz and Carlos Fernandez and Christian Kinzig and Guillermo S. Guitierrez-Cabello and Hendrik Königshof and Fabian Immel and Richard Schwarzkopf and Nils Alexander Rack and Kevin Rösch and Kaiwen Wang and Jan-Hendrik Pauls and Martin Lauer and Igor Gilitschenski and Holger Caesar and Christoph Stiller}, year={2026}, eprint={2603.23607}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2603.23607}, } ``` **APA:** Wagner, R., Taş, Ö. Ş., Villa, J., Hauser, F., Shen, Y., Steiner, M., Strutz, D., Fernandez, C., Kinzig, C., Guitierrez-Cabello, G. S., Königshof, H., Immel, F., Schwarzkopf, R., Rack, N. A., Rösch, K., Wang, K., Pauls, J.-H., Lauer, M., Gilitschenski, I., Caesar, H., & Stiller, C. (2026). *LongTail Driving Scenarios with Reasoning Traces: The KITScenes LongTail Dataset*. arXiv:2603.23607. ## More Information The released subset contains the `test` split and a small number of `train` samples (few-shot examples); the full `train` and `validation` splits, along with stitched 360° images, are announced for a later dataset version. ## Dataset Card Contact For the underlying dataset: info@mrt.kit.edu