#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs.
# Dataset Card for CropAndWeed
This is a [FiftyOne](https://github.com/voxel51/fiftyone) dataset with 8,034 samples.
## Installation
If you haven’t already, install FiftyOne:
```bash
pip install -U fiftyone
```
## Usage
```python
import fiftyone as fo
import fiftyone.utils.huggingface as fouh
dataset = fouh.load_from_hub("Voxel51/CropAndWeed")
session = fo.launch_app(dataset)
```
## Dataset Details
### Dataset Description
The CropAndWeed dataset is a large-scale, high-variability image dataset for Precision
Agriculture, targeting the fine-grained identification of crop and weed species to support
automated, plant-specific weed intervention. It provides bounding boxes, stem positions,
and semantic segmentation masks for about 112k plant instances across more than 8k
high-resolution real-world images, along with rich per-image meta-annotations describing
recording conditions.
Images come from two complementary sources: an **Application Set** of images captured at
several hundred commercially used agricultural sites in Austria (realistic, uncontrolled
field conditions), and an **Experimental Set** captured on dedicated experimental plots
specifically cultivated to enrich the dataset with otherwise underrepresented, rare weed
species.
- **Curated by:** Daniel Steininger, Andreas Trondl, Gerardus Croonen, Julia Simon, and
Verena Widhalm, AIT Austrian Institute of Technology
- **Funded by:** Austrian Federal Ministry for Climate Action, Environment, Energy,
Mobility, Innovation and Technology, and the Austrian Research Promotion Agency (FFG),
under the “ICT of the Future” research project HARV.EST (FFG No. 892347)
- **Shared by:** AIT Austrian Institute of Technology (original release); this FiftyOne
format conversion was prepared by Harpreet Sahota
- **Language(s):** N/A (imagery, no text)
- **License:** Custom non-commercial research license (see [Dataset Sources](#dataset-sources) and [More Information](#more-information))
### Dataset Sources
- **Repository:** https://github.com/cropandweed/cropandweed-dataset
- **Paper:** [The CropAndWeed Dataset: A Multi-Modal Learning Approach for Efficient Crop and Weed Manipulation](https://openaccess.thecvf.com/content/WACV2023/papers/Steininger_The_CropAndWeed_Dataset_A_Multi-Modal_Learning_Approach_for_Efficient_Crop_WACV_2023_paper.pdf) (WACV 2023)
## Uses
### Direct Use
- Training and benchmarking object detection, fine-grained plant classification, semantic/
panoptic segmentation, and stem/anchor-point (keypoint) localization models for
Precision Agriculture and automated crop/weed manipulation.
- Academic research, teaching, and personal experimentation into robust, high-variability
agricultural computer vision, including studying how incorporating multiple weed species
into training affects crop-detection accuracy.
- Analysis of model robustness to environmental conditions using the per-image moisture,
soil, lighting, and separability meta-annotations.
### Out-of-Scope Use
- Any commercial use, including licensing, selling, or otherwise using the data (or the
raw data itself in redistributed form) for commercial gain — explicitly disallowed by
the dataset license.
- Redistributing the dataset or modified versions of it. Derivative works that are
abstract representations not allowing recovery of the original data (e.g. trained
model weights) are permitted; redistributing the images/annotations themselves is not.
- Use as a certified or standalone agronomic/regulatory plant-identification tool without
expert validation — annotations were produced for computer-vision benchmarking purposes.
## Dataset Structure
This is a flat (non-grouped) image dataset with `media_type="image"` and 8,034 samples,
one sample per annotated image (1920x1088 RGB JPEG). It was parsed from the official
raw annotation release (`bboxes/`, `labelIds/CropAndWeed/`, `params/`), preserving every
modality shipped by the dataset authors.
| Field | FiftyOne type | Description |
|------------------------------------|----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `filepath` | `StringField` | Absolute path to the source RGB image |
| `metadata` | `ImageMetadata` | Standard image metadata (`width`, `height`, `size_bytes`, `mime_type`, `num_channels`) |
| `recording_set` | `StringField` | `"Application Set"` or `"Experimental Set"`, parsed from the `ave-`/`vwg-` filename prefix |
| `session` | `StringField` | 4-digit recording-session identifier (verbatim from filename) |
| `image_id` | `StringField` | 4-digit image identifier within the session (verbatim from filename) |
| `ground_truth` | `Detections` | One `Detection` per annotated plant instance (2D bounding box), `label` is one of 100 fine-grained species/growth-stage classes (or the `Vegetation` fallback class) |
| `stem_points` | `Keypoints` | One `Keypoint` per annotated instance marking its stem (root) position |
| `segmentation` | `Segmentation` | Per-pixel semantic mask referencing a single-channel PNG (`mask_path`), same 100 classes as `ground_truth` |
| `moisture` / `moisture_id` | `StringField` / `IntField` | Soil moisture at capture time: `dry`, `medium`, or `wet` (raw id 0-2) |
| `soil` / `soil_id` | `StringField` / `IntField` | Soil granularity: `fine`, `medium`, or `coarse` (raw id 0-2) |
| `lighting` / `lighting_id` | `StringField` / `IntField` | Lighting condition: `sunny` or `diffuse` (raw id 0-1) |
| `separability` / `separability_id` | `StringField` / `IntField` | Subjective difficulty of separating plants from soil/each other: `easy`, `medium`, or `hard` (raw id 0-2) |
**Label types and why:**
- **`ground_truth` (`Detections`)** — each row of the source `bboxes/*.csv` files
(`Left, Top, Right, Bottom, LabelID, StemX, StemY`) is converted to a `Detection` with
a relative `[x, y, w, h]` bounding box (required by FiftyOne). Each `Detection` also
carries `label_id` (the raw numeric class id, 0-99 or 255) and `stem_x`/`stem_y`
(the same stem position, kept as absolute pixel coordinates for reference).
- **`stem_points` (`Keypoints`)** — the dataset authors treat stem/anchor-point
localization as a first-class learning task in its own right (Section 5.3 of the
paper), so the stem position accompanying every bounding-box row is *also* materialized
as a normalized-coordinate `Keypoint`, letting it be visualized and queried independently
of `ground_truth`.
- **`segmentation` (`Segmentation`)** — the `labelIds/CropAndWeed/*.png` semantic masks are
referenced via `mask_path` (not embedded as in-database arrays) to avoid duplicating
~8k full-resolution masks in the database. `dataset.mask_targets["segmentation"]` maps
all 101 raw pixel values (0-99 plus 255) to their class names.
- **Primitive fields for `params`** — the four categorical per-image parameters are stored
both as a human-readable string (`moisture`, `soil`, `lighting`, `separability`) and the
raw numeric id shipped in `params/*.csv` (`moisture_id`, etc.), since the string versions
are more convenient for filtering/grouping in the FiftyOne App.
**`dataset.classes`** — `ground_truth` and `stem_points` are both registered with the same
100-class list (99 fine-grained species/growth-stage labels plus the `Vegetation` fallback).
**`dataset.info`** contains the source repository URL, paper citation, and a short license
summary.
**Parsing decisions:**
- Only the base, 100-class `CropAndWeed` label specification was imported. The official
repository also defines simplified variants (`Fine24`, `CropsOrWeed9`, `CropOrWeed2`,
`Maize1`/`Maize2`, `SugarBeet1`/`SugarBeet2`, etc.), which are pure label-ID remappings
of these same raw annotations with no additional information — equivalent groupings can
be reconstructed from `ground_truth.detections.label_id` using FiftyOne label-mapping
views if needed, without re-importing.
- 234 of the 8,034 images have no annotated plant instances (their source bounding-box
CSVs are empty); these samples have no `ground_truth`/`stem_points` but still have a
`segmentation` mask (all background/`Soil`). All 8,034 images have a `segmentation` mask.
- `label_id` `255` (`Vegetation`) is a fallback class for tiny (<16² px), excluded, or
visually ambiguous instances, and appears in both `ground_truth` and `segmentation`.
- No official train/val/test split is distributed with the raw annotation release (the
paper’s 70:15:15 split was generated for the authors’ internal benchmarking only and is
not shipped in the public download), so no split/tag field is set on this dataset.
## Dataset Creation
### Curation Rationale
Precision Agriculture approaches to automated weed intervention require large amounts of
highly variable, annotated training data, which the authors found lacking in prior work
(most existing datasets are small, laboratory-controlled, provide only one annotation
modality, or cover far fewer species). CropAndWeed was curated to close this gap: a
large-scale, high-variability dataset spanning 74 crop and weed species (16 crop, 58 weed),
with multi-modal annotations (bounding boxes, semantic masks, and stem positions) rarely
provided together elsewhere, and an emphasis on realistic field conditions plus specially
cultivated plots for rare species.
### Source Data
#### Data Collection and Processing
Images were captured over a period of four years (March-July) using a semi-professional
full-frame SLR camera with a constant 50mm focal length, in auto-exposure mode, from an
approximate top-down perspective at a height of about 1.1 meters. Each capturing session
corresponds to a unique agricultural site or experimental plot at a specific point in time,
yielding an average of about 20 images per session at intervals of at least three meters
to avoid redundancy.
| | Sessions (recorded / annotated) | Images (recorded / annotated) |
|------------------|-----------------------------------|---------------------------------|
| Experimental Set | 1,363 / 665 | 22,597 / 4,990 |
| Application Set | 738 / 264 | 21,217 / 3,044 |
| **Total** | **2,101 / 929** | **43,814 / 8,034** |
The 8,034 annotated images contain 111,953 annotated plant instances in total. Each
recording session is also enriched with the `moisture`, `soil`, `lighting`, and
`separability` meta-annotations included in this dataset, plus timestamps, GPS coordinates,
and camera parameters recorded by the authors (not part of this FiftyOne conversion).
#### Who are the source data producers?
The AIT Austrian Institute of Technology research team (Daniel Steininger, Andreas Trondl,
Gerardus Croonen, Julia Simon, Verena Widhalm), in cooperation with the experimental farm
Groß-Enzersdorf, produced and recorded the source imagery.
### Annotations
#### Annotation process
To increase annotation throughput, each image was first automatically pre-segmented into
soil vs. vegetation pixels (initially via Otsu color thresholding, later via a CNN trained
on already-annotated data). Annotators refined these masks and assigned the target species
class. Bounding boxes were then auto-generated from the refined mask and manually corrected,
and each plant’s stem position was manually annotated. Image batches with high ambiguity or
a high number of plant instances were reviewed by voting among multiple annotators as an
additional quality-control step.
#### Who are the annotators?
The AIT annotation team, credited in the paper as Gulnar Bakytzhan, Vanessa Klugsberger,
and Marlene Glawischnig, with additional support from Caroline Huber, Reinhard
Neugschwandtner, and Helmut Wagentristl at the experimental farm Groß-Enzersdorf.
#### Personal and Sensitive Information
None. The dataset consists exclusively of top-down agricultural field imagery (soil, crops,
and weeds); it contains no people, faces, or other personal or sensitive information.
## Citation
**BibTeX:**
```bibtex
@InProceedings{Steininger_2023_WACV,
author = {Steininger, Daniel and Trondl, Andreas and Croonen, Gerardus and Simon, Julia and Widhalm, Verena},
title = {The CropAndWeed Dataset: A Multi-Modal Learning Approach for Efficient Crop and Weed Manipulation},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
month = {January},
year = {2023},
pages = {3729-3738}
}
```
**APA:**
Steininger, D., Trondl, A., Croonen, G., Simon, J., & Widhalm, V. (2023). The CropAndWeed
dataset: A multi-modal learning approach for efficient crop and weed manipulation. In
*Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*
(pp. 3729-3738).
## More Information
The dataset is released by AIT Austrian Institute of Technology to academic and
non-academic entities for non-commercial purposes only (academic research, teaching,
scientific publications, or personal experimentation). Redistribution of the dataset or
modified versions of it is not permitted; derivative works that are abstract
representations not allowing recovery of the original data (e.g. trained models) are
permitted. See the [official license](https://github.com/cropandweed/cropandweed-dataset/blob/main/LICENCE)
for the full terms.
The official repository additionally provides scripts to remap the raw 100-class
annotations into 20 predefined simplified dataset variants (e.g. `Fine24`,
`CropsOrWeed9`, `CropOrWeed2`) for applications that don’t require the full fine-grained
species/growth-stage taxonomy.
## Dataset Card Authors
Harpreet Sahota (FiftyOne conversion and dataset card)