Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Hugging Face

Dataset Card for AgroMind#

image/png

This is a FiftyOne dataset with 21,339 samples.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

# Load the dataset
# Note: other available arguments include 'max_samples', 'persistent', 'overwrite' etc
dataset = fouh.load_from_hub("Voxel51/agromind")

# Launch the App
session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

AgroMind is a comprehensive agricultural remote sensing visual-question-answering (VQA) benchmark covering four task dimensions — Spatial Perception, Object Understanding, Scene Understanding, and Scene Reasoning — spanning 13 task types, from crop identification and health monitoring to environmental and climate analysis. It integrates nine public remote sensing / agricultural computer vision datasets and one private global farmland-parcel dataset into a single “Sky–Air–Ground” (satellite, UAV, and ground-camera) benchmark, comprising 20,850 images and 28,482 question-answer pairs spanning 106 geographic regions.

This FiftyOne dataset re-packages the original AgroMind release (images + raw per-task QA JSON files) into a single, unified, explorable collection: one sample per unique image, with every associated question-answer pair preserved and, where a task has a well-defined categorical/numeric/spatial ground truth, promoted into a typed FiftyOne field (Classification, Classifications, Detection, Keypoint, int, or float) for direct filtering and visualization in the App.

  • Curated by: Qingmei Li, Yang Zhang, Zurong Mai, Yuhang Chen, Shuohong Lou, Henglian Huang, Jiarui Zhang, Zhiwei Zhang, Yibin Wen, Weijia Li, Haohuan Fu, Jianxi Huang, Juepeng Zheng (Tsinghua University, Sun Yat-Sen University, China Agricultural University, Southwest Jiaotong University, National Supercomputing Center in Shenzhen)

  • Funded by: [More Information Needed]

  • Shared by: AgroMind authors (original release); re-packaged into FiftyOne format for this card

  • Language(s): English (all questions, options, and answers)

  • License: CC BY-SA 4.0

Dataset Sources#

Uses#

Direct Use#

  • Zero-shot / few-shot benchmarking of large multimodal models (LMMs) on agricultural remote sensing understanding, across spatial perception, object understanding, scene understanding, and scene reasoning tasks. The original paper evaluated 24 LMMs (4 closed-source, 20 open-source) this way.

  • Supervised fine-tuning of vision-language models on agricultural VQA: every question-answer pair carries a rendered prompt (instruction text, with options inlined for choice-based questions) and a resolved answer_text (human-readable ground truth), specifically added in this FiftyOne version to make constructing a training set (e.g. a PyTorch Dataset) straightforward without re-deriving option/letter mappings.

  • Exploratory data analysis and curation of agricultural remote sensing imagery (crop/pest/disease classification, tree cover and climate zone analysis, cultivated-parcel boundary detection, anomaly detection in UAV farmland tiles, tassel/fruit/oil-palm counting) using FiftyOne’s filtering, tagging, and visualization tools.

Out-of-Scope Use#

  • Operational or safety-critical agronomic decision-making (e.g. real-world pesticide application, yield forecasting for financial decisions) — this is a research benchmark, and the paper reports that even top LMMs and human volunteers achieve well under 50% overall accuracy.

  • Redistribution of the “Global Parcel Dataset” component (mapped to the Rural sub-dataset here) beyond research use — this component is proprietary to the original authors, not one of the nine public source datasets.

  • Any use requiring ground-truth pixel-level segmentation masks — the original source datasets often have richer annotations (masks, polygons) than are exposed through AgroMind’s QA-pair format; this FiftyOne dataset reflects only what AgroMind’s released QA pairs make available.

Dataset Structure#

This is a flat (non-grouped, non-video) image dataset: media_type = "image", 21,339 samples, not split into train/val/test (AgroMind is released as a single evaluation set).

Of the 21,339 images, 19,363 are referenced by at least one of the benchmark’s 28,482 QA pairs; the remaining ~1,976 are source images present in the raw archives but not sampled into any question (tagged via the qa/multi_image_refs fields both being unset — see the “Background Images (No QA)” saved view below).

Each sample’s source_dataset field indicates which of the 10 original sub-datasets/folders it came from (Agriculture, Fruit, Leaf_diseases, Oil_palm_trees, Pest, Rural, Trees, corn, crop, CropHarvest) — see Source Data below for how these map to the paper’s named datasets.

Raw QA pairs (qa field)#

Roughly 85% of AgroMind’s questions (24,375 of 28,482) are about a single image. All of these are preserved verbatim as a list of embedded dicts in each sample’s qa field — nothing is dropped, even for questions whose answer wasn’t promoted to a typed field. Each entry has:

Key

Type

Description

task

str

2-4 letter AgroMind task code (SL, SR, BD, OC, PDD, GSR, SC, CO, AS, VPR, AR, CTR, PL)

type_id

int

Original AgroMind question-format id (1-11: MCQ, yes/no, open numeric, image-choice, free text, etc.)

level1_id / level2_id / level3_id

int

Original 3-level task-hierarchy ids (dimension / subtype / specific task)

item_id

int

Original per-file question id

question

str

Raw question text

options

dict or null

Raw lettered options dict, when applicable

answer

str/list/float

Raw ground-truth answer, exactly as released (a letter, list of letters, number, or coordinate list)

prompt

str

Added in this FiftyOne version. The question with options rendered inline (e.g. "...\nA. Top-Left\nB. Bottom-Left\n...") — ready to use as a model instruction without further formatting

answer_text

str

Added in this FiftyOne version. The resolved, human-readable ground truth (e.g. an MCQ letter resolved to its option text)

The remaining ~15% of questions (4,107 of 28,482) are inherently multi-image — either a 4-way “which image…” selection question, or a direct 2-image comparison. Since FiftyOne samples are single-image, these live in dataset.info["multi_image_qa"] (a list of dicts, same schema as above plus image_paths and option_labels, resolving each lettered option to an actual image path), keyed by a stable qid. Every image that participates in one of these questions carries a lightweight backreference in its multi_image_refs field ({"task", "qid", "role"}), so participants are still filterable/discoverable directly from the App without duplicating the full question payload onto every image.

Saved views#

The dataset ships with 17 saved views for common exploration/filtering needs:

View

Samples

Description

Spatial Perception Tasks

2,793

Images with an SL/SR/BD QA pair

Object Understanding Tasks

3,336

Images with an OC/PDD/GSR QA pair

Scene Understanding Tasks

6,862

Images with an SC/CO/AS QA pair

Scene Reasoning Tasks

5,239

Images with a VPR/AR/CTR/PL QA pair

Diseased Leaves

1,285

disease set and not "healthy"

Healthy Leaves

226

disease == "healthy"

Cultivated Land Boundaries

400

Has a cultivated_area Detection

Multi-Anomaly Farmland Tiles

84

2+ co-occurring anomaly labels

Anomaly Detections

748

Has an anomalies label (any)

Severe Crop Occlusion

234

crop_occlusion == “Severe occlusion”

High Yield-Reduction Risk

32

yield_reduction_risk == “Yes”

Tropical Zone Ecosystems

169

climate_zone == “Tropical Zone”

Multi-Image Comparison Participants

6,350

Has a multi_image_refs backreference

Background Images (No QA)

1,976

No qa and no multi_image_refs

Data Quality Flag - Coverage Over 100pct

11

cultivated_coverage_pct > 100 (source-data quirk, see above)

Dense Oil Palm Plantations

506

oil_palm_tree_count >= 70

Largest Tree Crowns

268

tree_crown_diameter_m >= 9

dataset.info contents#

  • multi_image_qa: list of ~4,107 multi-image question dicts (see above)

  • source: link to the original AgroMind Hugging Face dataset

  • paper: link to the AgroMind arXiv paper

Dataset Creation#

Curation Rationale#

Prior agricultural remote sensing benchmarks were limited to narrow scenarios (a single crop or a specific task like disease classification) and oversimplified task designs (basic classification/detection), which fails to capture the complexity of real-world agricultural decision-making. AgroMind was curated to integrate diverse scenes (five core domains: crop phenology monitoring, pest/disease detection, multi-sensor anomaly diagnosis, precision parcel management, and forestry assessment) and a hierarchical, four-dimension / 13-task evaluation framework — reflecting a “seeing clearly” to “reasoning accurately” progression — so that large multimodal models can be assessed on realistic, compositional agricultural understanding rather than isolated perception tasks.

Source Data#

Data Collection and Processing#

AgroMind integrates nine public datasets and one private field-parcel dataset, spanning satellite, UAV, and ground-camera sensors across 106 geographic regions. The mapping between the paper’s named source datasets and this FiftyOne dataset’s source_dataset values is:

source_dataset

Paper’s source dataset

Scenario

Agriculture

Agriculture-Vision Challenge (Chiu et al., 2020)

Multi-spectral (RGB+NIR) UAV anomaly detection, 512x512

Fruit

ACFR Orchard Fruit Dataset (Bargoti & Underwood, 2016)

Orchard fruit counting/classification

Leaf_diseases

2018 AI Challenge Dataset

Leaf disease/health classification, 26 disease types across 9 crops

Oil_palm_trees

OilPalmUAV Dataset (Zheng et al., 2021)

UAV oil-palm tree growth-status and counting, 1024x1024

Pest

IP102 Dataset (Wu et al., 2019)

102-category insect pest recognition, VOC-format

Rural

Global Parcel Dataset (private, “7-Province Ground Dataset”)

Multispectral GeoTIFF + shapefile farmland parcels, reprojected to EPSG:3857

Trees

OAM-TCD Dataset (Veitch-Michaelis et al., 2024)

Global tree-cover canopy maps, MS-COCO annotations, 2048x2048

corn

Maize Tassel Identification Dataset (iFLYTEK AI Competition 2024)

High-resolution maize tassel counting/localization, YOLO-format, 2736x1824

crop

PhenoBench Dataset (Weyler et al., 2024)

Crop/weed semantic + instance segmentation, pixel-level visibility maps, 1024x1024

CropHarvest

CropHarvest Dataset (Tseng et al., 2021)

Global multi-spectral (Sentinel-1/2, SRTM, ERA5) crop-type classification, 12-month temporal sequences

Per the paper’s preprocessing pipeline: TIFF remote sensing images were converted to PNG/JPG (colorful imagery) or concatenated grayscale blocks (non-visible spectral bands); high-resolution images underwent randomized parcel-boundary cropping for multi-scale sampling; augmented/duplicate/artifact-laden images were manually screened out; and instance-level statistics plus bounding boxes were derived from the original labels/masks. Questions were then generated via both rule-based templates (for normativity/logic) and human-authored templates (for flexibility/diversity), with a “logic enhancement” strategy that constructs step-by-step reasoning paths (e.g. locate box -> identify tree -> measure pixel coverage -> convert to physical scale) rather than using raw annotations directly as answers.

Who are the source data producers?#

The nine public source datasets were produced by their respective original authors (see the paper’s Table 4 and reference list, items [48]-[54] and the 2018 AI Challenge Dataset). The Global Parcel Dataset (mapped to Rural here) is a private dataset covering farmland parcels across seven Chinese provinces plus parts of Southeast Asia and Europe, collected by the AgroMind authors.

Annotations#

Annotation process#

Ground-truth QA pairs were derived from each source dataset’s original annotations (bounding boxes, segmentation masks, disease/species labels, GeoJSON crop labels, etc.) via dataset-specific processing (e.g. IP102’s VOC annotations were used to sample same-superclass pest distractors for multiple-choice questions; Agriculture-Vision’s per-image anomaly masks were used to compute area/overlap statistics and multi-anomaly spatial relationships; OilPalmUAV’s bounding boxes were used to overlay red highlight boxes and derive growth-stage/counting questions). Model-answered responses were then evaluated against these human-expert-annotated standard answers as part of a quality-control stage.

For human-performance benchmarking (not part of this FiftyOne dataset’s fields, but reported in the paper), approximately twenty student volunteers each independently answered a subset of questions (at least 3 volunteers per question), with only the required answer format specified (no additional guidance), and were compensated above minimum wage standards.

Who are the annotators?#

Ground-truth answers derive from the original source datasets’ expert/instance-level annotations, refined and restructured into QA form by the AgroMind authors. Human baseline responses (reported in the paper, not included in this dataset) were collected from approximately 20 undergraduate/graduate student volunteers.

Personal and Sensitive Information#

The dataset consists of agricultural remote sensing imagery (satellite, UAV, and ground-camera images of crops, farmland, orchards, and forests) and does not target or intentionally include personal or sensitive information. [More Information Needed] regarding whether any incidental personal information (e.g. people or vehicles visible in ground-camera images) was screened by the original source dataset authors.

Citation#

BibTeX:

@misc{li2025largemultimodalmodelsunderstand,
      title={Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind},
      author={Qingmei Li and Yang Zhang and Zurong Mai and Yuhang Chen and Shuohong Lou and Henglian Huang and Jiarui Zhang and Zhiwei Zhang and Yibin Wen and Weijia Li and Haohuan Fu and Jianxi Huang and Juepeng Zheng},
      year={2025},
      eprint={2505.12207},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2505.12207},
}

APA:

Li, Q., Zhang, Y., Mai, Z., Chen, Y., Lou, S., Huang, H., Zhang, J., Zhang, Z., Wen, Y., Li, W., Fu, H., Huang, J., & Zheng, J. (2025). Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind. arXiv preprint arXiv:2505.12207.

More Information#

This FiftyOne dataset was built directly from the official AgroMind Hugging Face release (images + QA.zip), parsing all 13 raw per-task JSON files into the unified structure described above. Beyond the fields and structure released by the original authors, this version adds: (1) rendered prompt and resolved answer_text fields on every QA pair, purpose-built to make constructing a fine-tuning dataset (e.g. a PyTorch Dataset) straightforward; (2) ~30 promoted, typed, filterable/visualizable ground-truth fields derived from the raw QA answers; and (3) 17 saved views for common exploration patterns, including two data-quality-flag views documenting quirks found in the original source data.