Skip to content

Data Access Guide

GreenFlux does not store raw calibration datasets in the public Git repository. The repository stores code, small packaged reference resources, project definitions and documentation; large measured datasets stay local under data/raw/.

This keeps clones small, avoids redistributing third-party datasets under the wrong license and makes analyses reproducible from the original source.

Expected Local Layout

Create this structure at the repository root:

data/
  raw/
    agc_2019/
      Weather/
        Weather.csv
      Reference/
        GreenhouseClimate.csv
        CropParameters.csv
        Production.csv
        Resources.csv
        TomQuality.csv
      AICU/
      Automatoes/
      Digilog/
      IUACAAS/
      TheAutomators/
    greenlight_led_hps/
      Raw data/
        Weather raw.csv
        HPS raw.csv
        LED raw.csv
      Processed data/
        dataHPS.csv
        dataLED.csv

Files under data/raw/ are ignored by git.

Dataset 1: Autonomous Greenhouse Challenge 2019

Use this dataset for tomato crop calibration, climate/crop coupling, resources, harvest and root-zone follow-up work.

Source:

  • DOI: https://doi.org/10.4121/uuid:88d22c60-21b3-4ea8-90db-20249a5be2a7
  • Repository page: https://data.4tu.nl/articles/_/12764777/2
  • License reported by the repository page: CC0

Manual download:

  1. Open the DOI or repository page.
  2. Download the Autonomous Greenhouse Challenge second-edition archive.
  3. Place the archive under data/raw/.
  4. Extract it into data/raw/agc_2019/.
  5. Confirm that data/raw/agc_2019/Weather/Weather.csv exists.

The local archive used during development was named:

data/raw/AutonomousGreenhouseChallenge_edition2.7z

The extracted folder should contain team folders such as Reference, AICU, Automatoes, Digilog, IUACAAS and TheAutomators.

Dataset 2: GreenLight LED/HPS Bleiswijk

Use this dataset for measured greenhouse replay, HPS/LED climate calibration, residual attribution and project export validation.

Source:

  • DOI: https://doi.org/10.4121/78968e1b-eaea-4f37-89f9-2b98ba3ed865
  • Repository page: https://data.4tu.nl/datasets/78968e1b-eaea-4f37-89f9-2b98ba3ed865
  • License reported by the repository page: CC BY-SA 4.0

Manual download:

  1. Open the DOI or repository page.
  2. Download the readme, methodology, raw-data archive and processed-data archive.
  3. Place the downloaded files under data/raw/.
  4. Extract the raw and processed archives into data/raw/greenlight_led_hps/.
  5. Confirm that data/raw/greenlight_led_hps/Raw data/Weather raw.csv exists.

The local files used during development were named:

data/raw/greenlight_Readme.txt
data/raw/greenlight_Methodology.txt
data/raw/greenlight_Raw_data.zip
data/raw/greenlight_Processed_data.zip

The optional Simulation data.zip archive is large and is not required for the current GreenFlux calibration and validation workflows.

Verify The Data Layout

GreenFlux includes a helper script that can download and prepare the supported external datasets:

uv run python scripts/download_calibration_data.py --dataset all

The script downloads files into data/downloads/ and extracts them into data/raw/. GreenLight archives are extracted with Python's standard zipfile module. The AGC archive is a .7z file, so extraction requires a local 7z, 7zz or 7za executable. If no 7zip executable is installed, the script still downloads the AGC archive and tells you where to extract it manually.

Useful variants:

uv run python scripts/download_calibration_data.py --dataset greenlight
uv run python scripts/download_calibration_data.py --dataset agc --no-extract
uv run python scripts/download_calibration_data.py --verify-only

If you already downloaded the archives manually, reuse them:

uv run python scripts/download_calibration_data.py \
  --dataset greenlight \
  --no-download \
  --greenlight-raw-archive data/raw/greenlight_Raw_data.zip \
  --greenlight-processed-archive data/raw/greenlight_Processed_data.zip

Run the dataset inspection script after extraction:

uv run python examples/inspect_calibration_datasets.py

The script checks the default locations:

data/raw/agc_2019/
data/raw/greenlight_led_hps/

If your data live elsewhere, pass explicit roots:

uv run python examples/inspect_calibration_datasets.py \
  --agc-root /path/to/agc_2019 \
  --greenlight-root /path/to/greenlight_led_hps

Expected key files:

data/raw/agc_2019/Reference/GreenhouseClimate.csv
data/raw/agc_2019/Weather/Weather.csv
data/raw/greenlight_led_hps/Raw data/Weather raw.csv
data/raw/greenlight_led_hps/Raw data/HPS raw.csv
data/raw/greenlight_led_hps/Raw data/LED raw.csv

Commands Enabled By These Datasets

GreenLight measured replay:

uv run python examples/calibrate_greenlight_replay.py --installation HPS --limit 289
uv run python examples/calibrate_greenlight_replay.py --installation LED --limit 289
uv run python examples/validate_greenlight_long_crop_coupled.py --installation HPS --days 7
uv run python examples/validate_greenlight_long_crop_coupled.py --installation LED --days 7

AGC crop calibration:

uv run python examples/calibrate_agc_crop.py --team Reference --days 100

Project-local GreenLight workflows:

uv run python projects/greenlight_hps_bleiswijk/scripts/calibrate.py --run-id baseline --limit 289
uv run python projects/greenlight_hps_bleiswijk/scripts/validate.py --run-id 24h --hours 24
uv run --extra plots python projects/greenlight_hps_bleiswijk/scripts/run_simulation.py --run-id baseline --plots

Data Redistribution Policy

Do not commit downloaded archives, extracted raw data or generated calibration outputs to this repository.

Acceptable repository data:

  • small synthetic fixtures for unit tests
  • small packaged reference tables needed by examples
  • documentation describing dataset sources and expected layout
  • lightweight calibration reports and parameter JSON files when they are intentionally curated

Not acceptable in normal commits:

  • data/raw/
  • data/downloads/
  • outputs/
  • projects/*/calibration/
  • projects/*/results/
  • large notebooks with embedded outputs

If a future release needs fully reproducible large artifacts, prefer a separate data release, Zenodo record, GitHub Release asset, DVC remote or Hugging Face Dataset, and keep the code repository pointing to that artifact with clear checksums and license metadata.

More Details

See data/downloaded_datasets.md for the local inventory used during development, including row counts and important file roles. See dataset_equipment_control_audit.md for the equipment, actuator, setpoint and hidden-control interpretation used by the calibration workflows.