SafeOrch: Runtime Safety Orchestration for Sensor-Driven Multi-Agent IoT Systems

Code and data for the paper submitted to MDPI Sensors.

Quick Start

# 1. Install dependencies
pip install -r requirements.txt

# 2. Install Ollama and pull models
# https://ollama.ai
ollama pull qwen2.5:3b
ollama pull qwen2.5:7b
ollama pull llama3.1:8b

# 3. Run all experiments
python run_all.py

Repository Structure

safeorch/
├── config.py                  # Safety rules (R01-R12), compatibility table, LLM config
├── safeorch.py                # Core pipeline: C1 + 4 agents + Coordinator + C2
├── llm_agents.py              # LLM-backed domain agents (Energy/Security/Comfort/Privacy)
│
├── experiments/
│   ├── e2_experiment.py       # E1: C1 sensor-side evaluation (500 attacks)
│   ├── e2_extended.py         # E1 extended: confusion matrix, per-class metrics
│   ├── e3_experiment.py       # E2: 29-scenario conflict resolution (rule agent)
│   ├── e3_llm_experiment.py   # E2: 29-scenario conflict resolution (LLM agents)
│   ├── e3_repeat_runs.py      # Stability: 5× repeated runs
│   ├── e3_temp_sweep.py       # Temperature sensitivity: 4 temps × 3 models
│   └── e_t2_injection.py      # T2: 3-phase prompt injection probe
│
├── data/
│   ├── benign_states.json     # 57 benign Z-Wave snapshots from BCCC dataset
│   └── benign_profile.json    # Calibrated sensor profiles
│
├── results/                   # Pre-computed results (JSON)
│   ├── e2_extended_results.json
│   ├── e3_llm_results.json
│   ├── e3_repeat_results.json
│   ├── e3_temp_sweep_results_qwen3b.json
│   ├── e3_temp_sweep_results_qwen7b.json
│   ├── e3_temp_sweep_results_ollama8b.json
│   └── e_t2_injection_results.json
│
├── requirements.txt
├── run_all.py                 # Master script to reproduce all experiments
└── README.md

Hardware & Software

Component	Version
OS	Windows 11 / Ubuntu 24.04
CPU	AMD Ryzen 7 5800H (8-core)
RAM	14 GB
GPU	None (CPU-only inference)
Ollama	0.3.10
Python	3.11+
Qwen2.5-3B	Q4_K_M, sha256-5ee4f07c
Qwen2.5-7B	Q4_K_M, sha256-2bada8a7
Llama3.1-8B	Q4_K_M

Experiments

E1: C1 Plausibility Monitor (Table 6 in paper)

python experiments/e2_extended.py --attacks-per-type 100

• 500 synthetic attacks (5 types × 100) + 29 benign windows
• Output: confusion matrix, per-class recall, TPR/FPR

E2: Action-Side Conflict Resolution (Table 7 in paper)

# Rule agent baseline
python experiments/e3_experiment.py

# LLM agents (change model in config.py)
python experiments/e3_llm_experiment.py

Stability Test (Section 4.1)

python experiments/e3_repeat_runs.py    # 5× repeated runs
python experiments/e3_temp_sweep.py     # Temperature sweep (0.1-0.7)

T2 Prompt Injection Probe (Section 4.3)

python experiments/e_t2_injection.py

• Phase 1: 20 adversarial MQTT payloads → Perception Agent
• Phase 2: 5 hijack prompts → Security Agent (bypass mode)
• Phase 3: Full pipeline safety net

Dataset

Benign sensor traces are extracted from the BCCC-IoT-IDS-Zwave-2025 dataset. The original pcap files are publicly available; benign_states.json contains the pre-parsed MQTT readings used in our evaluation.

Sampling Parameters

• Temperature: 0.1 (default), sweep: 0.1, 0.3, 0.5, 0.7
• Top-p: 0.9 (Ollama default)
• No fixed random seed

License

MIT License

Citation

@article{zhou2026safeorch,
  title={SafeOrch: Runtime Safety Orchestration for Sensor-Driven Multi-Agent IoT Systems},
  author={Zhou, Wanyi and Li, Delong and Wang, Xu},
  journal={Sensors},
  year={2026}
}