AISBench · Huafubing · Aug 12, 2026
diff --git a/din_integration/README.md b/din_integration/README.md
@@ -0,0 +1,138 @@
+# din_integration — SWE-bench DinD Evaluation Pipeline
+
+> **状态**: 本目录是对 SWE-bench DinD (Docker-in-Docker) 多 agent × 多 case 评测流水线的**整包搬运**,
+> 作为 AISBench 仓库根目录下的独立子目录提交。
+>
+> **历史**: 代码原位于 `/home/zengziyu/mini_matrix/cli/` + `mini_matrix/scripts/` + `mini_matrix/config/`,
+> 经评审后整合到本目录,便于在 AISBench 仓内统一治理。
+
+---
+
+## ⚠ 路径假设(本包当前已知局限)
+
+本包直接拷贝自 `mini_matrix/`,**未做 env-var 化路径重构**。这意味着:
+
+| 路径硬编码 | 假设值 |
+|---|---|
+| `swebench_dind/launcher.py:58` api_key.env | `/home/zengziyu/mini_matrix/scripts/api_key.env` |
+| `swebench_dind/config.py:14-17` ROOT | 由 `Path(__file__).parent.parent` 自动推导 → 当前为 `din_integration/` |
+| `swebench_dind/config.py:27-35` jobs/tasks/logs | 位于 ROOT 同级 → 当前为 `din_integration/{jobs,tasks,logs}/` |
+| `swebench_dind/config.py:31` api_key.env | `ROOT/scripts/api_key.env` → 当前为 `din_integration/scripts/api_key.env` |
+
+**实测工作方式**(以最小集 PR 为目标):
+
+```bash
+# 1. 把 secrets 放在 launcher.py 期望的位置(原 mini_matrix/scripts/api_key.env)
+ln -sf /path/to/your/api_key.env /home/zengziyu/mini_matrix/scripts/api_key.env
+
+# 2. 在 din_integration/ 下装包
+cd din_integration
+pip install -e .
+
+# 3. 准备运行时数据目录(可指向任意位置)
+mkdir -p ~/swebench_dind_{jobs,tasks,logs}
+
+# 4. 跑 CLI(注意 launcher.py 仍会从 mini_matrix/scripts/ 读 key)
+swebench-dind --version
+```
+
+**完整 env-var 重构方案**见设计文档 [方案 A §5](file:///home/zengziyu/aisbench/docs/research/24-swebench-dind-integration-plan-A-slim-2026-08.md#5-代码修改清单共-9-处),
+本次未执行(以最小集 PR 为目标)。
+
+---
+
+## 📂 本目录结构
+
+```
+din_integration/
+├── README.md                                       ← 本文件
+├── pyproject.toml                                  ← Python 包元数据 (name=swebench-dind)
+├── bin/swebench-dind                               ← shell wrapper
+├── docs/
+│   ├── CLI-USAGE.md                                ← CLI 完整命令参考
+│   └── MIGRATION.md                                ← 老脚本 → CLI 对照表
+├── swebench_dind/                                  ← 核心 Python 包
+│   ├── __init__.py                                 ← __version__ = "0.1.0"
+│   ├── __main__.py
+│   ├── cli.py                                      ← Typer 7 子命令入口
+│   ├── config.py                                   ← 单一真相源(常量 + tag 推导)
+│   ├── container.py                                ← DinD 容器生命周期
+│   ├── builder.py                                  ← L3/L4 镜像烤制(Jinja2)
+│   ├── launcher.py                                 ← harbor jobs start 拼装 + Rich 进度
+│   ├── patcher.py                                  ← idempotent install probe 注入
+│   ├── summarizer.py                               ← result.json → md/csv/json
+│   ├── dockerfiles/                                ← L1/L2 Dockerfile 模板 (5 个 .j2)
+│   │   ├── Dockerfile.l1-base.j2
+│   │   ├── Dockerfile.l2-agent-aider.j2
+│   │   ├── Dockerfile.l2-agent-msa.j2
+│   │   ├── Dockerfile.l2-agent-oh.j2
+│   │   └── Dockerfile.l2-agent-qwen.j2
+│   └── aisbench_adapter/                           ← AISBench BaseTask 适配
+│       ├── __init__.py
+│       ├── task.py                                 ← SwebenchDindTask (BaseTask 子类)
+│       ├── result_writer.py                        ← harbor → AISBench schema
+│       └── runner.py                               ← subprocess 入口
+├── configs/
+│   ├── matrix.yaml                                 ← Harbor JobConfig (15 task × 4 agent = 60 trial)
+│   └── swebench_dind_3x3.py                        ← AISBench config 示例 (3 cases × 3 agents)
+└── scripts/
+    ├── start_orchestrator.sh                       ← 启动 DinD 容器 + bind mount
+    ├── summarize.py                                ← 汇总 jobs/*/result.json
+    └── filter_matrix.py                            ← 子集过滤
+```
+
+---
+
+## 🚀 快速使用(在 din_integration/ 内)
+
+```bash
+cd din_integration
+pip install -e .
+
+# 启 DinD (需要 host 已安装 docker + qemu binfmt)
+bash scripts/start_orchestrator.sh
+
+# 跑单个 trial
+swebench-dind launch trial --case 11099 --agent aider --wait
+
+# 汇总结果
+swebench-dind summarize
+```
+
+---
+
+## 📦 跟 AISBench 的集成方式
+
+`aisbench_adapter/task.py` 实现 `SwebenchDindTask`,继承自 `ais_bench.benchmark.tasks.base.BaseTask`,
+通过 `ais_bench.benchmark.registry.TASKS.register_module()` 注册。
+
+**AISBench config 示例**见 [configs/swebench_dind_3x3.py](configs/swebench_dind_3x3.py),
+3 cases × 3 agents = 9 trial 的最小矩阵。
+
+⚠ **本包不通过 `setup.py` entry_point 注册到 `ais_bench.benchmark_plugins`** —— 因为:
+1. swebench-dind 是**重量级 CLI + DinD 镜像**集成,不是传统意义上的 plugin (单文件 import)
+2. AISBench plugin 接口需要 `pip install` 后才能 import,本包需要 host 上 docker + QEMU 准备
+3. 用户显式选择 "最小集 PR" 路径(参见 doc 23/24 讨论)
+
+如需 AISBench 标准 plugin 形式接入,后续可加 `setup.py` + `entry_points`.
+
+---
+
+## 📚 关联文档
+
+- [mini_matrix/docs/research/18-swebench-dind-complete-project-doc-2026-08.md](../../../mini_matrix/docs/research/18-swebench-dind-complete-project-doc-2026-08.md) — 工程实现细节
+- [aisbench/docs/research/24-swebench-dind-integration-plan-A-slim-2026-08.md](../../../aisbench/docs/research/24-swebench-dind-integration-plan-A-slim-2026-08.md) — 方案 A 设计文档(精简版)
+- [aisbench/docs/research/23-swebench-dind-integration-into-aisbench-2026-08.md](../../../aisbench/docs/research/23-swebench-dind-integration-into-aisbench-2026-08.md) — 方案 B 设计文档(完整版)
+
+---
+
+## 📊 已验证
+
+- 12 trial 历史 (9 PASS / 75% pass@1) + 1 次 e2e-cli-11099-aider PASS (4min 49s)
+- 详见 [mini_matrix/docs/research/18 §11](../mini_matrix/docs/research/18-swebench-dind-complete-project-doc-2026-08.md)
+
+---
+
+## 📝 License
+
+MIT (沿用 swebench-dind 包原始 license)
diff --git a/din_integration/bin/swebench-dind b/din_integration/bin/swebench-dind
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+# Thin shell wrapper around `python -m swebench_dind`.
+# After `pip install -e .` you can also call `swebench-dind` directly.
+set -euo pipefail
+exec python -m swebench_dind "$@"
diff --git a/din_integration/configs/matrix.yaml b/din_integration/configs/matrix.yaml
@@ -0,0 +1,116 @@
+# ============================================================================
+# Multi-Bench × Multi-Agent Matrix Configuration
+# ============================================================================
+#
+# 这是 Harbor 0.20.x 的 JobConfig 格式(harbor.models.job.config.JobConfig)。
+# 一份 yaml 即可驱动 N 个 dataset × M 个 agent × K 个 model 的笛卡尔积评测。
+#
+# 引用:
+#   - Harbor CLI: harbor jobs start -c <this.yaml>
+#   - Schema: harbor/src/harbor/models/job/config.py
+#
+# 重新跑矩阵:
+#   bash scripts/run_matrix.sh
+#
+# ============================================================================
+
+job_name: matrix-mini-001
+
+# ---------------------------------------------------------------------------
+# 全局超时(harbor 0.20.x 单位:秒,multiplier 是相对默认值)
+# ---------------------------------------------------------------------------
+timeout_multiplier: 2.0           # 整体 ×2(覆盖 default)
+agent_setup_timeout_multiplier: 4.0   # agent install 段 ×4(aider 装要 17 分钟)
+verifier_timeout_multiplier: 2.0     # verifier 跑测试 ×2
+environment_build_timeout_multiplier: 8.0  # 首次 image build ×8
+
+# ---------------------------------------------------------------------------
+# 路径(env vars 注入到 trial container,harbor 不会自动传 orchestrator 的 env)
+# ---------------------------------------------------------------------------
+environment:
+  env:
+    OPENAI_API_BASE: "https://api.siliconflow.cn/v1"
+    OPENAI_API_KEY: "${OPENAI_API_KEY}"
+    LLM_BASE_URL: "https://api.siliconflow.cn/v1"
+    LLM_MODEL: "openai/Qwen/Qwen3-Coder-30B-A3B-Instruct"
+
+verifier:
+  env:
+    OPENAI_API_KEY: "${OPENAI_API_KEY}"
+
+# ---------------------------------------------------------------------------
+# 并发(默认 harbor 串行,本矩阵 n_concurrent_trials=2 = 2 个 trial 并行)
+# ---------------------------------------------------------------------------
+n_concurrent_trials: 2
+
+# ---------------------------------------------------------------------------
+# Datasets: 当前 baked 矩阵覆盖的 6 个 SWE-bench Django case
+# 路径指向 /opt/swebench/data/tasks/<task-name>(容器内)/
+# ---------------------------------------------------------------------------
+datasets:
+  - path: /opt/swebench/data/tasks/django__django-10097-aider
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-10097-msa
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-10554-aider
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-10554-msa
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-10880-aider
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-10880-msa
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-11099-aider
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-11099-msa
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-11099-oh
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-12308-aider
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-12308-msa
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-12308-oh
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-13741-aider
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-13741-msa
+    n_tasks: 1
+  - path: /opt/swebench/data/tasks/django__django-13741-oh
+    n_tasks: 1
+
+# ---------------------------------------------------------------------------
+# Agents: 4 个 agent × 1 个 LLM
+# Harbor 自动按 tasks × agents 生成笛卡尔积,= 15 × 4 = 60 trials
+# ---------------------------------------------------------------------------
+agents:
+  - name: oracle
+    model_name: null            # oracle 不调 LLM
+    n_concurrent: 2
+
+  - name: aider
+    model_name: "openai/Qwen/Qwen3-Coder-30B-A3B-Instruct"
+    env:
+      AIDER_API_KEY: "${OPENAI_API_KEY}"
+      LITELLM_API_KEY: "${OPENAI_API_KEY}"
+      LLM_API_KEY: "${OPENAI_API_KEY}"
+    n_concurrent: 1
+
+  - name: mini-swe-agent
+    model_name: "openai/Qwen/Qwen3-Coder-30B-A3B-Instruct"
+    env:
+      MSWEA_API_KEY: "${OPENAI_API_KEY}"
+      LITELLM_API_KEY: "${OPENAI_API_KEY}"
+      LLM_API_KEY: "${OPENAI_API_KEY}"
+    n_concurrent: 1
+
+  - name: openhands-sdk
+    model_name: "openai/Qwen/Qwen3-Coder-30B-A3B-Instruct"
+    env:
+      LLM_API_KEY: "${OPENAI_API_KEY}"
+    n_concurrent: 1
+
+# ---------------------------------------------------------------------------
+# 重试(每 trial 失败后跑几次)
+# ---------------------------------------------------------------------------
+n_attempts: 1
diff --git a/din_integration/configs/swebench_dind_3x3.py b/din_integration/configs/swebench_dind_3x3.py
@@ -0,0 +1,86 @@
+"""SWE-bench DinD 3×3 example config for ais_bench.
+
+Demonstrates how to invoke the din_integration/ SWE-bench DinD pipeline
+via AISBench's task system. Run after pip-installing both ais_bench and
+this swebench_dind package:
+
+    pip install ais_bench
+    cd din_integration && pip install -e .
+    ais_bench din_integration/configs/swebench_dind_3x3.py
+
+Note: This config relies on the swebench-dind CLI being on PATH and the
+DinD orchestrator container already running. See din_integration/README.md
+for setup.
+"""
+from mmengine.config import read_base
+
+# Use a no-op task for the standard Infer stage; the SWE-bench DinD pipeline
+# doesn't fit AISBench's standard model-inference shape.
+with read_base():
+    infer = dict(runner=dict(task=dict(type="EmptyTask")))
+
+# Eval stage uses our custom SwebenchDindTask.
+eval = dict(
+    runner=dict(
+        task=dict(
+            type="SwebenchDindTask",
+        ),
+    ),
+)
+
+# Summarizer: HarborSummarizer reads <work_dir>/results/<model>/<dataset>.json
+summarizer = dict(attr="accuracy", type="HarborSummarizer")
+
+work_dir = "./outputs/swebench_dind_3x3/"
+
+# 3 cases × 3 agents = 9 trials
+models = [
+    dict(
+        abbr="qwen3-coder-30b",
+        type="LiteLLMModel",
+        model_names=["openai/Qwen/Qwen3-Coder-30B-A3B-Instruct"],
+        agent_name="aider",
+        agent_kwargs={},
+        agent_env={},
+    ),
+    dict(
+        abbr="qwen3-coder-30b",
+        type="LiteLLMModel",
+        model_names=["openai/Qwen/Qwen3-Coder-30B-A3B-Instruct"],
+        agent_name="mini-swe-agent",
+        agent_kwargs={},
+        agent_env={},
+    ),
+    dict(
+        abbr="qwen3-coder-30b",
+        type="LiteLLMModel",
+        model_names=["openai/Qwen/Qwen3-Coder-30B-A3B-Instruct"],
+        agent_name="qwen-coder",
+        agent_kwargs={},
+        agent_env={},
+    ),
+]
+
+datasets = [
+    dict(
+        abbr="django-11099",
+        type="SwebenchDindDataset",
+        args=dict(
+            path="/opt/swebench/data/tasks/django__django-11099-aider",
+        ),
+    ),
+    dict(
+        abbr="django-12308",
+        type="SwebenchDindDataset",
+        args=dict(
+            path="/opt/swebench/data/tasks/django__django-12308-msa",
+        ),
+    ),
+    dict(
+        abbr="django-13741",
+        type="SwebenchDindDataset",
+        args=dict(
+            path="/opt/swebench/data/tasks/django__django-13741-aider",
+        ),
+    ),
+]