P3D-Bench

Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning

Yikang Yang^1,† · Zhanpeng Hu^1,† · Youtian Lin¹ · Mengqi Zhou¹ · Jingxi Xu² · Feihu Zhang² · Jiaheng Liu¹ · Yao Yao^1,‡

¹Nanjing University    ²Envision

_{† Equal contribution  ·  ‡ Corresponding author}

_{Scores of different models across the three tasks in P3D-Bench. Each task score is the average of that task's applicable metric buckets (Geometry, Topology, Judge, plus Part for Assembly-3D), rescaled to 0–100.}

---

News

• [2026-06] 🎉 We released P3D-Bench — the paper (arXiv), the evaluation code, and the Dataset on HuggingFace.

---

Abstract

Multimodal large language models can write code to produce complex programs as well as use programs to do 3D modeling, which opens up a new avenue for 3D generation powered by their priors, world knowledge and reasoning. Yet existing benchmarks rarely evaluate 3D modeling through code. Such modeling demands more than runnable code: from a text or visual specification, a model must generate a parametric 3D program that is geometrically precise, semantically aligned and assembly-consistent.

We introduce P3D-Bench, a benchmark for parametric 3D generation. Unlike a 3D mesh, a parametric 3D program exposes explicit dimensions, construction operations and part relations, revealing whether a model recovers a design's structure, not just its appearance. Under a unified protocol, P3D-Bench covers three task families (Text-to-3D, Image-to-3D and Assembly-3D) and scores each output for executability, geometric fidelity, topology, text-grounded constraints, multiview semantic alignment and part-level structure. We evaluate frontier MLLMs and text-only LLMs on 400 text cases, 400 image cases and 203 annotated assemblies, with domain-specific models as reference points.

Our extensive evaluation yields three findings. First, assemblies are the hardest setting, where models still fail to compose multiple parts into a coherent structure. Second, models can often recover the global shape and semantic identity of the target object, yet fail to reproduce the precise parametric geometry specified by the input. Third, part-level modeling remains weak on assemblies, where models recover neither the geometry of each part nor the right number of parts. These results position P3D-Bench as a benchmark for evaluating precise parametric geometry and part-level structure in parametric 3D generation.

---

Environment Setup

1. Install

git clone https://github.com/SpatiaOS/P3D-Bench.git
cd P3D-Bench

# create an environment (conda or venv)
conda create -n p3dbench python=3.10 -y
conda activate p3dbench

# core install (CLI + model adapters + config)
pip install -e .

Heavy geometry/render dependencies are optional extras, installed only for the metric buckets and output formats that need them. For the in-repo demo smoke test, the core install is enough when you use --dry-run; real geometry scoring needs the geometry extra.

pip install -e ".[geometry]"   # OCC/OCP + trimesh → Geometry / Topology / Part metrics
pip install -e ".[render]"     # pyrender / Blender → Judge multiview renders
pip install -e ".[cadquery]"   # CadQuery output format
pip install -e ".[all]"        # everything

Each output format must also be compiled to STL before any metric can read it, so pick the extra that matches the --format you plan to run:

--format	Needed to compile it
openscad	the openscad binary (external runtime, no pip extra)
threejs	Node.js (the Three.js runtime ships vendored under p3dbench/compile/three/, so no npm install is needed)
cadquery	pip install -e ".[cadquery]"
minimal-json	pip install -e ".[geometry]" (the Text2CAD interpreter needs SciPy)

openscad and threejs write STL directly from their own runtime, so they compile on the core install; cadquery and minimal-json go through the shared OCP + trimesh mesher and report 0 valid cases without the extra above.

2. API keys

Bring your own keys. Secrets never go in YAML — configs/models.yaml holds only metadata (provider, model id, base URL, and the name of the env var that holds the key); .env.example lists the key names.

cp .env.example .env            # then fill in your keys

# .env  (key NAMES only; values stay local)
OPENROUTER_API_KEY=             # default examples use OpenRouter
HF_TOKEN=                       # reserved for future full-split downloads
P3DBENCH_CACHE_DIR=.cache/p3dbench

Register a model by adding a block to configs/models.yaml and the matching key in .env. Any OpenAI-compatible endpoint (OpenRouter, vLLM, LM Studio, …) works via the openai_compatible provider:

models:
  qwen:
    provider: openai_compatible
    model: qwen/qwen3.6-plus
    api_key_env: OPENROUTER_API_KEY
    base_url: https://openrouter.ai/api/v1
    temperature: 0.7
    max_tokens: 65536

  local-qwen:
    provider: openai_compatible
    model: qwen2.5-vl-instruct
    api_key_env: OPENROUTER_API_KEY   # set a dummy value if your local server ignores auth
    base_url: http://localhost:8000/v1

---

Quick Start

An evaluation run is defined by three orthogonal choices — task, output format, and metric bucket — that you pin independently from the CLI:

Axis	Flag	Choices
Task	--task	text-to-3d · image-to-3d · assembly-3d
Format	--format	minimal-json · openscad · cadquery · threejs
Metric	--metric	valid · geometry · topology · judge · part · all

The CLI validates --format against the chosen task's supported formats.

1. Check the demo data (a few local cases per task; see Dataset):

p3dbench download --split demo
p3dbench validate --split demo

For the full 400 / 400 / 203 split, how much you need locally depends on the task.

Text-to-3D needs nothing local. Its 400 GT programs are Text2CAD-derived, which is redistributable under CC BY-NC-SA 4.0, so they ship on the Hub and materialize in one command:

pip install -e ".[geometry]"                        # the GT programs are compiled while materializing
p3dbench download --split full --tasks text-to-3d   # 400 cases, straight from HuggingFace
p3dbench validate --split full --tasks text-to-3d

Image-to-3D and Assembly-3D need the upstream Fusion 360 Gallery geometry, which Autodesk's license does not let us redistribute — the Hub ships their UID lists, annotations and QA banks, but not the raw CAD. Obtain it under its own license, point --source-root at it, and the download / prepare stages build the evaluator-ready data/full/ tree from it (see Dataset and docs/DATA.md):

# A) If you already have the research-prepared _shared_cache (one-click):
p3dbench download --split full --source-root /path/to/cad_dataset   # materialize from a prebuilt cache

# B) If you have only the raw upstream (Fusion 360 Gallery + Text2CAD v1.1):
p3dbench prepare --split full --source-root /path/to/cad_dataset    # build _shared_cache from raw, then materialize

p3dbench validate --split full

download, prepare and validate all take --tasks to work on a subset (download and prepare also take --limit). prepare reuses an existing _shared_cache when present (so path A keeps working unchanged) and otherwise reproduces it with the same data-processing pipeline as the research repo: the input image is an OCC single-view render and the judge images are Blender clay multiviews. It needs the geometry + render extras, a Blender binary on $P3DBENCH_BLENDER, and Xvfb + OCP (OCP ships with the cadquery extra). Running prepare for Text-to-3D needs no Blender (its cache holds only the OCC single view), but it does still render that view, so Xvfb + OCP are required. The Hub-only download path above needs neither — just the geometry extra.

2. Smoke-test prompt construction without API keys:

MODEL=qwen examples/run_smoke.sh

The smoke script uses --dry-run, so it validates local demo manifests and builds prompts without calling a model or compiling generated CAD.

3. Run one task × one format × one metric with a configured model:

p3dbench run --task image-to-3d --format openscad --metric geometry \
  --model qwen --split demo --limit 1

run chains the four stages and writes results under results/<run-id>/. You can also run each stage on its own — every stage reads/writes a plain JSONL artifact, with no resume/checkpoint state — so you can re-score the same predictions under a different metric without re-running inference. Each stage defaults its output next to its input, so pinning --out on infer lets the rest chain by bare filename:

p3dbench infer     --task text-to-3d --format minimal-json --model qwen --split demo --limit 1 --out predictions.jsonl
p3dbench compile   --pred predictions.jsonl                              # → compiled.jsonl
p3dbench score     --compiled compiled.jsonl --metric topology           # → metrics.jsonl
p3dbench summarize --metrics metrics.jsonl                               # → summary.json

(Without --out, infer writes to results/<run-id>/predictions.jsonl; point --pred at that path instead.)

Useful flags: --limit N (first N cases), --dry-run (build prompts / validate config without calling a model), --split demo, --refine-attempts N (Image-/Assembly-3D only: compile-check-retry with error feedback, default 3, 1 disables; Text-to-3D is always single-shot), --text-mode {parametric,descriptive} (Text-to-3D only — picks the input spec, parametric vs descriptive, and the metric panel reported; descriptive falls back to the parametric text when no descriptive annotation ships with the case, e.g. the demo split).

---

Dataset

• Demo split (3 cases per task) ships in data/demo/ with manifests under data/manifests/ — a zero-setup smoke test.

• Full split (Text-to-3D 400 / Image-to-3D 400 / Assembly-3D 203). 🤗 HuggingFace publishes everything redistributable — the benchmark UID lists, the P3D-derived text / assembly annotations, the Text-to-3D QA banks (the MCQ banks scored by the Judge bucket), and the 400 Text-to-3D GT programs themselves (Text2CAD-derived minimal-JSON, CC BY-NC-SA 4.0).

  • Text-to-3D therefore needs no local upstream at all: p3dbench download --split full --tasks text-to-3d materializes all 400 cases from the Hub.
  • Image-to-3D / Assembly-3D additionally need the upstream Fusion 360 Gallery raw geometry, which Autodesk's license does not permit us to redistribute. Obtain it yourself, then let the CLI bridge the gap: download (Hub metadata + materialize from a prebuilt _shared_cache) or prepare (build the per-case _shared_cache from the raw trees first), each writing data/full/ + data/manifests/*_full.jsonl. p3dbench prepare --source-root <path> reproduces the cache with the research data-processing pipeline (OCC single-view input + Blender clay judge multiviews); a prebuilt _shared_cache is auto-detected and reused.

  See docs/DATA.md for the expected --source-root layout, the prepare stage, and licensing.

---

License

Code and data are licensed separately, and the data follows the terms of its upstream sources (non-commercial research use only, with attribution):

Component	Source	License
Benchmark code (this repo)	—	MIT (see LICENSE)
P3D-Dataset — Text-to-3D split	derived from Text2CAD v1.1	CC BY-NC-SA 4.0
P3D-Dataset — Image-to-3D & Assembly-3D splits	derived from Fusion 360 Gallery Dataset	Fusion 360 Gallery Dataset License (Autodesk, non-commercial)

Both dataset sources permit non-commercial research use only and require attribution; redistributed portions and modifications must carry the same restrictions. By using the P3D-Dataset you agree to the upstream license terms.

---

Citation

If you find P3D-Bench useful, please cite our paper:

@misc{yang2026p3dbenchbenchmarkingmllmsparametric,
      title={P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning}, 
      author={Yikang Yang and Zhanpeng Hu and Youtian Lin and Mengqi Zhou and Jingxi Xu and Feihu Zhang and Jiaheng Liu and Yao Yao},
      year={2026},
      eprint={2606.11152},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.11152}, 
}