CityCleanDetection

3D perception for the CityClean cell: detects where to grab a carpet and where a crate's rim is, from a depth camera, and serves those poses to the Unity robot app.

Everything is plain numpy + Open3D — no ML, no GPU. Each detector is a pure function over a point cloud, wrapped in a small TCP server that Unity talks to.

---

Quick start

./run.sh                # create venv (first run), install deps, open the web UI on :8077
./run.sh 9000           # ... on a different port
./run.sh --setup-only   # just build the venv and exit

run.sh is only a convenience wrapper (venv + deps + launch). Once the venv exists you can run the UI directly — this is the usual way on Windows, where the camera lives:

python tools/flow_viz/server.py                    # :8077
python tools/flow_viz/server.py 9000               # custom port
python tools/flow_viz/server.py --host 0.0.0.0     # expose to Unity / the LAN

# Windows (camera rig), using the venv interpreter explicitly:
venv\Scripts\python.exe tools\flow_viz\server.py

Then open http://localhost:8077. Without a camera you can still browse and re-run every saved capture in captures/ offline — that's the normal way to tune.

---

The pieces

Piece	What it is
carpet/	Side-view carpet grab detection (curve → seam → fold arbitration) + conveyor mode
box/	Open-crate rim detection → 4 rim corners, centre, normal, long/short axis
handeye/	Eye-in-hand calibration: solves camera → flange from a fixed marker seen at many poses
segmentation/	Camera backends (Vzense/NYX650 via ScepterSDK, RealSense) + point-cloud acquisition
tools/flow_viz/	The web UI — live view, config editor, calibration wizards, capture browser
SolidCylinder/	Legacy cylinder marker detector (superseded by the box marker; kept for reference)

---

Running things

1. The web UI (main tool)

./run.sh                                        # or, directly:
python tools/flow_viz/server.py [port] [--host 0.0.0.0]

--host 0.0.0.0 also exposes GET /api/grab so Unity can poll the latest grab pose over the LAN (it defaults to 127.0.0.1, local only).

Modes across the top: Carpet · Box · Conveyor · Eye-in-Hand · Gripper.

• Pick a saved capture to run the detector and watch the decision chain stage by stage.
• Start live view streams from the camera (raw feed beside the cropped/processed cloud).
• Config edits the active profile live and saves back to config/*.json.
• Save the current frame to captures/ as .npy (cloud) or .ply.

2. Production TCP servers (what Unity connects to)

python carpet_server.py           # port 9879, "CRPT" protocol
python box_server.py              # port 9880, "BOX?" protocol

Both accept --host, --port, --config.

3. One-shot CLI (headless / debugging)

python box_detection.py    --input captures/foo.npy          # run on a saved cloud
python carpet_detection.py --input captures/foo.npy
python carpet_detection.py --camera-type vzense --save captures/new.npy   # grab one frame
python box_detection.py    --no-visualize --input captures/foo.npy        # print metrics only

4. Tests

python -m pytest -q     # 143 tests; open3d-dependent ones skip if it's missing

---

How the Unity integration works

One TCP server per detector. There is no "mode" flag — the port IS the detector.

Detector	Command	Port	Magic	Reply
Carpet (side-view grab)	python carpet_server.py	9879	CRPT	JSON furrow_summary
Box (crate rim)	python box_server.py	9880	BOX?	JSON box_summary

Wire format (both): header <IBII = magic(u32) + msg_type(u8) + seq(u32) + length(u32), little-endian.

• 0x01 DETECT_REQ → payload = N×3 float32 XYZ, camera frame, metres (12 B/point)
• 0x81 DETECT_RESP ← UTF-8 JSON
• 0x03 PING / 0x83 PONG, 0xFF ERROR ({code, message})

Two camera-ownership modes (box server)

The box server picks the mode per request, from the payload — same binary, no config:

Payload	Who owns the camera	Use when
empty	Python grabs its own Vzense frame	Unity doesn't need the cloud
N points	Unity captures and sends the cloud	You want the point cloud rendered in Unity

The camera is single-owner — only one process can open it. If both Unity's Vzense driver and the Python server try, you get SC_OPENED / Failed to open Vzense camera. Pick one owner.

Full Unity-side guide: docs/trubase_box_pick_integration.md.

---

Eye-in-hand calibration

Solves X = camera → flange: the fixed transform from the arm's flange to the camera bolted on it. Once you have X, any camera-frame detection converts into robot base coordinates.

How it works. A box sits fixed on the table. At each capture the app pairs (a) the robot's live TCP pose with (b) a box rim detection (center → point, normal → direction). Since the box never moves, the solver finds the single X that makes every mapped reading agree.

Procedure (Eye-in-Hand mode in the web UI):

1. Jog the arm so the camera sees the box → Capture. (Bursts 3 detections and takes a consensus.)
2. Repeat for ≥3 poses to solve, ≥12 for READY. Vary orientation aggressively — rotational diversity is what constrains X.
3. Watch RMS + leave-one-out stability, then Save → captures/cal_handeye_phaseA_<ts>.json:

{
  "camera_to_flange": { "translation_m": [x,y,z], "rpy_deg": [rx,ry,rz] },
  "quality": { "band": "Fair", "pos_rms_mm": 6.0, "dir_rms_deg": 1.05 },
  "n_samples": 14
}

Rotation is extrinsic XYZ, R = Rz·Ry·Rx, degrees; translation in metres (handeye/transforms.py). Unity must use the identical convention or the pick lands rotated.

Robot pose source. Read live from the Dobot over the Dashboard socket (handeye/dobot_client.py, port 29999, GetPose() — stdlib sockets, no SDK). Configure IP in config/dobot.json. Probe it standalone:

python -m handeye.dobot_client --watch      # poll the live TCP pose
python -m handeye.dobot_client --info       # robot mode + raw reply

Calibrate with Tool 0 / User 0 active so the reported pose is the flange. If a tool frame is set on the pendant, X silently absorbs that offset.

---

Configuration

All tuning lives in config/*.json — editable by hand or through the UI's Config panel.

File	Controls
carpet_params.json	Carpet crop, tilt, plane removal, furrow/curve/seam thresholds
carpet_params_conveyor.json · _fold · _seam	Alternate carpet profiles
box_params.json	Box crop + rim detection (also used for the calibration marker)
dobot.json	Dobot IP / Dashboard port
nyx650_config.json	Vzense NYX650 camera settings

Every config has a crop block (x/y/z bounds + tilt_*_deg). This is the single most important thing to get right: the crop must contain your object and exclude the table/belt, and the tilt levels the cloud. Most "no detection" problems are a wrong crop.

---

Camera notes (Vzense NYX650)

• Driven via ScepterSDK; set its path as camera.scepter_sdk_path in the config.
• Windows only in practice. The .so loads under WSL, but GigE discovery doesn't cross the WSL2 NAT — the camera is only reachable from Windows. Develop/test on saved .npy captures in WSL; run live capture on Windows.
• Cold start takes several seconds at low FPS; the servers warm up before the first grab.

---

Repo layout

carpet/ box/ handeye/ segmentation/   detection + calibration packages
tools/flow_viz/                       web UI (stdlib HTTP server + single-page JS)
config/                               tuning profiles (JSON)
captures/                             saved clouds (.npy/.ply), PNGs, calibrations
tests/                                pytest suite (143 tests)
docs/                                 perception reports + Unity integration guide
*_server.py                           production TCP servers (Unity-facing)
*_detection.py                        one-shot CLI / interactive tuner

---

Troubleshooting

Symptom	Cause / fix
SC_OPENED or Failed to open Vzense camera	Two processes want the camera. Close the other owner (Unity's Vzense driver, flow-viz, a stale bridge) or replug.
camera not ready (no frame yet)	NYX650 cold start — wait, it warms up over ~12 s.
"No detection" on a good-looking cloud	Crop is wrong, or plane.enabled is removing the object. Check the crop box in live view.
Detection works on captures but not live	Live gates (live_min_points, live_min_confidence) are tuned for a different scene.
Unity gets Bad magic / JSON parse errors	Two overlapping Detect() calls desynced the socket. Serialize requests.