Vendor Harness Core and add Phase B ESP32 OCR handoff

demo_rokid_phase_b_harness.py

@@ -0,0 +1,7 @@
+"""Stable entry point for the Rokid Phase B Harness runtime."""
+
+from extensions.assistive_harness.phase_b.rokid_runtime import main
+
+
+if __name__ == "__main__":
+    main()

extensions/__init__.py

@@ -0,0 +1 @@
+"""OpenGlass extension packages."""

extensions/assistive_harness/.gitignore

@@ -0,0 +1,4 @@
+__pycache__/
+*.pyc
+runs/
+test_artifacts/

extensions/assistive_harness/OPENGLASS_MIGRATION_MAP.md

@@ -0,0 +1,25 @@
+# OpenGlass Migration Map
+
+Phase A deliberately separates portable policy from browser-specific transport.
+
+| Phase A component | OpenGlass-side destination |
+|---|---|
+| `schemas.py` | shared structured control-event schema |
+| `router.py` | local deterministic voice-command router |
+| `registry.py` + prompts | configurable Skill registry/prompt bundle |
+| `echo_guard.py` | output/playback-aware ASR echo filter |
+| `state_machine.py` | device-independent control priority and dedupe |
+| `asr/` | phone/host local ASR service behind the same interface |
+| `cv/base.py` | future advisory perception provider interface |
+| `server.py` WS messages | replaceable host/phone transport adapter |
+| `browser-session-adapter.js` | reference semantics for an OpenGlass session adapter |
+
+OpenGlass migration should keep `ControlEvent`, `SkillRegistry`, prompt SHA,
+generation fence and STOP priority stable. Replace only AudioMirror capture,
+session lifecycle calls, and the control transport. No dependency on ESP32, Rokid,
+DOM layout or the MiniCPM private wire format exists in the Python core.
+
+The first Rokid implementation of this boundary now lives in
+[`phase_b/`](phase_b/README.md). It consumes the existing APK JPEG/PCM endpoints,
+reuses this Core and the existing Gateway, and implements the browser adapter's
+STOP/RESUME/RESET/Skill generation contract in Python.

extensions/assistive_harness/README.md

@@ -0,0 +1,79 @@
+# Assistive Voice Skill Harness (Phase A / Phase B Core)
+
+This directory is the transport-neutral control Core shared by the optional
+MiniCPM-o browser sidecar and OpenGlass Phase B device adapters. It is disabled
+by default and does not replace the native microphone/video path.
+
+## Clean-clone setup
+
+From the OpenGlass repository root, install the Core and explicitly download
+the public ASR model:
+
+```powershell
+python -m pip install -r extensions/assistive_harness/requirements.txt
+python -m extensions.assistive_harness.download_modelscope_model
+```
+
+The second command prints the downloaded local directory and a complete start
+command. Model weights are not stored in this Git repository. The runtime never
+downloads a model implicitly: `--model-path` must point to an existing local
+directory, otherwise startup fails clearly.
+
+Start the sidecar explicitly:
+
+```powershell
+python -m extensions.assistive_harness.server --enabled `
+  --model-path "C:\path\to\a\local\FunASR\model"
+```
+
+Then opt the browser tab in with `?assistive_harness=1`. Browser integration
+assets and their hook contract live in
+[`../../integrations/minicpm_browser/`](../../integrations/minicpm_browser/).
+Test-only transcript
+injection additionally requires `--allow-test-injection`; it is never enabled by
+the normal command above.
+
+The browser hook is deliberately thin. Control, ASR, prompt registry, echo guard,
+state machine, metrics, and CV shadow interfaces live under this directory so the
+module can later be moved behind another transport (for example OpenGlass).
+
+The first Rokid/OpenGlass transport is implemented in
+[`phase_b/`](phase_b/README.md). It preserves this Core and the existing 8040
+Gateway while replacing browser microphone, camera, playback, and Session calls
+with a Python device adapter.
+
+## User-editable Skills
+
+System prompts live in `extensions/assistive_harness/prompts/`. Existing prompt
+files are read on every activation, so users can replace a prompt and activate
+the Skill again without restarting the sidecar. Skill IDs, enable flags and
+voice phrases are configured in `config/skills.example.yaml`; changing that YAML
+does require restarting the sidecar.
+
+Enabled by default:
+
+- `帮我找<物体>` -> `find_object` -> hot Session restart with `{{target}}`
+- `读一下` / `帮我识字` -> `read_text` -> hot Session restart
+- `描述一下` / `看看周围` -> `describe_scene` -> hot Session restart
+- `帮我避障` / `前面有障碍吗` -> `obstacle_avoidance` -> hot Session restart
+- `回到聊天` / `恢复普通聊天` -> `idle_chat` -> hot Session restart
+
+`obstacle_avoidance` contains the frozen AAAI_SI prompt and is enabled only for
+stationary, supervised validation. Its mobility safety has not been accepted;
+never treat this Demo as a navigation or safety device.
+
+With the sidecar running, `GET http://127.0.0.1:8021/skills` reports the active
+configuration and absolute prompt path for every registered Skill.
+
+## CV V1 shadow pipeline
+
+Browser JPEG mirrors enter a capacity-one latest-frame queue and are analyzed
+on a dedicated single-thread worker. `cv_mode: disabled` drops frames before
+inference; `cv_mode: shadow` writes `CVObservation` records without changing a
+Skill or taking ownership of MiniCPM output. Slow, timed-out, or failed plugins
+remain outside the audio/control receive path.
+
+`find_object` uses the local `yolo_onnx` reference provider. Other Skills keep
+the `noop` provider until a task-specific plugin is registered. Provider setup,
+the observation schema, and an OCR implementation template are documented in
+[`cv/README.md`](cv/README.md).

extensions/assistive_harness/__init__.py

@@ -0,0 +1,13 @@
+"""Portable voice-controlled Skill Harness core for Phase A."""
+
+from .router import RuleIntentRouter
+from .registry import SkillRegistry
+from .schemas import ASREvent, ControlEvent, ControlIntent
+
+__all__ = [
+    "ASREvent",
+    "ControlEvent",
+    "ControlIntent",
+    "RuleIntentRouter",
+    "SkillRegistry",
+]

extensions/assistive_harness/asr/__init__.py

@@ -0,0 +1,5 @@
+from .base import ASREngine, ASRResult
+from .energy_vad import EnergyVAD, UtteranceAudio
+from .funasr_engine import FunASREngine
+
+__all__ = ["ASREngine", "ASRResult", "EnergyVAD", "FunASREngine", "UtteranceAudio"]

extensions/assistive_harness/asr/base.py

@@ -0,0 +1,29 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Protocol
+
+import numpy as np
+
+
+@dataclass(frozen=True, slots=True)
+class ASRResult:
+    text: str
+    confidence: float
+    model: str
+    device: str
+
+
+class ASREngine(Protocol):
+    def transcribe(self, audio: np.ndarray, sample_rate: int) -> ASRResult:
+        ...
+
+
+class ScriptedASREngine:
+    def __init__(self, transcripts: list[str]):
+        self.transcripts = list(transcripts)
+
+    def transcribe(self, audio: np.ndarray, sample_rate: int) -> ASRResult:
+        del audio, sample_rate
+        text = self.transcripts.pop(0) if self.transcripts else ""
+        return ASRResult(text=text, confidence=1.0, model="scripted", device="cpu")

extensions/assistive_harness/asr/energy_vad.py

@@ -0,0 +1,91 @@
+from __future__ import annotations
+
+from collections import deque
+from dataclasses import dataclass
+
+import numpy as np
+
+
+@dataclass(frozen=True, slots=True)
+class UtteranceAudio:
+    audio: np.ndarray
+    started_at_ms: float
+    ended_at_ms: float
+
+
+class EnergyVAD:
+    """Small streaming endpoint detector; it does not perform recognition."""
+
+    def __init__(
+        self,
+        sample_rate: int = 16_000,
+        rms_threshold: float = 0.012,
+        min_speech_ms: int = 180,
+        end_silence_ms: int = 450,
+        max_utterance_ms: int = 8_000,
+        preroll_ms: int = 200,
+    ):
+        self.sample_rate = sample_rate
+        self.rms_threshold = rms_threshold
+        self.min_speech_ms = min_speech_ms
+        self.end_silence_ms = end_silence_ms
+        self.max_utterance_ms = max_utterance_ms
+        self.preroll_ms = preroll_ms
+        self._preroll: deque[tuple[np.ndarray, float]] = deque()
+        self._active: list[np.ndarray] = []
+        self._started_at_ms: float | None = None
+        self._last_voice_ms: float | None = None
+
+    def _duration_ms(self, audio: np.ndarray) -> float:
+        return float(audio.size) * 1000.0 / self.sample_rate
+
+    def feed(self, audio: np.ndarray, frame_started_at_ms: float) -> UtteranceAudio | None:
+        frame = np.asarray(audio, dtype=np.float32).reshape(-1).copy()
+        if frame.size == 0:
+            return None
+        duration_ms = self._duration_ms(frame)
+        frame_end_ms = frame_started_at_ms + duration_ms
+        rms = float(np.sqrt(np.mean(np.square(frame, dtype=np.float64))))
+        voiced = rms >= self.rms_threshold
+
+        if self._started_at_ms is None:
+            self._preroll.append((frame, frame_started_at_ms))
+            while self._preroll and frame_end_ms - self._preroll[0][1] > self.preroll_ms:
+                self._preroll.popleft()
+            if not voiced:
+                return None
+            self._started_at_ms = self._preroll[0][1] if self._preroll else frame_started_at_ms
+            self._active = [item[0] for item in self._preroll]
+            self._preroll.clear()
+            self._last_voice_ms = frame_end_ms
+        else:
+            self._active.append(frame)
+            if voiced:
+                self._last_voice_ms = frame_end_ms
+
+        active_ms = frame_end_ms - float(self._started_at_ms)
+        silence_ms = frame_end_ms - float(self._last_voice_ms or frame_end_ms)
+        if active_ms >= self.max_utterance_ms or silence_ms >= self.end_silence_ms:
+            return self._finish(frame_end_ms)
+        return None
+
+    def _finish(self, ended_at_ms: float) -> UtteranceAudio | None:
+        if self._started_at_ms is None or not self._active:
+            self.reset()
+            return None
+        audio = np.concatenate(self._active).astype(np.float32, copy=False)
+        started = self._started_at_ms
+        voiced_duration = max(0.0, float(self._last_voice_ms or ended_at_ms) - started)
+        self.reset()
+        if voiced_duration < self.min_speech_ms:
+            return None
+        return UtteranceAudio(audio=audio, started_at_ms=started, ended_at_ms=ended_at_ms)
+
+    def flush(self, ended_at_ms: float) -> UtteranceAudio | None:
+        return self._finish(ended_at_ms)
+
+    def reset(self) -> None:
+        self._preroll.clear()
+        self._active = []
+        self._started_at_ms = None
+        self._last_voice_ms = None

extensions/assistive_harness/asr/funasr_engine.py

@@ -0,0 +1,84 @@
+from __future__ import annotations
+
+import threading
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from .base import ASRResult
+
+
+class FunASREngine:
+    """Lazy local FunASR adapter. It never downloads a model implicitly."""
+
+    def __init__(self, model_path: str, device: str = "cpu", model_kwargs: dict[str, Any] | None = None):
+        path = Path(model_path).expanduser().resolve()
+        if not path.is_dir():
+            raise FileNotFoundError(f"FunASR model path does not exist: {path}")
+        self.model_path = str(path)
+        self.device = device
+        self.model_kwargs = dict(model_kwargs or {})
+        self._model: Any = None
+        self._load_lock = threading.Lock()
+        self._infer_lock = threading.Lock()
+
+    @property
+    def loaded(self) -> bool:
+        return self._model is not None
+
+    def ensure_loaded(self) -> None:
+        if self._model is not None:
+            return
+        with self._load_lock:
+            if self._model is not None:
+                return
+            from funasr import AutoModel
+
+            self._model = AutoModel(
+                model=self.model_path,
+                device=self.device,
+                disable_update=True,
+                **self.model_kwargs,
+            )
+
+    def warm_up(self) -> None:
+        """Load the model and run one short silent inference before serving clients."""
+        self.ensure_loaded()
+        silence = np.zeros(8_000, dtype=np.float32)
+        with self._infer_lock:
+            self._model.generate(
+                input=silence,
+                cache={},
+                is_final=True,
+                batch_size_s=0,
+            )
+
+    def transcribe(self, audio: np.ndarray, sample_rate: int) -> ASRResult:
+        if sample_rate != 16_000:
+            raise ValueError(f"FunASR Phase A requires 16 kHz audio, got {sample_rate}")
+        self.ensure_loaded()
+        waveform = np.asarray(audio, dtype=np.float32).reshape(-1)
+        with self._infer_lock:
+            results = self._model.generate(
+                input=waveform,
+                cache={},
+                is_final=True,
+                batch_size_s=0,
+            )
+        text = ""
+        confidence = 1.0
+        if isinstance(results, list) and results:
+            first = results[0]
+            if isinstance(first, dict):
+                text = str(first.get("text") or "").strip()
+                if isinstance(first.get("confidence"), (int, float)):
+                    confidence = float(first["confidence"])
+            else:
+                text = str(first).strip()
+        return ASRResult(
+            text=text,
+            confidence=confidence,
+            model=self.model_path,
+            device=self.device,
+        )