Replace mmsemseg with SAM 3 for object segmentation - #1220
Open
Miliya-Ai wants to merge 9 commits into
Open
Conversation
…s to SAM itself (just adding a coordinate scaling parameter). The major involves creating a preprocessor that calls SAM and renaming the LLM object detection module before fixing all downstream dependencies
- object-segmentation.yml: push trigger listed the wrong branch name (object-segmentation instead of sam-segmentation), so pushes to this branch wouldn't build a test image. - Bump schemas submodule to add the objectID field the new object-segmentation preprocessor already emits.
Base docker-compose.yml now runs object-segmentation CPU-only, so unicorn stays CPU-only by default. prod-docker-compose.yml adds the nvidia device reservation for pegasus, which has GPU to spare. Verified with `docker compose -f docker-compose.yml config` (no GPU block) and `-f docker-compose.yml -f prod-docker-compose.yml config` (GPU block present).
sam3.pt is gated on Hugging Face (facebook/sam3), so the model can't be mirrored to our own server and wget'd like sam2.1 was. Instead the CI build now pulls it directly at build time using an HF_TOKEN build secret (BuildKit --secret, not ARG, so the token never lands in the image's layer history) from an account with approved access. - ultralytics bumped to 8.4.137 (SAM 3 support landed in 8.3.237) - Dockerfile downloads sam3.pt via huggingface-cli instead of wget - object-segmentation.yml passes HF_TOKEN through to the build step - SAM_MODEL_PATH default updated; multistage-diagram-segmentation is untouched and stays on SAM 2.1, out of scope for this change Requires an HF_TOKEN repository secret to be added on GitHub before this will build.
Caught by a local docker build test: current huggingface_hub ships `hf` as the CLI entry point and `huggingface-cli` now hard-fails instead of just warning. Verified the `hf download` invocation reaches HF's servers correctly with a throwaway token (got a clean "requires approval" response, confirming syntax + secret mount + network path all work).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you've followed the checklist and provide all the required information before requesting a review.
If you do not have everything applicable to your PR, it will not be reviewed!
If you don't know what something is or if it applies to you, ask!
Please note that PRs from external contributors who have not agreed to our Contributor License Agreement will not be considered.
To accept it, include
I agree to the [current Contributor License Agreement](/CLA.md)in this pull request.Don't delete below this line.
Required Information
Coding/Commit Requirements
New Component Checklist (mandatory for new microservices)
docker-compose.ymlandbuild.yml..github/workflows.README.mdfile that describes what the component does and what it depends on (other microservices, ML models, etc.).OR
Tracking issue
Closes #1152
Summary
Adds a new
object-segmentationpreprocessor that uses SAM 3 (via Ultralytics) to segment objects already found byobject-detection-llm, producing precise per-object polygon outlines instead of just bounding boxes.mmsemseg/semantic-segmentationis scoped down to only the background/environmental classes (sky, walls, floors, etc.) that an object detector can't see - object-detection-llm + object-segmentation now own everything else.photo-tactile-svgis updated to draw SAM 3's precise outlines in place of bounding-box rectangles wherever they're available.What's in this PR
preprocessors/object-segmentation/service (SAM 3, gatedfacebook/sam3weights pulled at build time via anHF_TOKENbuild secret). Output is schema-compatible with the existingsemanticSegmentationpreprocessor, with an addedobjectIDfield tying each segment back to the object it came fromdocker-compose.yml/prod-docker-compose.yml(CPU by default, GPU opt-in on pegasus)handlers/photo-tactile-svg/tactile_svg.py: now consumesobject-segmentationas an optional dependency and draws its polygon outline for a detected object in place of a plain bounding-box rectangle whenever one is available, falling back to the rectangle per-object otherwisephoto-audio-handlerupdated to read eitherobjectDetectionorobjectDetectionLLMvia a shared coalescing helper (utils/object_detection)object-segmentation(lint + build/push)