sensing: add the SENSING SG10A GMSL camera plugin - #998
Draft
jiwenc-nv wants to merge 1 commit into
Draft
Conversation
camera_plugin_sensing captures the Astra S56C through libargus on an AGX Orin and either encodes H.264 on the Jetson V4L2 M2M engine -- Jetson ships no libnvidia-encode -- or serves frames to another process as CUDA memory, which camera_viz consumes as `type: cuda_ipc`. The shape follows the OAK plugin: repeatable --add-stream, raw Annex-B H.264 per sensor, per-frame metadata to MCAP or an OpenXR schema push. Bring-up is two-sided and neither half can do the other's job, hence setup_host.sh for the kernel modules, the device-tree overlay and the POC/PWM writes, and setup_container.sh for the Argus socket and build headers. The recurring failure they exist to stop: the vendor install.sh never copies the sensor .ko files into /lib/modules, so /dev/video* is empty after every reboot. Two traps fail misleadingly and are called out in code and docs. Legacy CUDA IPC returns cudaSuccess on Tegra and only breaks in the far process, so the transport uses the VMM API over SCM_RIGHTS. And GLVND picks the EGL vendor from DISPLAY, so losing the probe to Mesa surfaces as EGL_NO_STREAM_KHR out of Argus; main() clears DISPLAY before the first EGL call. 1080p60 sustained, ~1 ms producer to consumer on an AGX Orin. Setup, recording and live streaming are documented in docs/source/device/sensing.rst. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| if self._sock is not None: | ||
| try: | ||
| self._sock.close() | ||
| except OSError: |
Collaborator
Author
|
/preview-docs |
Contributor
|
✅ Preview deployed: https://NVIDIA.github.io/IsaacTeleop/preview/pr-998/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
New C++ plugin
camera_plugin_sensingfor the SENSING SG10A-AGON-G2M-A1 GMSL carrier on a Jetson AGX Orin (JetPack 6.2 / L4T R36.4.3). It captures the Astra S56C through libargus — the sensor is 10-bit Bayer, so it has to go through the ISP — and either encodes H.264 on the Jetson V4L2 M2M engine or serves frames to another process as CUDA device memory. camera_viz gains a matchingtype: cuda_ipcsource: no encode, no host copy, ~1 ms producer-to-consumer at 1080p60. Also here: host/container setup scripts (bring-up is two-sided — kernel modules and the device-tree overlay only exist on the host, the Argus socket and build headers only matter in the container) and a newdocs/source/device/sensing.rst.Two things not to "simplify", both commented in-tree. Legacy CUDA IPC returns
cudaSuccesson Tegra and only fails in the far process, so the transport uses the VMM API overSCM_RIGHTS. And GLVND picks the EGL vendor fromDISPLAY; losing that probe to Mesa surfaces asEGL_NO_STREAM_KHRout of Argus, somain()clearsDISPLAYbefore the first EGL call.SHF3L/SHF3H modules on the same carrier are out of scope — they are ISP-processed already, and the stock
type: v4l2source reads them.Type of change
Testing
AGX Orin, JetPack 6.2 / L4T R36.4.3, SG10A carrier with one Astra S56C:
camera_plugin_sensing --add-stream=sensor=2,ipc=/tmp/sensing2.sockcaptures 1920x1080 through Argus into CUDA;camera_viz.sh run configs/cuda_ipc.yamlrenders it. 60 fps sustained, ~1 ms producer to consumer.ffplay -f h264and withnvv4l2decoderon the Jetson.sensing_ipc_testsrcplusexamples/camera_viz/tests/test_cuda_ipc_source.pycover the IPC protocol with no camera attached (the test pattern stamps its frame number into the pixels, so a stale or torn slot fails the assert).make current-docs(sphinx-build -W --keep-going) builds clean.SKIP=check-copyright-year pre-commit run --all-filespasses.Checklist
SKIP=check-copyright-year pre-commit run --all-filesgit commit -s) per the DCO