Add NVIDIA nsys GPU capture for adhoc flamegraphs

[prashantbytesyntax]

What

Capture NVIDIA CUDA kernel activity via host-installed Nsight Systems (nsys) and upload the result through the existing adhoc path, so GPU profiles appear in Performance Studio's Adhoc Profiling view next to CPU profiles. Two views from the same capture: a GPU flamegraph (default) and an opt-in CPU/GPU timeline with optional click-for-stack backtraces.

This is the agent-side half; the Studio UI + sandbox demo are in a companion PR on gprofiler-performance-studio (#86).

Changes

• gprofiler/nsys_profiler.py — host-detect nsys (PATH / NSYS_PATH / common install dirs), run a capture, export cuda_gpu_kern_sum (fallback cuda_api_sum), convert the stats CSV to collapsed stacks, and render flamegraph HTML (template when the full agent runtime is present, a simple self-contained fallback otherwise). Also houses the timeline exports/rendering described below. nsys is detected, never bundled (large and NVIDIA-licensed) — mirrors the PerfSpect approach.
• gprofiler/main.py — --enable-nsys / --nsys-path / --nsys-workload / --nsys-timeline / --nsys-timeline-stacks CLI args; run the GPU capture on a background thread in parallel with CPU profilers and prefer the GPU HTML for the adhoc upload when a capture succeeds.
• gprofiler/dynamic_profiling_management/__init__.py — honor enable_nsys / nsys_path / nsys_workload / nsys_timeline / nsys_timeline_stacks from combined_config (PerfSpect-shaped control plane) and tag perf_events with nsys-cuda (plus nsys-timeline / nsys-stacks when those modes are active) so the Adhoc UI can show a GPU / nsys chip and distinguish the view.
• scripts/prepare_centos.sh — add sqlite-devel so the executable build's Python has _sqlite3 (the timeline-stacks SQLite export needs it; the agent degrades gracefully — timeline without backtraces plus a warning — if it's still missing at runtime).
• tests/test_nsys_profiler.py — unit coverage for CSV→collapsed, nsys discovery, timeline trace parsing/rendering, and callchain extraction. No GPU/nsys required.
• docs/NSYS_GPU_PROFILING.md — enabling, packaging, sandbox topology.

Timeline view (opt-in)

--nsys-timeline (CLI) or combined_config.nsys_timeline (heartbeat) exports cuda_gpu_trace + cuda_api_trace from the same capture and uploads a self-contained canvas timeline HTML instead of the flamegraph:

• Swim lanes per CPU thread (CUDA API calls) and GPU device/stream (kernels/memcpy); clicking an event highlights its correlation ID, linking each cudaLaunchKernel to the kernel it launched.
• Navigable at scale: opens auto-zoomed to a readable window, an overview minimap strip shows full-capture CPU/GPU density with a draggable viewport box, keyboard nav (arrows pan, +/- zoom, n/p step through events, 0 = full span). At low zoom, sub-pixel events shade lanes by occupancy instead of collapsing into solid bars.
• Falls back to the flamegraph when the trace export yields no events; captures over 20k events keep the longest ones to bound upload size.

--nsys-timeline-stacks / combined_config.nsys_timeline_stacks additionally captures with --cudabacktrace=kernel -s process-tree -b dwarf (heavier, so opt-in), exports the report to SQLite, and joins RUNTIME.callchainId → CUDA_CALLCHAINS → StringIds (stdlib sqlite3, no new deps) into a deduped stacks table:

• Clicking an event opens a panel with the launching CPU backtrace; GPU kernels resolve their launch stack through the correlation ID. Stacks are native frames (e.g. libtorch/ATen), not Python lines.
• An aggregate launch-stack flamegraph renders below the timeline (frame width = summed event duration per call path, preferring GPU kernel time; click to zoom subtrees), answering "which call paths cost the most GPU time overall".

nsys vs iaprof

The implemented backend is NVIDIA nsys. There is no iaprof code in this PR — Intel iaprof (Xe/Battlemage EU-stall GPU flamegraphs) doesn't run on NVIDIA hardware and is documented as a possible later Phase-5 backend behind the same GPU-profiler control, not something integrated here.

Testing

• python3 -m pytest tests/test_nsys_profiler.py → 27 passed (run with --noconftest in a no-Docker env; the repo conftest.py imports the docker module).
• Offline CSV→collapsed→HTML path verified by hand on the sample kernel CSV.
• Timeline HTML interactions (zoom/pan, minimap jump, keyboard nav, CorrID highlight, stack panel, flamegraph zoom/reset, occupancy shading) verified in headless Chromium against synthetic and real captures.
• End-to-end on an A10G GPU host: timeline capture (3850 events) and timeline+stacks capture (5532 events, kernel clicks resolving full launch backtraces through CorrID) both verified through the sandbox upload path. The stacks run is what surfaced the missing _sqlite3 in the PyInstaller bundle, fixed here.

🤖 Generated with Claude Code