Skip to content

fix: ship unstripped kernel image and modules in the -dbg package - #74

Merged
Christopher Obbard (obbardc) merged 8 commits into
qcom/debian/latestfrom
wip/obbardc/fix-dbg-stripped
Aug 28, 2026
Merged

fix: ship unstripped kernel image and modules in the -dbg package#74
Christopher Obbard (obbardc) merged 8 commits into
qcom/debian/latestfrom
wip/obbardc/fix-dbg-stripped

Conversation

@obbardc

@obbardc Christopher Obbard (obbardc) commented Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The vmlinux in linux-image-<KVER>-dbg was unusable for crash/kdump/systemtap. Fix it.

Verification

  • resolute: check debug package is not stripped, check non-debug package is not stripped
  • trixie, qcom-next: check debug package is not stripped, check non-debug package is not stripped
  • trixie, qcom-next-debug: check debug package is not stripped, check non-debug package is not stripped
  • forky, qcom-next: check debug package is not stripped, check non-debug package is not stripped
  • forky, qcom-next-debug: check debug package is not stripped, check non-debug package is not stripped

@obbardc
Christopher Obbard (obbardc) force-pushed the wip/obbardc/fix-dbg-stripped branch 3 times, most recently from a142e01 to 5f2708f Compare August 27, 2026 18:31
github-advanced-security[bot]
github-advanced-security Bot found potential problems Aug 27, 2026
View reviewed changes
Comment thread .github/workflows/kernel-debusine-build.yml Fixed
Comment thread .github/workflows/kernel-debusine-build.yml Fixed
@obbardc

Christopher Obbard (obbardc) commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

appears to include the missing segments now ! :D

@obbardc
Christopher Obbard (obbardc) marked this pull request as ready for review August 28, 2026 08:04
@obbardc Christopher Obbard (obbardc) changed the title ship a usable vmlinux in the -dbg package fix: don't strip vmlinux in the -dbg package Aug 28, 2026
@obbardc Christopher Obbard (obbardc) changed the title fix: don't strip vmlinux in the -dbg package fix: ship unstripped kernel image and modules in the -dbg package Aug 28, 2026
@obbardc
Christopher Obbard (obbardc) marked this pull request as draft August 28, 2026 09:02
@obbardc
Christopher Obbard (obbardc) force-pushed the wip/obbardc/fix-dbg-stripped branch 2 times, most recently from 33e0fe2 to a2318ae Compare August 28, 2026 13:31
@obbardc
Christopher Obbard (obbardc) marked this pull request as ready for review August 28, 2026 14:44
Merged
@bjordiscollaku

Bjordis Collaku (bjordiscollaku) commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Thanks for pulling the dead debug profile in here, that covers my note on #83.

The last build on this branch failed on your own new guard: No DWARF in the extracted debug file for msm_kgsl.ko on the resolute leg (run 33175748736). All four Debian legs went green through Debusine, so kgsl picks up debug info fine there, but on Ubuntu it still comes out with no DWARF even with DIRECTIVE: STRIP="no". The guard is doing its job, the DKMS half just isn't there yet on Ubuntu. Current head has no build on it at all, so worth a rerun once that is sorted.

Related: #84 drops resolute because iris-vpu and camx fail to build, which is a different failure from this one. Dropping it would hide the kgsl case rather than fix it, keep them separate?

On dwarf-debug-info.config, what does it change in practice? arch/arm64/configs/defconfig already has CONFIG_DEBUG_KERNEL=y and CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y, and qcom.config already carries # CONFIG_DEBUG_INFO_REDUCED is not set, so as far as I can tell every line is already the effective value. Is it there to pin them against defconfig drift? Worth a line in the commit message if so.

Also this and #86 both rewrite the override_dh_auto_configure comment block and touch prepare-source.sh, so whichever lands second needs a rebase.

@obbardc
Christopher Obbard (obbardc) marked this pull request as draft August 28, 2026 16:54
@obbardc
packaging: drop unused debug build profile
922677b 
There is no longer any need to apply a separate configuration when
building the package with the debug build profile. Remove all mention
of it from the repository, along with the README sections and examples
describing them.

--profiles is kept as a generic DEB_BUILD_PROFILES pass-through; only
the debug-specific example goes.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
d/rules: keep dh_strip and dh_dwz away from the shipped debug symbols
59bdf0d 
dh_strip does not exempt /usr/lib/debug/; its only debug-path exclusions are
.../debug/.build-id/ directories and debug/*.so. The vmlinux staged into
linux-image-<KVER>-dbg is an ordinary ELF executable so dh_strip stripped it,
removing the DWARF and the symbol table. Nothing was saved first: the package
name already ends in -dbg, so dh_strip's automatic dbgsym extraction is
skipped. The result was a -dbg package useless for crash, kdump and systemtap.

dh_dwz, which runs immediately before, would also run dwz over that same
multi-gigabyte vmlinux.

Exclude by path rather than by package name. -X is a substring match against
the full path, so it covers everything the -dbg package ships and needs no
lookup of the kernel release to build the package name.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
d/rules: stop dh_strip generating automatic -dbgsym packages
1740421 
Debug symbols for this source package are shipped by the declared
linux-image-<KVER>-dbg package. The -dbgsym package debhelper generated
alongside it duplicated that role for the headers tree without being
declared or documented in debian/control.in.

Drop it from the documented outputs at the same time: a build now
publishes five binary packages per variant, all declared in
debian/control.in.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
d/rules: mirror the kernel/ path level in the module debug tree
bef6b98 
modules.order lists build-tree-relative paths, but modules_install puts
in-tree modules under $MODLIB/kernel/. The debug files were written without
that level, so the debug tree did not mirror the module tree.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
bundle-dkms-modules: don't strip modules after build
25c9454 
The DKMS modules in linux-image-<KVER>-dbg carried no debug information.
The extraction itself was fine: dkms had already stripped the module before
bundle-dkms-modules.sh ever saw it.

dkms strips by default. read_conf resolves an unset STRIP[index] to
${strip[0]:-yes} and build_module then runs:

    strip -g "$built_module"

which is --strip-debug: the same operation as Stage 3 here, but applied
before Stage 2 has extracted anything. objcopy --only-keep-debug therefore
copied out an ELF with no .debug_* sections, and the shipped module had no
debug information to lose either.

Pass --directive 'STRIP=no' to dkms build to stop stripping the modules
after building.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
bundle-dkms-modules: fail if a module contains no debug information
961cabb 
objcopy --only-keep-debug succeeds whether or not the module it reads
carries DWARF, so a module built without debug information yields a
debug file with no symbols and the -dbg package ships it. Assert that
the extracted file has a .debug_info section, and fail if it does not.

Match against readelf's captured output rather than piping it into
`grep -q`. grep -q exits at the first match, readelf dies of SIGPIPE on
its next write, and under this script's `set -o pipefail` the pipeline
reports that failure instead of grep's success - firing the assertion on
a file that does carry DWARF. Whether readelf gets far enough to be
killed depends on stdio flush timing against grep's startup, so the
pipeline form would fail only sometimes, which is worse than failing
always.

Read readelf's stderr into the same variable rather than discarding it,
so a readelf that cannot read the file reports why instead of being
indistinguishable from a file with no DWARF, and print the section list
when the assertion fires: that is the evidence needed to tell "built
without debug information" from "something stripped it".

Report a failing objcopy the same way. Falling back to a full copy keeps
the -dbg package usable when extraction fails, but it is a degradation
and should not happen silently.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
bundle-dkms-modules: report the build host's dkms configuration
88e62b8 
dkms hardcodes where it reads configuration from: /etc/dkms/framework.conf
and /etc/dkms/framework.conf.d/*.conf in read_framework_conf and the
/etc/dkms/<module>*.conf overrides in read_conf. Neither set can be
redirected by an option or an environment variable, so a build cannot opt
out of whatever the host ships; it can only be explicit about it.

Most of the exposure is already closed. framework.conf accepts a fixed
variable list; it is sourced before the command line is parsed, so
--dkmstree and --kernelsourcedir win over it; and --directive is
applied after every conf file. What remains is tmp_location,
parallel_jobs, the compress_*_opts and post_transaction, the last being
an arbitrary command dkms will run.

Print whatever config is present, once for the framework configuration
and again per module for the overrides that apply to it, so a
surprising host setting appears in the build log instead of acting
silently. Deliberately not fatal: a build host is allowed to have dkms
configured; this is diagnostic rather than a policy.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
bundle-dkms-modules: always print the dkms build log
0841cf3 
make.log was printed only when the build failed, and then only its last
300 lines. A module that builds is not necessarily a module built
correctly: the debug information check exists precisely because the
result can be wrong while dkms reports success, and when it fires the
compiler command lines are the first thing wanted. They were gone by
then - the private dkms tree is deleted on EXIT, so nothing survives the
build to look at.

Print make.log for every module, before the outcome is judged, so it is
in the log whether the build succeeded or not. debian/rules exports
KBUILD_VERBOSE, so the full compiler command lines are there.

A failed build still hard-fails, and the BUILD_EXCLUSIVE gate analysis
for the "dkms attempted no build" case is unchanged; the failure path now
refers to the log above rather than printing a tail of it a second time.

Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
@obbardc
Christopher Obbard (obbardc) marked this pull request as ready for review August 28, 2026 19:26
@obbardc

Christopher Obbard (obbardc) commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

I wrote a small script to compare two packages, it's useless to me now this PR has been satisfied, but attaching here in case it's useful in future:

compare-debug-artifacts-stripped.sh

All of the binaries built in CI appear to now have:

  • binaries stripped of debug info in the main package,
  • binaries with debug info included in the -dbg package.

Running the script above against the last CI run on this branch:

======================================================================
resolute
======================================================================
normal: ./resolute/linux-image-7.2.0-qcom-next-20260826-pr74_7.2.0+20260826-0qli~26.04.1~_arm64.deb
debug:  ./resolute/linux-image-7.2.0-qcom-next-20260826-pr74-dbg_7.2.0+20260826-0qli~26.04.1~_arm64.deb

--- kernel ---
kernel-build/resolute/image/boot/vmlinuz-7.2.0-qcom-next-20260826-pr74:                      Linux kernel ARM64 boot executable Image, little-endian, 4K pages
kernel-build/resolute/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-20260826-pr74/vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=efdc146ec9a97bc5213750a3d0b71ee7f7ed5dbe, with debug_info, not stripped

--- extra module ---
kernel-build/resolute/image/lib/modules/7.2.0-qcom-next-20260826-pr74/extra/msm_kgsl.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=244a1860eba24637db48b92850ecbde77927927f, not stripped
kernel-build/resolute/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-20260826-pr74/extra/msm_kgsl.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=244a1860eba24637db48b92850ecbde77927927f, with debug_info, not stripped

--- kernel module ---
kernel-build/resolute/image/lib/modules/7.2.0-qcom-next-20260826-pr74/kernel/sound/soundcore.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=038b9b4120c9d93470e813d8023a18ecbd0a46c0, not stripped
kernel-build/resolute/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-20260826-pr74/kernel/sound/soundcore.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=038b9b4120c9d93470e813d8023a18ecbd0a46c0, with debug_info, not stripped

======================================================================
qcom-next-debug-trixie
======================================================================
normal: ./qcom-next-debug-trixie/linux-image-7.2.0-qcom-next-debug-20260826-pr74_7.2.0+20260826-0qli~bpo13+1~_arm64.deb
debug:  ./qcom-next-debug-trixie/linux-image-7.2.0-qcom-next-debug-20260826-pr74-dbg_7.2.0+20260826-0qli~bpo13+1~_arm64.deb

--- kernel ---
kernel-build/qcom-next-debug-trixie/image/boot/vmlinuz-7.2.0-qcom-next-debug-20260826-pr74:                      Linux kernel ARM64 boot executable Image, little-endian, 4K pages
kernel-build/qcom-next-debug-trixie/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=803f9a6d5855df5f1add661f14ff8056e28b6aae, with debug_info, not stripped

--- extra module ---
kernel-build/qcom-next-debug-trixie/image/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/extra/msm_kgsl.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=d4949ff8ea08cccb031e17a919cb35c6d79cdbd1, not stripped
kernel-build/qcom-next-debug-trixie/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/extra/msm_kgsl.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=d4949ff8ea08cccb031e17a919cb35c6d79cdbd1, with debug_info, not stripped

--- kernel module ---
kernel-build/qcom-next-debug-trixie/image/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/kernel/sound/soundcore.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=41c8b4498b8df9b645b15fd912502a5bbc0662c2, not stripped
kernel-build/qcom-next-debug-trixie/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/kernel/sound/soundcore.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=41c8b4498b8df9b645b15fd912502a5bbc0662c2, with debug_info, not stripped

======================================================================
qcom-next-forky
======================================================================
normal: ./qcom-next-forky/linux-image-7.2.0-qcom-next-20260826-pr74_7.2.0+20260826-0qli~_arm64.deb
debug:  ./qcom-next-forky/linux-image-7.2.0-qcom-next-20260826-pr74-dbg_7.2.0+20260826-0qli~_arm64.deb

--- kernel ---
kernel-build/qcom-next-forky/image/boot/vmlinuz-7.2.0-qcom-next-20260826-pr74:                      Linux kernel ARM64 boot executable Image, little-endian, 4K pages
kernel-build/qcom-next-forky/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-20260826-pr74/vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=aa0a3ac662f7e463f0ec4b9d31a143025beddaeb, with debug_info, not stripped

--- extra module ---
kernel-build/qcom-next-forky/image/lib/modules/7.2.0-qcom-next-20260826-pr74/extra/msm_kgsl.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=8c55129b06a2c0a038dd2e8eca761ec049d3ad7a, not stripped
kernel-build/qcom-next-forky/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-20260826-pr74/extra/msm_kgsl.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=8c55129b06a2c0a038dd2e8eca761ec049d3ad7a, with debug_info, not stripped

--- kernel module ---
kernel-build/qcom-next-forky/image/lib/modules/7.2.0-qcom-next-20260826-pr74/kernel/sound/soundcore.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=b23e0571492879042a8cec074847ffc7365024a0, not stripped
kernel-build/qcom-next-forky/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-20260826-pr74/kernel/sound/soundcore.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=b23e0571492879042a8cec074847ffc7365024a0, with debug_info, not stripped

======================================================================
qcom-next-debug-forky
======================================================================
normal: ./qcom-next-debug-forky/linux-image-7.2.0-qcom-next-debug-20260826-pr74_7.2.0+20260826-0qli~_arm64.deb
debug:  ./qcom-next-debug-forky/linux-image-7.2.0-qcom-next-debug-20260826-pr74-dbg_7.2.0+20260826-0qli~_arm64.deb

--- kernel ---
kernel-build/qcom-next-debug-forky/image/boot/vmlinuz-7.2.0-qcom-next-debug-20260826-pr74:                      Linux kernel ARM64 boot executable Image, little-endian, 4K pages
kernel-build/qcom-next-debug-forky/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=f2d25260d6c1788975592458244156d603dc04b7, with debug_info, not stripped

--- extra module ---
kernel-build/qcom-next-debug-forky/image/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/extra/msm_kgsl.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=c14c4feabffa8e23e9abcd2e152dcc5d20dd526d, not stripped
kernel-build/qcom-next-debug-forky/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/extra/msm_kgsl.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=c14c4feabffa8e23e9abcd2e152dcc5d20dd526d, with debug_info, not stripped

--- kernel module ---
kernel-build/qcom-next-debug-forky/image/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/kernel/sound/soundcore.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=93dfd59086660b4cd0a2a254ffb78be307cf9b68, not stripped
kernel-build/qcom-next-debug-forky/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-debug-20260826-pr74/kernel/sound/soundcore.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=93dfd59086660b4cd0a2a254ffb78be307cf9b68, with debug_info, not stripped

======================================================================
qcom-next-trixie
======================================================================
normal: ./qcom-next-trixie/linux-image-7.2.0-qcom-next-20260826-pr74_7.2.0+20260826-0qli~bpo13+1~_arm64.deb
debug:  ./qcom-next-trixie/linux-image-7.2.0-qcom-next-20260826-pr74-dbg_7.2.0+20260826-0qli~bpo13+1~_arm64.deb

--- kernel ---
kernel-build/qcom-next-trixie/image/boot/vmlinuz-7.2.0-qcom-next-20260826-pr74:                      Linux kernel ARM64 boot executable Image, little-endian, 4K pages
kernel-build/qcom-next-trixie/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-20260826-pr74/vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=71b5b2c1e72e65b4b52ed9be8c51c910b2784ab5, with debug_info, not stripped

--- extra module ---
kernel-build/qcom-next-trixie/image/lib/modules/7.2.0-qcom-next-20260826-pr74/extra/msm_kgsl.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=bb9cd7068977d7ff40b694d13f76110ca82f823a, not stripped
kernel-build/qcom-next-trixie/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-20260826-pr74/extra/msm_kgsl.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=bb9cd7068977d7ff40b694d13f76110ca82f823a, with debug_info, not stripped

--- kernel module ---
kernel-build/qcom-next-trixie/image/lib/modules/7.2.0-qcom-next-20260826-pr74/kernel/sound/soundcore.ko:               ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=e2683a11219d36ae9126c56dfca34d96ab43394f, not stripped
kernel-build/qcom-next-trixie/debug/usr/lib/debug/lib/modules/7.2.0-qcom-next-20260826-pr74/kernel/sound/soundcore.ko: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), BuildID[sha1]=e2683a11219d36ae9126c56dfca34d96ab43394f, with debug_info, not stripped

@obbardc
Christopher Obbard (obbardc) merged commit 763f851 into qcom/debian/latest Aug 28, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bjordiscollaku Bjordis Collaku (bjordiscollaku) bjordiscollaku approved these changes
+1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@obbardc Christopher Obbard (obbardc)

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@obbardc @bjordiscollaku @github-advanced-security