HADOOP-19974. Publish CycloneDX 1.7 SBOMs that mark shaded dependencies

[ppkarwasz]

Description of PR

Every JAR module now attaches a CycloneDX 1.7 XML SBOM under the dist profile (the profile used to deploy artifacts to Maven Central, see HADOOP-18590).

CycloneDX 1.7 introduces an isExternal attribute that makes it possible to distinguish dependencies shaded inside a JAR from those merely referenced on the classpath — a distinction essential for vulnerability management. Since cyclonedx-maven-plugin does not yet support maven-shade-plugin (see CycloneDX/cyclonedx-maven-plugin#472), this PR post-processes the generated BOM with a Groovy script (dev-support/sbom/sbom.groovy, run by gmavenplus-plugin in the verify phase):

• Ordinary dependencies get an explicit isExternal="true"; dependencies embedded by maven-shade-plugin rely on the schema default of false.
• Relocated dependencies are technically derivative libraries, so they receive a pedigree ancestor carrying the original artifact's purl and SHA-256 hash. Both the relocated copy and its ancestor deliberately share the original's PURL, because vulnerability scanners cannot match a newly minted PURL against vulnerability databases and do not yet consult the pedigree.
• Dependencies embedded verbatim (modules shading without relocations) keep the original's hash and get a notes-only pedigree documenting the embedding.
• The shaded artifact set is derived from the effective maven-shade-plugin configuration, so it cannot drift from the shade artifactSet; -DskipShade builds classify everything as external, matching the jar actually produced.
• Hash lists are trimmed to SHA-256 and the result is re-validated against the CycloneDX 1.7 schema, failing the build on violations.

Out of scope (follow-ups): SBOMs for the binary distribution tarball, correcting license metadata, and a possible switch to JSON output (or publishing both formats).

How was this patch tested?

mvn clean install -Pdist -DskipTests on representative modules, asserting the generated target/bom.xml:

• hadoop-annotations (no shading): all components isExternal="true", no pedigree.
• hadoop-bos (include-style artifactSet with relocations): only the 3 compile-scope includes embedded, with pedigree ancestors carrying the SHA-256; provided-scope includes correctly reported external — verified against the actual jar contents.
• hadoop-tos (no artifactSet, no relocations): all 5 compile-scope dependencies embedded verbatim, hashes kept, notes-only pedigree.
• hadoop-client-runtime / hadoop-client-api (exclude-style with wildcards / include-style): all 63 externals match the shade excludes (incl. io.netty:* and org.glassfish.* wildcard families); 70 embedded components carry relocation pedigrees.
• -DskipShade and -Dcyclonedx.skip paths verified as no-ops.
• Schema validity is enforced by the script itself on every module.

For code changes:

• Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: Have the integration tests been executed and the endpoint
  declared according to the connector-specific documentation? Note: Automated CI
  testing doesn't cover all cases so manual testing with cloud storage is still
  required.
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? (build-time only: gmavenplus-plugin, org.apache.groovy:groovy and org.cyclonedx:cyclonedx-core-java are all Apache-2.0; nothing is bundled in released artifacts, so LICENSE-binary/NOTICE-binary are unaffected)
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

• The PR includes the phrase "Contains content generated by Claude Code"
• My use of AI contributions follows the ASF legal policy
  https://www.apache.org/legal/generative-tooling.html

[ppkarwasz]

For reviewers, here are three real entries taken from the generated BOMs (<description> and <externalReferences> elided for brevity).

1. External dependency — commons-logging in hadoop-client-runtime/target/bom.xml. It is on the shade artifactSet exclude list (left unshaded so downstream users can configure logging), so it gets an explicit isExternal="true" and keeps its hash:

<component type="library" bom-ref="pkg:maven/commons-logging/commons-logging@1.3.0?type=jar" isExternal="true">
  <publisher>The Apache Software Foundation</publisher>
  <group>commons-logging</group>
  <name>commons-logging</name>
  <version>1.3.0</version>
  <scope>required</scope>
  <hashes>
    <hash alg="SHA-256">66d3c980470b99b0c511dad3dfc0ae7b265ec1fb144e96bc0253a8a175fd34d9</hash>
  </hashes>
  <licenses>
    <license>
      <id>Apache-2.0</id>
      <url>https://www.apache.org/licenses/LICENSE-2.0</url>
    </license>
  </licenses>
  <purl>pkg:maven/commons-logging/commons-logging@1.3.0?type=jar</purl>
</component>

2. Non-relocated embedded dependency — httpclient5 in hadoop-tos/target/bom.xml. hadoop-tos shades without relocations, so the embedded classes are verbatim copies: the component omits isExternal (the schema default is false), keeps the original's hash, and carries a notes-only pedigree:

<component type="library" bom-ref="pkg:maven/org.apache.httpcomponents.client5/httpclient5@5.5?type=jar">
  <publisher>The Apache Software Foundation</publisher>
  <group>org.apache.httpcomponents.client5</group>
  <name>httpclient5</name>
  <version>5.5</version>
  <scope>required</scope>
  <hashes>
    <hash alg="SHA-256">496b4b0e8d5f3a8139a5d2638486d304758bac3a9c39d76989f663cfd9354fc9</hash>
  </hashes>
  <licenses>
    <license>
      <id>Apache-2.0</id>
    </license>
  </licenses>
  <purl>pkg:maven/org.apache.httpcomponents.client5/httpclient5@5.5?type=jar</purl>
  <pedigree>
    <notes>Embedded verbatim into this artifact by maven-shade-plugin, without package relocation; the hashes describe the original standalone artifact.</notes>
  </pedigree>
</component>

3. Relocated embedded dependency — commons-lang3 in hadoop-client-runtime/target/bom.xml. The embedded copy is a rewritten derivative (relocated to org.apache.hadoop.shaded), so the component carries no hash of its own; the SHA-256 of the original standalone artifact sits on the pedigree ancestor, which deliberately shares the component's PURL so vulnerability scanners can still match it:

<component type="library" bom-ref="pkg:maven/org.apache.commons/commons-lang3@3.20.0?type=jar">
  <publisher>The Apache Software Foundation</publisher>
  <group>org.apache.commons</group>
  <name>commons-lang3</name>
  <version>3.20.0</version>
  <scope>optional</scope>
  <licenses>
    <license>
      <id>Apache-2.0</id>
      <url>https://www.apache.org/licenses/LICENSE-2.0</url>
    </license>
  </licenses>
  <purl>pkg:maven/org.apache.commons/commons-lang3@3.20.0?type=jar</purl>
  <pedigree>
    <ancestors>
      <component type="library">
        <group>org.apache.commons</group>
        <name>commons-lang3</name>
        <version>3.20.0</version>
        <hashes>
          <hash alg="SHA-256">69e5c9fa35da7a51a5fd2099dfe56a2d8d32cf233e2f6d770e796146440263f4</hash>
        </hashes>
        <purl>pkg:maven/org.apache.commons/commons-lang3@3.20.0?type=jar</purl>
      </component>
    </ancestors>
    <notes>Relocated and embedded into this artifact by maven-shade-plugin; the ancestor component describes the original standalone artifact.</notes>
  </pedigree>
</component>

[steveloughran]

Looks useful. Let's see what yetus says.

[pan3793]

Can Java be used instead? I'm not sure how many people in the Hadoop community are familiar with Groovy.

[ppkarwasz]

Groovy is totally compatible with Java syntax, which is why I chose it. Should I rewrite the PR to only use Java methods?

[pan3793]

I don't understand why things do not happen in CycloneDX/cyclonedx-maven-plugin if it matters (I read the issue link, but it's unclear what the exact blocker is that makes it can NOT happen in upstream.

I do have concerns about introducing a new language to work around such an issue. If some committers think we MUST do it, please write it in full Java syntax (suppose Groovy syntax is a real superset of Java)

[ppkarwasz]

This will happen upstream (CycloneDX/cyclonedx-maven-plugin#696), but we are uncertain about the details.

Hadoop is not the project that contains the most shaded content, but is certainly most commonly used shaded dependency. Having one version of Hadoop release an SBOM with the wrong mechanism beats hundreds of projects populating Maven Central with imperfect SBOMs.

This is a proposal on how to improve HADOOP's SBOM now (@steveloughran was talking about an alternative XSLT-based solution), so other projects can start experimenting consuming SBOMs for HADOOP's libraries.

SBOMs for normal (non-shading) libraries are much less interesting and don't require reusing the upstream SBOM.

[steveloughran]

If this could be done with an XSL transform, it'd be a lot cleaner...the maven ant plugin can just invoke along with whatever conditional wrapping is needed. DOM manipulation is exactly what xsl is for, and it's a tool worth knowing for that reason