[700] Bundle license texts for all non-Apache-2.0 shaded dependencies

[rangareddy]

What

LICENSE-bundled says "See licenses/ for text of these licenses", but many bundled non-Apache-2.0 dependencies shipped no text. This adds the missing texts and a check that keeps them correct.

Closes #700. Supersedes #866 and #885.

Approach

The check reads the built *-bundled.jar rather than the poms, because the artifact is the only thing that settles what is actually bundled — and it is what a release reviewer inspects.

release/scripts/validate_bundled_license_texts.py takes the exact coordinates Maven resolved from dependency:tree, keeps the ones the shade plugin bundles, and confirms each is present by checking that every class it declares is in the jar. Nothing is sampled or thresholded: with the resolved version in hand, 207 of 213 dependencies match exactly and the other 6 differ only by module-info.class, which shade cannot merge.

Each dependency's license comes from its own pom, walking the parent chain, and from the license file it ships when the chain declares none. Anything not Apache-2.0 must have META-INF/licenses/LICENSE-<artifactId>, and the committed text is checked against the declared license so a text for the wrong license, or the wrong version of one, is caught. It fails in the other direction too, so a text with no bundled dependency behind it is reported (#865).

It runs in mvn-license-check after a package of the shaded modules.

What this corrects

• javax.transaction:jta and transaction-api are CDDL 1.0, not Apache-2.0. Both ship that text in their own jars; they were previously recorded as Apache-2.0 and shipped no text at all. Both now ship the text their jars carry.
• Seven texts removed for dependencies that are not in the jar: activation, asm, asm-commons, asm-tree, commons-el, mail, oro.
• xml-apis and stax-api removed: both declare Apache-2.0, which the banner at the top of LICENSE-bundled already covers.

Result: xtable-aws 5 texts, xtable-hive-metastore 22, xtable-hudi-support-extensions 3.

Known gaps

KNOWN_TEXT_MISMATCHES records two texts that cannot be corrected from anything the artifacts carry. jamon-runtime declares MPL 1.1 through jamon-project-1.0.2 and the committed text is MPL 2.0. javolution 5.5.1 declares BSD at a dead URL and the committed text is the project's later MIT relicense. Both dependencies are genuinely bundled, so the texts cannot be dropped either. The jamon-runtime mismatch predates this PR and is worth its own issue.

Two things this deliberately does not do. It does not regenerate LICENSE-bundled, which is stale in ways tracked separately — jersey is listed at 1.9 where 1.19 resolves, junit at 4.11 where 4.12 does. And it does not cover third-party code compiled inside a bundled dependency: groovyjarjarantlr and groovyjarjarasm come from groovy-all and attribute to it, so 308 classes are not separately listed. That is the listing-completeness half of #700 and needs a different check.

Testing

Run on a clean build of all three shaded modules:

• apache-rat:check — pass
• spotless:check — pass
• validate_shaded_license_coverage.sh — pass
• validate_bundled_license_texts.py — exit 0, 5 / 22 / 3 dependencies, no missing, mismatched or orphaned texts

Emptying KNOWN_TEXT_MISMATCHES flips the run to exit 1 with both entries reported, so the allowlist is load-bearing rather than decorative.

Note on authorship

The curated license texts are @rangareddy's work and are the part of this that has to happen whichever mechanism produces the metadata. The later commits replace the pom-based generator rework with the jar-based check described above.

[vinishjail97]

@rangareddy I have ran the scripts in my local and it was breaking for few dependencies. Can you address the comments? I will take a look tomorrow again and merge for 0.4.0-incubating rc2.

[rangareddy]

@vinishjail97 all ten threads are answered inline and the branch is force-pushed (efbffc9 → 2f9ed96). The PR description is rewritten, since most of it no longer matched the diff. Summary of the round:

Your two direct questions

• Is the hadoop override required? No — you were right, it is deleted. It took closing two POM-parsing gaps to make that true, and neither was a ~/.m2 miss: hadoop-project-3.1.0.pom contains <Xlint:-unchecked/>, which expat rejects outright, and hadoop-yarn-client-2.7.1.pom binds the Maven namespace to a prefix so every element is unqualified and the namespaced XPaths matched nothing. Both POMs are authentic — their SHA-1s match Central. Auditing the rest of the table on top of that removed 18 overrides in total.
• Will errors="strict" break this today? It would have, on javax.activation:activation:1.1 and javax.mail:mail:1.4.1 (both latin-1 META-INF/LICENSE.txt), out of 363 embedded entries. After the classified-artifact fix neither is bundled any more, so the current run has zero decode failures. notice_text_for had none, so the pre-existing errors="replace" there was harmless — it is strict now too.

Things the review surfaced that turned out to be bugs, not gaps

• The parts[3] bug was not only aborting the run. Nine classified coordinates were being written into LICENSE-bundled with the classifier in place of the version, and six ARTIFACT_OVERRIDES existed solely to stop the resulting raise. hbase-common:jar:tests was also being counted as a bundled runtime dependency.
• Your family/text assertion caught a real mis-attribution: junit was shipping the Common Public License, but 4.12 is EPL 1.0. Your note about 4.11 bundling Hamcrest's BSD text is correct, but 4.12 ships its own correct EPL text under LICENSE-junit.txt. So the override was wrong; it is removed and the committed text corrected. That also means the original commit message's justification for override-first was wrong for the version we bundle, so I rewrote it rather than let it merge.

One I found while running your scripts, unrelated to any thread

validate_shaded_license_coverage.sh was a fail-open. It builds the module list with rg, which mvn-license-check.yml never installs; on a machine without ripgrep the list came back empty and the script printed "No modules with maven-shade-plugin were found" and exited 0. The whole release license gate passed while checking nothing — which is also why your local run and mine could differ. Now find/grep, and an empty list fails.

Two things I deliberately did not decide

1. javolution:javolution:5.5.1 — POM says BSD License, the curated text is MIT with a 2017 copyright for a 2010 artifact, and the jar has no license file. It predates this PR (came in with [678] Add LICENSE for xtable-hive-metastore module #691). Writing a BSD text with a copyright line I cannot verify would be fabricating a legal document, so the generator refuses on it. Needs your call — details in the inline thread.
2. xtable-hive-metastore's shade <includes> has drifted and the validator's pre-existing includes-vs-tree check fails on this branch and on main (this PR touches no pom.xml): 25 includes name dependencies no longer in the tree, 97 runtime dependencies are missing from the list. It reads like a list written for Hive 2.x and never updated for 3.1.3. This is why I left the coordinate lists unchanged — regenerating now would flip a currently-passing check to failing, because the committed metadata still carries those 25 dead entries. Happy to file it as its own issue and do the regeneration as a follow-up.

Also: I'd flag javax.servlet and javax.servlet.jsp, whose overrides claim Apache-2.0 while their POMs say CDDL + GPLv2 w/ CPE. That downgrades Category B to A and suppresses license texts those dependencies would otherwise ship. I left them alone since it is a legal call, but it deserves a second opinion.

Not rebased onto main yet (8 commits, none touching these files) — say the word and I'll rebase and re-run before you look.

[rangareddy]

@vinishjail97 the License Check failure on the previous push was the new gate working, and it turned up something worth knowing before rc2. Pushed a611ada; CI should be green now.

The License Check has never actually run in CI.

The script builds its module list with rg, which mvn-license-check.yml does not install and which the GitHub ubuntu-latest runner does not ship. The call sits inside a process substitution, so set -o errexit cannot see it fail; the list came back empty and the script hit its exit 0 path. From the most recent License Check on main:

release/scripts/validate_shaded_license_coverage.sh: line 92: rg: command not found
No modules with maven-shade-plugin were found.

Every one of the last ten runs on main is green for that reason. I need to correct myself here — in my earlier reply I said the runner happened to have ripgrep and the gate "works in CI only by luck". That was wrong: it has never run.

What that was hiding

Replacing rg with find/grep made the gate run for the first time, and it immediately failed on xtable-hive-metastore's existing <includes>-vs-runtime-tree assertion: 25 includes name dependencies that are no longer resolved (ant:ant, asm:*, javax.mail:mail, oro:oro, tomcat:jasper-*, …) and 97 runtime dependencies are missing from the list (all of jetty 9.x, jersey 2.x, hk2, calcite, the hbase-* server modules, …). It reads as a list written for a Hive 2.x dependency set and never updated for 3.1.3.

To confirm this is not something I introduced: this PR touches no pom.xml, and running main's own unmodified validator with ripgrep available reproduces the identical failure.

How I have handled it, and why not more

Filed as #880 with the full evidence, and this PR adds an explicit one-module KNOWN_INCLUDES_DRIFT entry that downgrades only the includes-vs-tree comparison for only xtable-hive-metastore to a loud WARN printing the full diff and referencing #880. Every other check, for every module, stays blocking — I verified the scoping by planting a bogus <include> in xtable-aws, which still fails.

I did not fix the drift here because the two candidate fixes have materially different consequences for the published artifact:

1. Grow the include list to match the tree, which is what AGENTS.md currently prescribes. That would bundle all of jetty, jersey, hk2, calcite and the hbase server modules into the shaded jar, changing its contents and size significantly, plus ~15 more bundled license texts.
2. Narrow those dependencies to provided, if the expectation is that a Hive Metastore deployment supplies them. In that case the include list is closer to correct than the tree is, and the validator's "must exactly match" contract itself deserves another look.

That is a release/design call rather than a tooling fix, so it seemed wrong to decide it inside a license-text PR — and worse to leave the gate vacuous in order to avoid it. Happy to do either once you or dev@ pick one.

Two things you may want to consider separately, given the gate has not been effective on any branch:

• whether mvn-license-check.yml should also run on branch-0.4;
• whether 0.4.0-incubating rc1 was validated by this gate at all.

[vinishjail97]

Replacing my earlier comment on scope, now that I have a concrete alternative to point at rather than just a suggestion.

Suggestion: check the built jar instead of generating from the poms

I pushed one as #885 — release/scripts/validate_bundled_license_texts.py, 152 lines. It reads each *-bundled.jar, takes the coordinate listing from that jar's own META-INF/LICENSE, confirms each non-Apache-2.0 coordinate's classes are genuinely present, and requires META-INF/licenses/LICENSE-<artifactId>. It fails in both directions, so an orphaned text fails too.

Reading the artifact rather than the dependency tree removes the need for POM family resolution, the override tables, the parse fallbacks and the <includes> intersection — and with them the dependency on #880, since the check no longer cares what the includes list claims. #885 reuses your curated texts as-is; that curation is the part of this PR that has to happen whichever mechanism produces the metadata, and it is genuinely careful work.

On duplication

Measured across the 51 license files in this PR: 25 distinct texts, 9,050 lines of content, of which 2,681 are distinct and 6,369 are duplicates.

The single largest item is the CDDL 1.1 text — 759 lines, committed 8 times: jersey-client, jersey-guice, jersey-json and stax-api, each in both license_overrides/ and xtable-hive-metastore/. That one text is larger than everything else in the PR put together, including all 653 lines of code.

Two causes:

• license_overrides/ is a second committed copy of every text that also lands in a module. It exists to feed the generator, so dropping the generator drops the layer.
• LICENSE-<artifactId> forces one copy per artifact even where several artifacts share identical boilerplate. A single licenses/LICENSE-CDDL-1.1, named from the LICENSE-bundled entries, would remove roughly 2,300 lines from xtable-hive-metastore alone. That is a convention change worth agreeing before either patch lands — it applies equally to [700] Ship a license text for every bundled non-Apache-2.0 dependency #885, which currently has the same problem.

One process request

Could the description be trimmed? It has grown past what a reviewer can hold in one pass, and parts of it no longer match the head commit: open item 1 still says the generator refuses on javolution, but the ARTIFACT_OVERRIDES entry resolves it and a clean run does not stop there; and the xtable-hive-metastore count of 21 does not match a clean run, which writes 20. Since this description is what the rc2 vote thread will point at, something short and current would serve it better. Happy to help condense it if that is useful.

Numbers above come from builds of both main and branch-0.4; the method is in #885.

[rangareddy]

Thanks — I reproduced all of your numbers exactly (51/25/9050/2681/6369, the CDDL 759×8, and 653 lines of code), and both description defects were real. Fixed in 71436f4:

• Description rewritten, 92 → 48 lines. The javolution open item is gone (the ARTIFACT_OVERRIDES entry does resolve it) and the count now reads 20, not 21 — 21 predated removing the xml-apis override, which moved it into the Apache family.
• Shared boilerplate is committed once under license_overrides/shared/: the CDDL text was there four times, now one file, −2,277 lines. Source layout only — each artifact still ships its own LICENSE-<artifactId> byte for byte, verified before/after.
• Left the LICENSE-CDDL-1.1 naming alone, since you're right that it changes the published layout and should be agreed first.

Gates: rat pass, validator exit 0 (6 OK, no FAIL), generator idempotent.

I also built and ran #885 rather than just reading it. Four things worth knowing before choosing between them:

1. It cannot see a dependency that is bundled but absent from the listing. I removed javolution:javolution:5.5.1 from a jar's META-INF/LICENSE while leaving its 305 classes in place — the check reports nothing. It validates texts for listed coordinates, so something still has to guarantee the listing is complete.
2. groovy-all:2.4.11 ships its own META-INF/licenses/*.txt (antlr2, asm, jline2, jsr223, normalize-stylesheet) into the same directory. Those are permanent false orphans — not fixable by curation. It's Multiple LICENSE files present in some jars #701 territory.
3. M2 is hardcoded to ~/.m2. The same xtable-aws jar reports OK (5 deps) normally and FAILs with 5 false orphans under a different local repo, so -Dmaven.repo.local — what a release manager uses for clean-room RC verification — breaks it. An unresolvable dependency is skipped for missing but counted as orphaned.
4. No skip list, so it validated the unpublished xtable-utilities bundle; and its CI step runs package without clean, which I hit locally — it validated a two-day-old jar.

None of that is fatal, and the approach genuinely finds things mine misses: on #857 alone it flags 7 stale texts (activation, asm*, commons-el, mail, oro) that the tree-based check cannot see. With #880 applied it comes down to just the five groovy files.

And it caught a real bug in #880, which I'd rather you knew now: reconciling <includes> to the runtime tree takes hive-metastore from 165 to 260 bundled artifacts (~131 MB → 193 MB) and drops io.airlift:aircompressor, because shade bundles the compile classpath while dependency:tree -Dscope=runtime does not list it. The jar still bundles 401 ORC classes that reference io/airlift/compress/lzo/LzoDecompressor, so #880 as written would ship ORC with a missing codec. That makes the includes-equals-runtime-tree rule in AGENTS.md unsound, and is a point in favour of reading the artifact. I'll fix #880 before it goes up.

[rangareddy]

Took the duplication point from #885 and applied it — 4c20169.

license_overrides/ was a second committed copy of every curated text, and 1,405 of its 1,520 lines were byte-identical to a module's META-INF/licenses/ copy. The generator now reads curated texts from the committed metadata, guided by a CURATED_TEXTS list naming the artifacts whose own jar has no usable license; everything else still comes from the jar, so an upgrade still picks up a changed text. Directory removed. Earlier in the same push, the CDDL text that was committed four times became one file.

	before	now
license files	51	30
total lines	9,050	5,253
duplicate lines	6,369	2,623

3,746 duplicate lines removed. PR is +2845/-1718 across 39 files, from +7356/-201 across 58.

The 2,623 that remain are the shipped per-module copies — the LICENSE-CDDL-1.1 convention change you flagged. Still leaving that alone until it's agreed.

Verified: all 28 shipped texts regenerate byte-identical, so the jars are unchanged; the generator is idempotent; and deleting a curated text now fails the run naming the path to commit. Gates: rat pass, validator exit 0 (6 OK, no FAIL). A negative test caught one bug in the refactor — the failure path still referenced the removed directory and raised NameError instead of reporting — fixed before pushing.

I did not swap the generator for #885's checker. Beyond the four defects in the earlier comment, it reads the coordinate listing from the jar it is validating, so it cannot see a dependency that is bundled but absent from that listing — I removed javolution from a jar's META-INF/LICENSE with its 305 classes still in place and the check stayed silent. Something still has to produce the listing correctly, which is what the generator does. I would rather add its jar-side check alongside, as a separate change, than trade one blind spot for another.

One thing that surfaced while testing that integration, worth its own issue: the jar's own META-INF/maven/*/*/pom.properties shows artifacts arriving inside uber-jar dependencies — avatica-1.11.0.jar carries 12, ehcache-3.3.1.jar carries 3 — and their code is unattributed in META-INF/LICENSE. On a #880-built jar that is gson (173 classes), avatica (2,817), commons-codec (85), offheap-store (199), commons-logging (28) and others. Neither mechanism catches it today. It is not reachable from main, since ehcache and avatica are only bundled once #880 adds them — which is a further reason I am reworking #880 rather than raising it.

[vinishjail97]

This is still being unnecessarily complicated...here's what we will do to close the issue from a functional perspective.

1. Add validate_bundle_license_texts and run it as part of GH workflow mvn-license-check..this is similar to the validation ASF mentors run on the release bundled jars.
2. Add/Edit/Remove license files to get the validations passing.

The scripts approach of reading pom.xml and trying to assess has pitfalls for bundled jars and that's lot of AI generated code in generated_shaded_license_metadata that's being added without tests. Adding a validation that works after building the jar and flagging the missing license texts is easier to maintain and understand.