HADOOP-19971. Remove servlet and Jetty types from classes that do not need them - #8703
Open
joseluisll wants to merge 10 commits into
Open
HADOOP-19971. Remove servlet and Jetty types from classes that do not need them#8703joseluisll wants to merge 10 commits into
joseluisll wants to merge 10 commits into
Conversation
… module classpath
Several modules resolved more than one Jetty release, and more than one servlet
API, on a single classpath. Both combinations compile and then fail at run time,
on whichever code path reaches the wrong jar.
* Four modules used org.eclipse.jetty from their main sources while no pom in
their inheritance chain declared Jetty: hadoop-mapreduce-client-app,
hadoop-mapreduce-client-shuffle, hadoop-yarn-server-router and
hadoop-yarn-server-applicationhistoryservice. Each now declares the artifact
its main sources use, without a version, and jetty-http gains the
dependencyManagement entry it was missing. The Router declares jetty-webapp at
compile scope, since a test scope would override the compile-scoped copy it
inherits and take it off the Router's own runtime classpath.
* jersey-test-framework-provider-jetty carried jetty-continuation onto the test
classpath of around twenty modules at its own, older release. It is managed at
${jetty.version} rather than excluded, because Jersey needs the class:
Continuation is a final field on JettyHttpContainer's ResponseWriter,
initialised in the constructor.
* solr-core carried nine Jetty artifacts, and a servlet API, into
hadoop-yarn-applications-catalog-webapp's tests at a much older release. Those
tests reach Solr only through EmbeddedSolrServer, which runs no servlet
container, so none of it is needed. The exclusions are applied to
solr-test-framework as well, whose own path to solr-core would otherwise
reintroduce the same jars.
* hadoop-project excluded javax.servlet-api from jetty-server under the group
org.eclipse.jetty, which jetty-server has never used, so the exclusion matched
nothing and javax.servlet:javax.servlet-api stayed on the classpath beside the
managed jakarta.servlet:jakarta.servlet-api. Both publish the javax.servlet
packages, so which one a module compiled and ran against was decided by the
order of the jars, and 73 modules carried both. An exclusion naming the real
coordinates is added beside the existing one, and the same on
javax-websocket-server-impl, which reaches the API again through
websocket-servlet. hadoop-common now declares jakarta.servlet-api itself.
* hadoop-common declared jakarta.servlet.jsp-api, which reached some
eighty-five classpaths. Nothing in the tree uses JSP, so it and its managed
version are removed, together with the LICENSE-binary and NOTICE-binary
entries for the two artifacts that no longer ship.
One module keeps two servlet APIs, hadoop-yarn-server-timelineservice-hbase-tests,
where the second arrives with HBase's own test stack.
Co-Authored-By: Claude <noreply@anthropic.com>
joseluisll
changed the title
joseluisll
force-pushed
the
jetty-phase-b
branch
from
August 26, 2026 16:35
7f3cccc to
784374a
Compare
jersey-test-framework-provider-jetty resolves to jersey-container-jetty-http,
which Jersey 2.46 builds against Jetty 9.4.55 and which needs jetty-continuation.
That artifact does not exist past Jetty 9, so managing it at ${jetty.version}
stops resolving once that is 12.0.x. Jersey 2.x has no Jetty 12 container: the
one in Jersey 3.1.x is jakarta.servlet and cannot be used here.
Switch to jersey-test-framework-provider-jdk-http. It is the same Jersey release
and speaks real HTTP, but it runs on the JDK's own com.sun.net.httpserver, so it
adds a single artifact and no transitive dependencies at all. It puts no Jetty
and no servlet API on any test classpath, so the jetty-continuation entry and
its shade exclusion go with it.
hadoop-client-minicluster relocates com/ and already keeps the JDK's own com/sun
packages out of that. Add com/sun/net/httpserver to the list, or the container's
references to it would be rewritten into the shaded namespace and fail to
resolve at run time.
LICENSE-binary gains jersey-container-jdk-http, which reaches
hadoop-client-minicluster at compile scope.
Co-Authored-By: Claude <noreply@anthropic.com>
JobEndNotifier logged through org.eclipse.jetty.util.log.Log, Jetty's own logging facade, which is the only reason hadoop-mapreduce-client-app needed a Jetty dependency at all. Use SLF4J, as the rest of the tree does, and the declaration added earlier in this PR goes with it. The module now names no Jetty type anywhere, in main sources or tests. Raised by pan3793 in review. Co-Authored-By: Claude <noreply@anthropic.com>
The tree took the Servlet 4.0 API from jakarta.servlet:jakarta.servlet-api. jetty-ee8 depends on org.eclipse.jetty.toolchain:jetty-servlet-api instead, so staying on the jakarta coordinate means moving off it again when the ee8 environment lands. Use jetty-servlet-api now. Both publish the same javax.servlet classes, so no source changes, and 4.0.9 is the version Jetty 12 pins for ee8. The coordinate is set once rather than twice. Exclusions naming the servlet API follow the coordinate where a Hadoop module brings it. Where a Jersey artifact declares it, they stay on jakarta.servlet, which is what those artifacts publish. jersey-test-framework-core is one of those: it is now excluded in dependencyManagement, or its copy of the API would sit on the test classpath beside jetty-servlet-api. Co-Authored-By: Claude <noreply@anthropic.com>
…in javadoc. The import is unused: only a javadoc @link mentions the type. Spell that link out instead, and the class no longer references the servlet API at all. Co-Authored-By: Claude <noreply@anthropic.com>
…a PEM. CertificateUtil parses a PEM and has nothing to do with servlets. It used ServletException only to wrap the CertificateException it already catches. The new toRSAPublicKey reports that exception directly and names no servlet type. parseRSAPublicKey stays, deprecated, delegating to it and wrapping the exception back, so callers outside this tree keep compiling and keep seeing the same exception and message. It goes with the move to jakarta, where its signature has to change anyway. Co-Authored-By: Claude <noreply@anthropic.com>
…ider. SignerSecretProvider.init takes a ServletContext, so every implementation names a servlet type whether or not it uses one. Only ZKSignerSecretProvider really needs it, and what it needs is an attribute store, not a servlet: it shares a CuratorFramework as a context attribute. SecretProviderContext is that store, and initialize(Properties, SecretProviderContext, long) is the method providers should now override. The four providers here do, and none of them names a servlet type any more. Nothing breaks. init stays, deprecated, and stays the entry point callers use. It is now concrete: it adapts the ServletContext and calls initialize, so a provider overriding only initialize is initialized through it, while a provider overriding init is called directly, as before. Implementations live outside this tree - signer.secret.provider takes a classname - and keep compiling, linking and running untouched. The bridge dispatches virtually, so a provider extending RolloverSignerSecretProvider, overriding init and chaining to super still gets its rollover scheduler started. TestSignerSecretProviderCompatibility pins that and the rest of the contract. The adapter writes through to the real ServletContext, so the curator client is still a ServletContext attribute under the same name, as Configuration.md documents and as DelegationTokenAuthenticationFilter expects. A null ServletContext, which used to throw, now yields a store private to the provider. AuthenticationFilter and its tests are unchanged, which is the point. Co-Authored-By: Claude <noreply@anthropic.com>
It used org.eclipse.jetty.util.UrlEncoded to percent-encode a user name. java.net.URLEncoder does the same job from the JDK, so the class no longer needs Jetty. Co-Authored-By: Claude <noreply@anthropic.com>
ShuffleChannelHandler is a Netty handler. It imported org.eclipse.jetty.http.HttpHeader only to name the Connection and Keep-Alive headers. That was the module's last Jetty reference, so the jetty-http declaration HADOOP-19970 added to its pom goes too. The names are spelled as constants rather than taken from Netty's HttpHeaderNames and HttpHeaderValues, whose constants are lowercase AsciiStrings: using those would have sent "connection: keep-alive" where this handler has always sent "Connection: Keep-Alive". The shuffle response is read by other projects, so no byte of it changes here. Co-Authored-By: Claude <noreply@anthropic.com>
joseluisll
force-pushed
the
jetty-phase-b
branch
from
August 31, 2026 16:51
784374a to
5a75ac9
Compare
Four things the review of this branch raised, none of which changed what the
branch does, only what it says and how loudly it says it.
parseRSAPublicKey now reports the CertificateException the parse raised rather
than the one toRSAPublicKey wraps it in. Routing the deprecated method through
the new one had added a level to the cause chain a caller walks: it was
ServletException -> CertificateException -> IOException before and had become
ServletException -> CertificateException -> CertificateException -> IOException.
The chain is what it always was again.
A provider given no ServletContext gets a store private to itself, and now says
so when something is written to it. Only a provider that shares an object
through the store writes there - ZKSignerSecretProvider and its Curator client
is the one case here - and there is nothing to share it with, so a second
provider in the same JVM builds a second client rather than finding the first.
That used to be a NullPointerException, which was at least loud.
SignerSecretProvider documents what the move to initialize costs code that
reflects over these classes: the providers here no longer declare init, so
getDeclaredMethod("init", ...) on a subclass raises NoSuchMethodException where
it used to succeed. Calls and overrides resolve as before, and getMethod still
finds it.
ApiServiceClient records that URLEncoder is not byte-for-byte what Jetty's
UrlEncoded produced - they disagree on '~' and '*' - and that both forms decode
to the same user name.
Tests pin the cause chain and the unshared store.
Co-Authored-By: Claude <noreply@anthropic.com>
joseluisll
force-pushed
the
jetty-phase-b
branch
from
August 31, 2026 17:29
b592e5d to
3e150ab
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Stacked on #8699 (HADOOP-19970). The first two commits belong to that PR — review the last five.
Hadoop will eventually move from
javax.servlettojakarta.servlet. When it does, every class that mentions a servlet type has to be revisited, and every class that mentions a Jetty type has to be revisited again when the Jetty API changes underneath it.Several of those classes have no real connection to either. A component that generates signing secrets was handed the web server's context even though it only wanted somewhere to stash an object. A certificate parser reported a bad certificate as a web server error. A file-transfer handler built on Netty borrowed two header names from Jetty.
This clears those cases out, so the later migration is smaller and the classes concerned stop being part of it.
The MapReduce shuffle module stops depending on Jetty altogether, in code and in its build file. In hadoop-auth, the secret providers and the authentication token no longer mention the servlet API; they are initialised instead against a small two-method store that holds whatever needs sharing, which is all the one provider that used the servlet context was ever doing with it. One new class does mention the servlet API, and exists solely to keep the old path working. The YARN services client and the shuffle handler each replaced a small Jetty utility with the equivalent from the JDK or from Netty.
Nothing breaks. No public signature is removed or changed, and no dependency, version or setting changes. Code built against today's release keeps compiling and keeps running without being rebuilt.
SignerSecretProvider.initandCertificateUtil.parseRSAPublicKeystay, deprecated, and stay the entry points callers use; a cleaner alternative sits alongside each. Both deprecated methods go with the namespace change, where they have to change anyway.The shuffle protocol is untouched, byte for byte.
Deliberately out of scope: no namespace change, no Jetty upgrade, no Jersey change, no EE environments — the tree stays on Jetty 9.4, Jersey 2 and
javax.servletthroughout. The authentication handler classes still mention the servlet API, because they sit on the extension point HBase, Hive, Spark, Ozone and Knox build against; changing that is precisely the break this avoids, and it belongs with the namespace change.This works whether HADOOP-19912 goes through Jetty 12's ee8 environment or straight to ee10, and commits the project to neither.
How was this patch tested?
TestSignerSecretProviderCompatibilityis new and exists to prove the deprecated path still behaves — a provider written against the old entry point is still called, still receives the realServletContext, and a provider extendingRolloverSignerSecretProviderthat chains tosuperstill has its rollover scheduler started.TestCertificateUtilasserts the deprecated method still reports the exception it always did.AuthenticationFilter,TestAuthenticationFilterandTestZKSignerSecretProviderare unchanged from trunk, which is the clearest evidence the change is compatible.hadoop-auth: 52 tests, 0 failures.
TestShuffleChannelHandler: 8 tests, 0 failures.For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.
LICENSE,LICENSE-binary,NOTICE-binaryfiles?No dependencies are added or removed; one is dropped from the shuffle module's pom.
AI Tooling
Contains content generated by Claude.
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html