From a0c7fa59f7ca63edd48ff08c6fb1533f50bdfba2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Fri, 21 Aug 2026 18:10:10 +0000 Subject: [PATCH 01/30] HADOOP-19970. Resolve a single Jetty release and servlet API on every module classpath Several modules resolved more than one Jetty release, and more than one servlet API, on a single classpath. Both combinations compile and then fail at run time, on whichever code path reaches the wrong jar. * Four modules used org.eclipse.jetty from their main sources while no pom in their inheritance chain declared Jetty: hadoop-mapreduce-client-app, hadoop-mapreduce-client-shuffle, hadoop-yarn-server-router and hadoop-yarn-server-applicationhistoryservice. Each now declares the artifact its main sources use, without a version, and jetty-http gains the dependencyManagement entry it was missing. The Router declares jetty-webapp at compile scope, since a test scope would override the compile-scoped copy it inherits and take it off the Router's own runtime classpath. * jersey-test-framework-provider-jetty carried jetty-continuation onto the test classpath of around twenty modules at its own, older release. It is managed at ${jetty.version} rather than excluded, because Jersey needs the class: Continuation is a final field on JettyHttpContainer's ResponseWriter, initialised in the constructor. * solr-core carried nine Jetty artifacts, and a servlet API, into hadoop-yarn-applications-catalog-webapp's tests at a much older release. Those tests reach Solr only through EmbeddedSolrServer, which runs no servlet container, so none of it is needed. The exclusions are applied to solr-test-framework as well, whose own path to solr-core would otherwise reintroduce the same jars. * hadoop-project excluded javax.servlet-api from jetty-server under the group org.eclipse.jetty, which jetty-server has never used, so the exclusion matched nothing and javax.servlet:javax.servlet-api stayed on the classpath beside the managed jakarta.servlet:jakarta.servlet-api. Both publish the javax.servlet packages, so which one a module compiled and ran against was decided by the order of the jars, and 73 modules carried both. An exclusion naming the real coordinates is added beside the existing one, and the same on javax-websocket-server-impl, which reaches the API again through websocket-servlet. hadoop-common now declares jakarta.servlet-api itself. * hadoop-common declared jakarta.servlet.jsp-api, which reached some eighty-five classpaths. Nothing in the tree uses JSP, so it and its managed version are removed, together with the LICENSE-binary and NOTICE-binary entries for the two artifacts that no longer ship. One module keeps two servlet APIs, hadoop-yarn-server-timelineservice-hbase-tests, where the second arrives with HBase's own test stack. Co-Authored-By: Claude --- LICENSE-binary | 2 - NOTICE-binary | 1 - hadoop-common-project/hadoop-common/pom.xml | 16 +++- .../hadoop-mapreduce-client-app/pom.xml | 5 + .../hadoop-mapreduce-client-shuffle/pom.xml | 5 + hadoop-project/pom.xml | 39 +++++++- .../pom.xml | 94 +++++++++++++++++++ .../pom.xml | 15 +++ .../hadoop-yarn-server-router/pom.xml | 14 +++ 9 files changed, 178 insertions(+), 13 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 0e9bc06209de3a..8c29d984386bcd 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -625,7 +625,6 @@ CDDL 1.1 + GPLv2 with classpath exception com.sun.xml.bind:jaxb-impl:2.2.3-1 javax.annotation:javax.annotation-api:1.3.2 javax.cache:cache-api:1.1.1 -javax.servlet:javax.servlet-api:3.1.0 javax.servlet.jsp:jsp-api:2.1 javax.websocket:javax.websocket-api:1.0 @@ -645,7 +644,6 @@ Eclipse Public License (EPL) 2.0 -------------------------- jakarta.ws.rs-api:jakarta.ws.rs-api:2.1.6 -jakarta.servlet.jsp:jakarta.servlet.jsp-api:2.3.6 jakarta.servlet:jakarta.servlet-api:4.0.4 Eclipse Public License (EPL) 2.0 with some parts being diff --git a/NOTICE-binary b/NOTICE-binary index 407f6c25d997a4..b3f7359b96ae2d 100644 --- a/NOTICE-binary +++ b/NOTICE-binary @@ -463,7 +463,6 @@ Oracle The following artifacts are CDDL + GPLv2 with classpath exception. https://glassfish.dev.java.net/nonav/public/CDDL+GPL.html - * javax.servlet:javax.servlet-api * javax.annotation:javax.annotation-api * javax.transaction:javax.transaction-api * javax.websocket:javax.websocket-api diff --git a/hadoop-common-project/hadoop-common/pom.xml b/hadoop-common-project/hadoop-common/pom.xml index f10ed425370661..dbfe26fe1e1ce8 100644 --- a/hadoop-common-project/hadoop-common/pom.xml +++ b/hadoop-common-project/hadoop-common/pom.xml @@ -92,11 +92,6 @@ commons-collections4 compile - - jakarta.servlet.jsp - jakarta.servlet.jsp-api - compile - jakarta.ws.rs jakarta.ws.rs-api @@ -126,6 +121,17 @@ jakarta.activation-api runtime + + + jakarta.servlet + jakarta.servlet-api + compile + org.eclipse.jetty jetty-server diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml index 4184166ebd3394..b669a2d558260b 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml @@ -104,6 +104,11 @@ com.fasterxml.jackson.core jackson-databind + + + org.eclipse.jetty + jetty-util + org.bouncycastle bcprov-jdk18on diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml index f54f4162a8ddb0..cb3428316b65ae 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml @@ -55,6 +55,11 @@ ${leveldbjni.group} leveldbjni-all + + + org.eclipse.jetty + jetty-http + ch.qos.logback logback-classic diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 79f9b8093daa48..478848baa05e56 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -909,8 +909,37 @@ org.eclipse.jetty javax.servlet-api + + + javax.servlet + javax.servlet-api + + + org.eclipse.jetty + jetty-http + ${jetty.version} + + + + org.eclipse.jetty + jetty-continuation + ${jetty.version} + org.eclipse.jetty jetty-util @@ -939,6 +968,11 @@ org.eclipse.jetty jetty-webapp + + + javax.servlet + javax.servlet-api + @@ -946,11 +980,6 @@ websocket-client ${jetty.version} - - jakarta.servlet.jsp - jakarta.servlet.jsp-api - 2.3.6 - jakarta.servlet jakarta.servlet-api diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml index 724a91431c188c..18740d02a851dd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml @@ -189,6 +189,53 @@ org.eclipse.jetty jetty-alpn-client + + + org.eclipse.jetty + jetty-alpn-java-server + + + org.eclipse.jetty + jetty-alpn-server + + + org.eclipse.jetty + jetty-continuation + + + org.eclipse.jetty + jetty-deploy + + + org.eclipse.jetty + jetty-jmx + + + org.eclipse.jetty + jetty-rewrite + + + org.eclipse.jetty + jetty-security + + + org.eclipse.jetty + jetty-servlets + + + org.eclipse.jetty.http2 + http2-server + + + javax.servlet + javax.servlet-api + commons-collections commons-collections @@ -242,6 +289,53 @@ org.eclipse.jetty jetty-alpn-client + + + org.eclipse.jetty + jetty-alpn-java-server + + + org.eclipse.jetty + jetty-alpn-server + + + org.eclipse.jetty + jetty-continuation + + + org.eclipse.jetty + jetty-deploy + + + org.eclipse.jetty + jetty-jmx + + + org.eclipse.jetty + jetty-rewrite + + + org.eclipse.jetty + jetty-security + + + org.eclipse.jetty + jetty-servlets + + + org.eclipse.jetty.http2 + http2-server + + + javax.servlet + javax.servlet-api + commons-collections commons-collections diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml index b35bb52121684e..e84dbee6778922 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml @@ -53,6 +53,21 @@ org.apache.hadoop hadoop-annotations + + + + org.eclipse.jetty + jetty-webapp + provided + + + + org.eclipse.jetty + jetty-servlet + provided + org.mockito mockito-inline diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml index 67fc6348f7edc0..bef0086d1ef2ee 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml @@ -60,6 +60,20 @@ hadoop-yarn-server-common + + + org.eclipse.jetty + jetty-webapp + + org.apache.hadoop hadoop-yarn-server-common From 22dc62fa8e887eb72a39e2704249257e50010eed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 24 Aug 2026 19:48:59 +0000 Subject: [PATCH 02/30] HADOOP-19970. Move Jersey's test container off Jetty. jersey-test-framework-provider-jetty resolves to jersey-container-jetty-http, which Jersey 2.46 builds against Jetty 9.4.55 and which needs jetty-continuation. That artifact does not exist past Jetty 9, so managing it at ${jetty.version} stops resolving once that is 12.0.x. Jersey 2.x has no Jetty 12 container: the one in Jersey 3.1.x is jakarta.servlet and cannot be used here. Switch to jersey-test-framework-provider-jdk-http. It is the same Jersey release and speaks real HTTP, but it runs on the JDK's own com.sun.net.httpserver, so it adds a single artifact and no transitive dependencies at all. It puts no Jetty and no servlet API on any test classpath, so the jetty-continuation entry and its shade exclusion go with it. hadoop-client-minicluster relocates com/ and already keeps the JDK's own com/sun packages out of that. Add com/sun/net/httpserver to the list, or the container's references to it would be rewritten into the shaded namespace and fail to resolve at run time. LICENSE-binary gains jersey-container-jdk-http, which reaches hadoop-client-minicluster at compile scope. Co-Authored-By: Claude --- LICENSE-binary | 3 ++- .../hadoop-client-minicluster/pom.xml | 4 +++- .../hadoop-client-runtime/pom.xml | 1 - .../hadoop-mapreduce-client-app/pom.xml | 2 +- .../hadoop-mapreduce-client/pom.xml | 2 +- .../hadoop-mapreduce-examples/pom.xml | 2 +- hadoop-project/pom.xml | 20 +------------------ hadoop-tools/hadoop-resourceestimator/pom.xml | 2 +- .../hadoop-yarn/hadoop-yarn-client/pom.xml | 2 +- .../hadoop-yarn/hadoop-yarn-common/pom.xml | 2 +- .../pom.xml | 2 +- .../pom.xml | 2 +- .../hadoop-yarn-server-nodemanager/pom.xml | 2 +- .../pom.xml | 2 +- .../pom.xml | 2 +- .../hadoop-yarn-server-web-proxy/pom.xml | 2 +- 16 files changed, 18 insertions(+), 34 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 8c29d984386bcd..450515be8d4ac6 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -658,9 +658,10 @@ org.glassfish.jersey.core:jersey-server:2.46 org.glassfish.jersey.inject:jersey-hk2:2.46 org.glassfish.jersey.core:jersey-client:2.46 org.glassfish.jersey.test-framework:jersey-test-framework-core:2.46 -org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-jetty:2.46 +org.glassfish.jersey.test-framework.providers:jersey-test-framework-provider-jdk-http:2.46 org.glassfish.jersey.containers:jersey-container-servlet:2.46 org.glassfish.jersey.containers:jersey-container-servlet-core:2.46 +org.glassfish.jersey.containers:jersey-container-jdk-http:2.46 org.glassfish.jersey.media:jersey-media-json-jettison:2.46 org.glassfish.jersey.media:jersey-media-jaxb:2.46 diff --git a/hadoop-client-modules/hadoop-client-minicluster/pom.xml b/hadoop-client-modules/hadoop-client-minicluster/pom.xml index 7042f2ce07880a..fb2767a2a1c034 100644 --- a/hadoop-client-modules/hadoop-client-minicluster/pom.xml +++ b/hadoop-client-modules/hadoop-client-minicluster/pom.xml @@ -632,7 +632,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http jakarta.ws.rs @@ -940,11 +940,13 @@ com/sun/security/* com/sun/jndi/* com/sun/management/* + com/sun/net/httpserver/* com/sun/tools/**/* com/sun/javadoc/**/* com/sun/security/**/* com/sun/jndi/**/* com/sun/management/**/* + com/sun/net/httpserver/**/* com/ibm/security/* com/ibm/security/**/* diff --git a/hadoop-client-modules/hadoop-client-runtime/pom.xml b/hadoop-client-modules/hadoop-client-runtime/pom.xml index de626ec9b18f1a..40d2c9e883a855 100644 --- a/hadoop-client-modules/hadoop-client-runtime/pom.xml +++ b/hadoop-client-modules/hadoop-client-runtime/pom.xml @@ -170,7 +170,6 @@ org.eclipse.jetty:jetty-util org.eclipse.jetty:jetty-util-ajax org.eclipse.jetty:jetty-server - org.eclipse.jetty:jetty-continuation org.ow2.asm:* org.bouncycastle:* diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml index b669a2d558260b..20f32c7fa8124c 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml @@ -136,7 +136,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml index acca311eb53c03..0ec8950a4234e8 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml @@ -161,7 +161,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml b/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml index 1e02c05fe75b32..45041b3668c74a 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml @@ -101,7 +101,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 478848baa05e56..277c4dc81771e8 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -928,18 +928,6 @@ jetty-http ${jetty.version} - - - org.eclipse.jetty - jetty-continuation - ${jetty.version} - org.eclipse.jetty jetty-util @@ -2206,14 +2194,8 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http ${jersey2.version} - - - javax.servlet - javax.servlet-api - - org.glassfish.jersey.media diff --git a/hadoop-tools/hadoop-resourceestimator/pom.xml b/hadoop-tools/hadoop-resourceestimator/pom.xml index 5bd5db3d5422df..7a7695db49cc96 100644 --- a/hadoop-tools/hadoop-resourceestimator/pom.xml +++ b/hadoop-tools/hadoop-resourceestimator/pom.xml @@ -67,7 +67,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml index 04a78a94a54e96..88c5b1ac392b82 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml @@ -91,7 +91,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml index fdfec1d7d1ca6a..b9d9b685c7d66b 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml @@ -148,7 +148,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml index e84dbee6778922..1131bae58efdd0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml @@ -107,7 +107,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/pom.xml index 05913c2832733b..a13710b991007a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/pom.xml @@ -119,7 +119,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml index afde1fa415a6bc..385e1fb05b6094 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml @@ -203,7 +203,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml index 40dcf5d390b558..c9362a9e5e4198 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml @@ -244,7 +244,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml index 9b6f8f4a79e967..7ee63ba3edcef4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml @@ -222,7 +222,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http org.glassfish.jersey.test-framework diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/pom.xml index 6c64dce18ebbee..591e3852c92fee 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/pom.xml @@ -123,7 +123,7 @@ org.glassfish.jersey.test-framework.providers - jersey-test-framework-provider-jetty + jersey-test-framework-provider-jdk-http test From 236e4bd76b7ebe4abf2621283fc12c1c846dd761 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Wed, 26 Aug 2026 04:41:52 +0000 Subject: [PATCH 03/30] HADOOP-19970. Log through SLF4J in JobEndNotifier. JobEndNotifier logged through org.eclipse.jetty.util.log.Log, Jetty's own logging facade, which is the only reason hadoop-mapreduce-client-app needed a Jetty dependency at all. Use SLF4J, as the rest of the tree does, and the declaration added earlier in this PR goes with it. The module now names no Jetty type anywhere, in main sources or tests. Raised by pan3793 in review. Co-Authored-By: Claude --- .../hadoop-mapreduce-client-app/pom.xml | 5 --- .../mapreduce/v2/app/JobEndNotifier.java | 34 +++++++++++-------- 2 files changed, 19 insertions(+), 20 deletions(-) diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml index 20f32c7fa8124c..9583523ce553c5 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml @@ -104,11 +104,6 @@ com.fasterxml.jackson.core jackson-databind - - - org.eclipse.jetty - jetty-util - org.bouncycastle bcprov-jdk18on diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java index ed49f82506bc6a..b98451223a664d 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java @@ -32,7 +32,8 @@ import org.apache.hadoop.mapreduce.CustomJobEndNotifier; import org.apache.hadoop.mapreduce.MRJobConfig; import org.apache.hadoop.mapreduce.v2.api.records.JobReport; -import org.eclipse.jetty.util.log.Log; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** *

This class handles job end notification. Submitters of jobs can choose to @@ -48,6 +49,9 @@ * (eg. SUCCEEDED/KILLED/FAILED) */ public class JobEndNotifier implements Configurable { + private static final Logger LOG = + LoggerFactory.getLogger(JobEndNotifier.class); + private static final String JOB_ID = "$jobId"; private static final String JOB_STATUS = "$jobStatus"; @@ -109,11 +113,11 @@ public void setConf(Configuration conf) { int port = Integer.parseInt(portConf); proxyToUse = new Proxy(proxyType, new InetSocketAddress(hostname, port)); - Log.getLog().info("Job end notification using proxy type \"" + LOG.info("Job end notification using proxy type \"" + proxyType + "\" hostname \"" + hostname + "\" and port \"" + port + "\""); } catch(NumberFormatException nfe) { - Log.getLog().warn("Job end notification couldn't parse configured" + LOG.warn("Job end notification couldn't parse configured" + "proxy's port " + portConf + ". Not going to use a proxy"); } } @@ -141,24 +145,24 @@ protected boolean notifyURLOnce() { private boolean notifyViaBuiltInNotifier() { boolean success = false; try { - Log.getLog().info("Job end notification trying " + urlToNotify); + LOG.info("Job end notification trying " + urlToNotify); HttpURLConnection conn = (HttpURLConnection) urlToNotify.openConnection(proxyToUse); conn.setConnectTimeout(timeout); conn.setReadTimeout(timeout); conn.setAllowUserInteraction(false); if(conn.getResponseCode() != HttpURLConnection.HTTP_OK) { - Log.getLog().warn("Job end notification to " + urlToNotify + LOG.warn("Job end notification to " + urlToNotify + " failed with code: " + conn.getResponseCode() + " and message \"" + conn.getResponseMessage() + "\""); } else { success = true; - Log.getLog().info("Job end notification to " + urlToNotify + LOG.info("Job end notification to " + urlToNotify + " succeeded"); } } catch(IOException ioe) { - Log.getLog().warn("Job end notification to " + urlToNotify + " failed", + LOG.warn("Job end notification to " + urlToNotify + " failed", ioe); } return success; @@ -169,7 +173,7 @@ private boolean notifyViaBuiltInNotifier() { */ private boolean notifyViaCustomNotifier() { try { - Log.getLog().info("Will be using " + customJobEndNotifierClassName + LOG.info("Will be using " + customJobEndNotifierClassName + " for Job end notification"); final Class customJobEndNotifierClass = @@ -180,15 +184,15 @@ private boolean notifyViaCustomNotifier() { boolean success = customJobEndNotifier.notifyOnce(urlToNotify, conf); if (success) { - Log.getLog().info("Job end notification to " + urlToNotify + LOG.info("Job end notification to " + urlToNotify + " succeeded"); } else { - Log.getLog().warn("Job end notification to " + urlToNotify + LOG.warn("Job end notification to " + urlToNotify + " failed"); } return success; } catch (Exception e) { - Log.getLog().warn("Job end notification to " + urlToNotify + LOG.warn("Job end notification to " + urlToNotify + " failed", e); return false; } @@ -215,24 +219,24 @@ public void notify(JobReport jobReport) try { urlToNotify = new URL(userUrl); } catch (MalformedURLException mue) { - Log.getLog().warn("Job end notification couldn't parse " + userUrl, mue); + LOG.warn("Job end notification couldn't parse " + userUrl, mue); return; } // Send notification boolean success = false; while (numTries-- > 0 && !success) { - Log.getLog().info("Job end notification attempts left " + numTries); + LOG.info("Job end notification attempts left " + numTries); success = notifyURLOnce(); if (!success) { Thread.sleep(waitInterval); } } if (!success) { - Log.getLog().warn("Job end notification failed to notify : " + LOG.warn("Job end notification failed to notify : " + urlToNotify); } else { - Log.getLog().info("Job end notification succeeded for " + LOG.info("Job end notification succeeded for " + jobReport.getJobId()); } } From 0bc0ff836e902badc3220c8e1ca008198d45eda8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 31 Aug 2026 12:07:05 +0000 Subject: [PATCH 04/30] HADOOP-19970. Take the servlet API from Jetty. The tree took the Servlet 4.0 API from jakarta.servlet:jakarta.servlet-api. jetty-ee8 depends on org.eclipse.jetty.toolchain:jetty-servlet-api instead, so staying on the jakarta coordinate means moving off it again when the ee8 environment lands. Use jetty-servlet-api now. Both publish the same javax.servlet classes, so no source changes, and 4.0.9 is the version Jetty 12 pins for ee8. The coordinate is set once rather than twice. Exclusions naming the servlet API follow the coordinate where a Hadoop module brings it. Where a Jersey artifact declares it, they stay on jakarta.servlet, which is what those artifacts publish. jersey-test-framework-core is one of those: it is now excluded in dependencyManagement, or its copy of the API would sit on the test classpath beside jetty-servlet-api. Co-Authored-By: Claude --- LICENSE-binary | 2 +- .../hadoop-client-minicluster/pom.xml | 16 +++++----- .../hadoop-auth-examples/pom.xml | 4 +-- hadoop-common-project/hadoop-auth/pom.xml | 4 +-- hadoop-common-project/hadoop-common/pom.xml | 4 +-- hadoop-common-project/hadoop-kms/pom.xml | 4 +-- hadoop-common-project/hadoop-nfs/pom.xml | 4 +-- .../hadoop-hdfs-httpfs/pom.xml | 4 +-- hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml | 4 +-- hadoop-hdfs-project/hadoop-hdfs/pom.xml | 4 +-- hadoop-project/pom.xml | 31 +++++++++++++++++-- hadoop-tools/hadoop-resourceestimator/pom.xml | 4 +-- .../hadoop-yarn-services-api/pom.xml | 4 +-- .../hadoop-yarn/hadoop-yarn-common/pom.xml | 4 +-- .../pom.xml | 4 +-- .../hadoop-yarn-server-common/pom.xml | 4 +-- .../pom.xml | 4 +-- .../hadoop-yarn-server-nodemanager/pom.xml | 4 +-- .../pom.xml | 4 +-- .../pom.xml | 4 +-- .../hadoop-yarn-server-web-proxy/pom.xml | 4 +-- 21 files changed, 73 insertions(+), 48 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 450515be8d4ac6..c3c0a92ca4d292 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -423,6 +423,7 @@ org.eclipse.jetty:jetty-util:9.4.58.v20250814 org.eclipse.jetty:jetty-util-ajax:9.4.58.v20250814 org.eclipse.jetty:jetty-webapp:9.4.58.v20250814 org.eclipse.jetty:jetty-xml:9.4.58.v20250814 +org.eclipse.jetty.toolchain:jetty-servlet-api:4.0.9 org.eclipse.jetty.websocket:javax-websocket-client-impl:9.4.58.v20250814 org.eclipse.jetty.websocket:javax-websocket-server-impl:9.4.58.v20250814 org.ehcache:ehcache:3.8.2 @@ -644,7 +645,6 @@ Eclipse Public License (EPL) 2.0 -------------------------- jakarta.ws.rs-api:jakarta.ws.rs-api:2.1.6 -jakarta.servlet:jakarta.servlet-api:4.0.4 Eclipse Public License (EPL) 2.0 with some parts being GNU General Public License (GPL), Version 2, With Classpath Exception, diff --git a/hadoop-client-modules/hadoop-client-minicluster/pom.xml b/hadoop-client-modules/hadoop-client-minicluster/pom.xml index fb2767a2a1c034..e2e9588a668048 100644 --- a/hadoop-client-modules/hadoop-client-minicluster/pom.xml +++ b/hadoop-client-modules/hadoop-client-minicluster/pom.xml @@ -124,8 +124,8 @@ javax.servlet-api - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api jakarta.ws.rs @@ -419,7 +419,7 @@ - + @@ -464,7 +464,7 @@ - + org.apache.hadoop @@ -476,8 +476,8 @@ javax.servlet-api - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api jakarta.ws.rs @@ -551,8 +551,8 @@ javax.servlet-api - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api jakarta.ws.rs diff --git a/hadoop-common-project/hadoop-auth-examples/pom.xml b/hadoop-common-project/hadoop-auth-examples/pom.xml index 331cffc7815597..f442839fda2aeb 100644 --- a/hadoop-common-project/hadoop-auth-examples/pom.xml +++ b/hadoop-common-project/hadoop-auth-examples/pom.xml @@ -32,8 +32,8 @@ - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api provided diff --git a/hadoop-common-project/hadoop-auth/pom.xml b/hadoop-common-project/hadoop-auth/pom.xml index 4b8854887c1966..4588a15f8cd8d0 100644 --- a/hadoop-common-project/hadoop-auth/pom.xml +++ b/hadoop-common-project/hadoop-auth/pom.xml @@ -62,8 +62,8 @@ test - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api provided diff --git a/hadoop-common-project/hadoop-common/pom.xml b/hadoop-common-project/hadoop-common/pom.xml index dbfe26fe1e1ce8..3f50fd7d47ac68 100644 --- a/hadoop-common-project/hadoop-common/pom.xml +++ b/hadoop-common-project/hadoop-common/pom.xml @@ -128,8 +128,8 @@ depending on which jar happened to come first on the classpath. --> - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api compile diff --git a/hadoop-common-project/hadoop-kms/pom.xml b/hadoop-common-project/hadoop-kms/pom.xml index e2874eb3a8033a..8ccf377c5cd5f8 100644 --- a/hadoop-common-project/hadoop-kms/pom.xml +++ b/hadoop-common-project/hadoop-kms/pom.xml @@ -69,8 +69,8 @@ compile - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api org.eclipse.jetty diff --git a/hadoop-common-project/hadoop-nfs/pom.xml b/hadoop-common-project/hadoop-nfs/pom.xml index 685397e131d427..f942f04718d27d 100644 --- a/hadoop-common-project/hadoop-nfs/pom.xml +++ b/hadoop-common-project/hadoop-nfs/pom.xml @@ -59,8 +59,8 @@ test - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api provided diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml b/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml index f0c0e21ac12e50..14e46f62bc52a0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml @@ -63,8 +63,8 @@ compile - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api compile diff --git a/hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml b/hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml index 38f293c1c34cc6..149a075b949362 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml @@ -149,8 +149,8 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd"> ${transient.protobuf2.scope} - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api compile diff --git a/hadoop-hdfs-project/hadoop-hdfs/pom.xml b/hadoop-hdfs-project/hadoop-hdfs/pom.xml index 07154ccdf7f872..1c2ff021e9aa47 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/pom.xml @@ -142,8 +142,8 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd"> ${transient.protobuf2.scope} - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api compile diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 277c4dc81771e8..d134d39745162b 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -37,6 +37,12 @@ true true 9.4.58.v20250814 + + 4.0.9 _ _ @@ -969,9 +975,9 @@ ${jetty.version} - jakarta.servlet - jakarta.servlet-api - 4.0.4 + org.eclipse.jetty.toolchain + jetty-servlet-api + ${jetty-servlet-api.version} @@ -2191,11 +2197,30 @@ org.glassfish.jersey.test-framework jersey-test-framework-core ${jersey2.version} + + + + jakarta.servlet + jakarta.servlet-api + + org.glassfish.jersey.test-framework.providers jersey-test-framework-provider-jdk-http ${jersey2.version} + + + + jakarta.servlet + jakarta.servlet-api + + org.glassfish.jersey.media diff --git a/hadoop-tools/hadoop-resourceestimator/pom.xml b/hadoop-tools/hadoop-resourceestimator/pom.xml index 7a7695db49cc96..ce05ed3c722f1c 100644 --- a/hadoop-tools/hadoop-resourceestimator/pom.xml +++ b/hadoop-tools/hadoop-resourceestimator/pom.xml @@ -71,8 +71,8 @@ test - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api org.glassfish.jersey.core diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/pom.xml index 4e5f602cf387ce..1aa02a539893bd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/pom.xml @@ -136,8 +136,8 @@ jakarta.ws.rs-api - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api commons-codec diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml index b9d9b685c7d66b..943f5e49cf8c8e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml @@ -65,8 +65,8 @@ commons-compress - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api commons-codec diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml index 1131bae58efdd0..69df71c598633b 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml @@ -38,8 +38,8 @@ - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml index fe166cd1c1abbb..1c1c243579735e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml @@ -119,8 +119,8 @@ leveldbjni-all - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api javax.cache diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/pom.xml index a13710b991007a..e4aea97967047a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/pom.xml @@ -124,8 +124,8 @@ - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml index 385e1fb05b6094..17fafd3e5b71ec 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml @@ -70,8 +70,8 @@ jettison - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api commons-codec diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml index c9362a9e5e4198..65e406d132ad72 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml @@ -33,8 +33,8 @@ - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml index 180b15713b4268..2206ed41874d35 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml @@ -88,8 +88,8 @@ - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/pom.xml index 591e3852c92fee..e48c2d965761c3 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/pom.xml @@ -33,8 +33,8 @@ - jakarta.servlet - jakarta.servlet-api + org.eclipse.jetty.toolchain + jetty-servlet-api compile From 3e9a59aef3d5d555c68ba5cce4597cfe498c5481 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 24 Aug 2026 19:49:37 +0000 Subject: [PATCH 05/30] HADOOP-19971. Drop the servlet import AuthenticationToken only names in javadoc. The import is unused: only a javadoc @link mentions the type. Spell that link out instead, and the class no longer references the servlet API at all. Co-Authored-By: Claude --- .../security/authentication/server/AuthenticationToken.java | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationToken.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationToken.java index 8295fe173f4b1a..68da16580ec944 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationToken.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationToken.java @@ -18,12 +18,10 @@ import java.security.Principal; -import javax.servlet.http.HttpServletRequest; - /** * The {@link AuthenticationToken} contains information about an authenticated * HTTP client and doubles as the {@link Principal} to be returned by - * authenticated {@link HttpServletRequest}s + * authenticated HTTP requests. *

* The token can be serialized/deserialized to and from a string as it is sent * and received in HTTP client responses and requests as a HTTP cookie (this is From 97b2f44e52a3b7ab0631211bfb4f9e4e5d448004 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 24 Aug 2026 19:49:37 +0000 Subject: [PATCH 06/30] HADOOP-19971. Add a servlet-free way to parse an RSA public key from a PEM. CertificateUtil parses a PEM and has nothing to do with servlets. It used ServletException only to wrap the CertificateException it already catches. The new toRSAPublicKey reports that exception directly and names no servlet type. parseRSAPublicKey stays, deprecated, delegating to it and wrapping the exception back, so callers outside this tree keep compiling and keep seeing the same exception and message. It goes with the move to jakarta, where its signature has to change anyway. Co-Authored-By: Claude --- .../JWTRedirectAuthenticationHandler.java | 7 +++- .../authentication/util/CertificateUtil.java | 28 +++++++++++++-- .../util/TestCertificateUtil.java | 35 ++++++++++++++----- 3 files changed, 59 insertions(+), 11 deletions(-) diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java index 2dcb60836b5e43..5fba0af1bf2a58 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/JWTRedirectAuthenticationHandler.java @@ -26,6 +26,7 @@ import java.util.Properties; import java.text.ParseException; +import java.security.cert.CertificateException; import java.security.interfaces.RSAPublicKey; import org.apache.hadoop.classification.VisibleForTesting; @@ -125,7 +126,11 @@ public void init(Properties config) throws ServletException { throw new ServletException( "Public key for signature validation must be provisioned."); } - publicKey = CertificateUtil.parseRSAPublicKey(pemPublicKey); + try { + publicKey = CertificateUtil.toRSAPublicKey(pemPublicKey); + } catch (CertificateException ce) { + throw new ServletException(ce.getMessage(), ce); + } } // setup the list of valid audiences for token validation String auds = config.getProperty(EXPECTED_JWT_AUDIENCES); diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/CertificateUtil.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/CertificateUtil.java index f25602c67d4a31..068d7a3c383f94 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/CertificateUtil.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/CertificateUtil.java @@ -38,8 +38,32 @@ public class CertificateUtil { * - the pem encoding from config without the header and footer * @return RSAPublicKey the RSA public key * @throws ServletException thrown if a processing error occurred + * @deprecated use {@link #toRSAPublicKey(String)}, which reports the + * {@link CertificateException} it is really raising instead of + * wrapping it in a servlet type. This method is kept so that + * existing callers still compile and behave as before, and will + * be removed with the move to the jakarta servlet namespace. */ - public static RSAPublicKey parseRSAPublicKey(String pem) throws ServletException { + @Deprecated + public static RSAPublicKey parseRSAPublicKey(String pem) + throws ServletException { + try { + return toRSAPublicKey(pem); + } catch (CertificateException ce) { + throw new ServletException(ce.getMessage(), ce); + } + } + + /** + * Gets an RSAPublicKey from the provided PEM encoding. + * + * @param pem + * - the pem encoding from config without the header and footer + * @return RSAPublicKey the RSA public key + * @throws CertificateException thrown if the PEM could not be parsed + */ + public static RSAPublicKey toRSAPublicKey(String pem) + throws CertificateException { String fullPem = PEM_HEADER + pem + PEM_FOOTER; PublicKey key = null; try { @@ -57,7 +81,7 @@ public static RSAPublicKey parseRSAPublicKey(String pem) throws ServletException } else { message = "CertificateException - PEM may be corrupt"; } - throw new ServletException(message, ce); + throw new CertificateException(message, ce); } return (RSAPublicKey) key; } diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestCertificateUtil.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestCertificateUtil.java index 0580bac9053be3..5ffbad88cd5031 100644 --- a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestCertificateUtil.java +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestCertificateUtil.java @@ -22,6 +22,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue; import static org.junit.jupiter.api.Assertions.fail; +import java.security.cert.CertificateException; import java.security.interfaces.RSAPublicKey; import javax.servlet.ServletException; @@ -45,9 +46,9 @@ public void testInvalidPEMWithHeaderAndFooter() throws Exception { + "9aTyR+HGHCfvwoCegc9rAVw/DLaRriSO/jnEXzYK6XLVKH+hx5UXrJ7Oyc7JjZUc3g9kCWORThCX" + "Mzc1xA==" + "\n-----END CERTIFICATE-----"; try { - CertificateUtil.parseRSAPublicKey(pem); - fail("Should not have thrown ServletException"); - } catch (ServletException se) { + CertificateUtil.toRSAPublicKey(pem); + fail("Should not have thrown CertificateException"); + } catch (CertificateException se) { assertTrue(se.getMessage().contains("PEM header")); } } @@ -66,9 +67,9 @@ public void testCorruptPEM() throws Exception { + "9aTyR+HGHCfvwoCegc9rAVw/DLaRriSO/jnEXzYK6XLVKH+hx5UXrJ7Oyc7JjZUc3g9kCWORThCX" + "Mzc1xA++"; try { - CertificateUtil.parseRSAPublicKey(pem); - fail("Should not have thrown ServletException"); - } catch (ServletException se) { + CertificateUtil.toRSAPublicKey(pem); + fail("Should not have thrown CertificateException"); + } catch (CertificateException se) { assertTrue(se.getMessage().contains("corrupt")); } } @@ -87,11 +88,29 @@ public void testValidPEM() throws Exception { + "9aTyR+HGHCfvwoCegc9rAVw/DLaRriSO/jnEXzYK6XLVKH+hx5UXrJ7Oyc7JjZUc3g9kCWORThCX" + "Mzc1xA=="; try { - RSAPublicKey pk = CertificateUtil.parseRSAPublicKey(pem); + RSAPublicKey pk = CertificateUtil.toRSAPublicKey(pem); assertNotNull(pk); assertEquals("RSA", pk.getAlgorithm()); - } catch (ServletException se) { + } catch (CertificateException se) { + fail("Should not have thrown CertificateException"); + } + } + + /** + * The deprecated entry point still reports a ServletException, so callers + * written against it keep compiling and keep catching what they caught + * before. + */ + @Test + @SuppressWarnings("deprecation") + public void testDeprecatedEntryPointStillThrowsServletException() { + String pem = "not a certificate"; + try { + CertificateUtil.parseRSAPublicKey(pem); fail("Should not have thrown ServletException"); + } catch (ServletException se) { + assertTrue(se.getMessage().contains("corrupt")); + assertTrue(se.getCause() instanceof CertificateException); } } From c1c3a375ecccdabecc9805d75a82f797c788e2e9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 24 Aug 2026 19:49:47 +0000 Subject: [PATCH 07/30] HADOOP-19971. Add a servlet-free way to initialize a SignerSecretProvider. SignerSecretProvider.init takes a ServletContext, so every implementation names a servlet type whether or not it uses one. Only ZKSignerSecretProvider really needs it, and what it needs is an attribute store, not a servlet: it shares a CuratorFramework as a context attribute. SecretProviderContext is that store, and initialize(Properties, SecretProviderContext, long) is the method providers should now override. The four providers here do, and none of them names a servlet type any more. Nothing breaks. init stays, deprecated, and stays the entry point callers use. It is now concrete: it adapts the ServletContext and calls initialize, so a provider overriding only initialize is initialized through it, while a provider overriding init is called directly, as before. Implementations live outside this tree - signer.secret.provider takes a classname - and keep compiling, linking and running untouched. The bridge dispatches virtually, so a provider extending RolloverSignerSecretProvider, overriding init and chaining to super still gets its rollover scheduler started. TestSignerSecretProviderCompatibility pins that and the rest of the contract. The adapter writes through to the real ServletContext, so the curator client is still a ServletContext attribute under the same name, as Configuration.md documents and as DelegationTokenAuthenticationFilter expects. A null ServletContext, which used to throw, now yields a store private to the provider. AuthenticationFilter and its tests are unchanged, which is the point. Co-Authored-By: Claude --- .../util/FileSignerSecretProvider.java | 3 +- .../util/RolloverSignerSecretProvider.java | 5 +- .../util/SecretProviderContext.java | 52 +++++ .../util/ServletSecretProviderContext.java | 95 +++++++++ .../util/SignerSecretProvider.java | 52 ++++- .../util/ZKSignerSecretProvider.java | 15 +- .../util/StringSignerSecretProvider.java | 3 +- ...TestSignerSecretProviderCompatibility.java | 200 ++++++++++++++++++ 8 files changed, 408 insertions(+), 17 deletions(-) create mode 100644 hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SecretProviderContext.java create mode 100644 hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ServletSecretProviderContext.java create mode 100644 hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestSignerSecretProviderCompatibility.java diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java index 2a8a712b595bac..66bd181771a9a1 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java @@ -17,7 +17,6 @@ import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.security.authentication.server.AuthenticationFilter; -import javax.servlet.ServletContext; import java.io.*; import java.nio.charset.StandardCharsets; import java.nio.file.Files; @@ -37,7 +36,7 @@ public class FileSignerSecretProvider extends SignerSecretProvider { public FileSignerSecretProvider() {} @Override - public void init(Properties config, ServletContext servletContext, + public void initialize(Properties config, SecretProviderContext context, long tokenValidity) throws Exception { String signatureSecretFile = config.getProperty( diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.java index ca95272cf9fd62..9edc3145c4a02c 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.java @@ -17,7 +17,6 @@ import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; -import javax.servlet.ServletContext; import org.apache.hadoop.classification.VisibleForTesting; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; @@ -61,12 +60,12 @@ public RolloverSignerSecretProvider() { * and starts the scheduler for the rollover to run at an interval of * tokenValidity. * @param config configuration properties - * @param servletContext servlet context + * @param context the attribute store to initialize against * @param tokenValidity The amount of time a token is valid for * @throws Exception thrown if an error occurred */ @Override - public void init(Properties config, ServletContext servletContext, + public void initialize(Properties config, SecretProviderContext context, long tokenValidity) throws Exception { initSecrets(generateNewSecret(), null); startScheduler(tokenValidity, tokenValidity); diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SecretProviderContext.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SecretProviderContext.java new file mode 100644 index 00000000000000..b002ab0c047303 --- /dev/null +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SecretProviderContext.java @@ -0,0 +1,52 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. See accompanying LICENSE file. + */ +package org.apache.hadoop.security.authentication.util; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; + +/** + * The attribute store a {@link SignerSecretProvider} is initialized against. + *

+ * A SignerSecretProvider that has to share an object with the rest of the + * application it is embedded in - as {@link ZKSignerSecretProvider} shares its + * CuratorFramework client - reads and writes that object here. When the + * provider is running inside a servlet container the attributes are those of + * the ServletContext, so the sharing is unchanged; see + * {@link org.apache.hadoop.security.authentication.server.AuthenticationFilter}. + *

+ * This interface exists so that a SignerSecretProvider need not name a servlet + * type. Implementations of it that are backed by a ServletContext do, but the + * providers themselves stay independent of the servlet API and so of which + * servlet namespace - javax or jakarta - the container provides. + */ +@InterfaceStability.Unstable +@InterfaceAudience.Private +public interface SecretProviderContext { + + /** + * Returns the attribute stored under the given name, or null if there is + * none. + * @param name the attribute name + * @return the attribute value, or null + */ + Object getAttribute(String name); + + /** + * Stores an attribute under the given name, replacing any previous value. + * @param name the attribute name + * @param value the attribute value + */ + void setAttribute(String name, Object value); +} diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ServletSecretProviderContext.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ServletSecretProviderContext.java new file mode 100644 index 00000000000000..e7f576f7fc59ae --- /dev/null +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ServletSecretProviderContext.java @@ -0,0 +1,95 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. See accompanying LICENSE file. + */ +package org.apache.hadoop.security.authentication.util; + +import java.util.HashMap; +import java.util.Map; +import javax.servlet.ServletContext; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; + +/** + * Presents a ServletContext to a {@link SignerSecretProvider} as a + * {@link SecretProviderContext}. + *

+ * Reads and writes go straight through to the ServletContext, so an object a + * provider shares this way - the CuratorFramework client of + * {@link ZKSignerSecretProvider} - is still a ServletContext attribute under + * the same name, and is still found there by everything that looks for it, + * including DelegationTokenAuthenticationFilter in hadoop-common. + *

+ * This class is the one place in the provider hierarchy that names a servlet + * type, and it goes away with the deprecated + * {@link SignerSecretProvider#init(java.util.Properties, ServletContext, long)} + * it exists to support. + */ +@InterfaceStability.Unstable +@InterfaceAudience.Private +final class ServletSecretProviderContext implements SecretProviderContext { + + private final ServletContext servletContext; + + private ServletSecretProviderContext(ServletContext servletContext) { + this.servletContext = servletContext; + } + + /** + * Returns a context backed by the given ServletContext, or one backed by a + * map of its own when there is no ServletContext. + *

+ * Most callers of the deprecated init pass null - only AuthenticationFilter + * ever passes a real context - and a provider that does not use the store is + * unaffected either way. Handing back a store rather than null keeps a + * provider that does use it from having to check. + * + * @param servletContext the servlet context, or null + * @return an attribute store, never null + */ + static SecretProviderContext of(ServletContext servletContext) { + return servletContext == null + ? new MapSecretProviderContext() + : new ServletSecretProviderContext(servletContext); + } + + @Override + public Object getAttribute(String name) { + return servletContext.getAttribute(name); + } + + @Override + public void setAttribute(String name, Object value) { + servletContext.setAttribute(name, value); + } + + /** + * The store used when there is no ServletContext to write through to. + * Attributes live as long as the provider does and are seen by nothing else, + * which is what passing a null ServletContext already meant. + */ + private static final class MapSecretProviderContext + implements SecretProviderContext { + + private final Map attributes = new HashMap<>(); + + @Override + public Object getAttribute(String name) { + return attributes.get(name); + } + + @Override + public void setAttribute(String name, Object value) { + attributes.put(name, value); + } + } +} diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SignerSecretProvider.java index e937862458e0d0..f187abc90a0512 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SignerSecretProvider.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SignerSecretProvider.java @@ -24,20 +24,66 @@ * do more complicated things in the backend. * See the RolloverSignerSecretProvider class for an implementation that * supports rolling over the secret at a regular interval. + *

+ * Implementations should override + * {@link #initialize(Properties, SecretProviderContext, long)}, which names no + * servlet type. The older {@link #init(Properties, ServletContext, long)} is + * still the entry point callers use and is still honoured when an + * implementation overrides it, so providers written against it keep working + * unchanged; it is deprecated and will be removed with the move to the jakarta + * servlet namespace. */ @InterfaceStability.Unstable @InterfaceAudience.Private public abstract class SignerSecretProvider { /** - * Initialize the SignerSecretProvider + * Initialize the SignerSecretProvider. + *

+ * The default implementation adapts the ServletContext to a + * {@link SecretProviderContext} and calls + * {@link #initialize(Properties, SecretProviderContext, long)}, so a provider + * that overrides only that method is initialized correctly through this + * entry point. A provider that overrides this method instead is called + * directly, as before. + * * @param config configuration properties * @param servletContext servlet context * @param tokenValidity The amount of time a token is valid for * @throws Exception thrown if an error occurred + * @deprecated override + * {@link #initialize(Properties, SecretProviderContext, long)}, + * which does not name a servlet type and so is unaffected by + * which servlet namespace the container provides. */ - public abstract void init(Properties config, ServletContext servletContext, - long tokenValidity) throws Exception; + @Deprecated + public void init(Properties config, ServletContext servletContext, + long tokenValidity) throws Exception { + initialize(config, ServletSecretProviderContext.of(servletContext), + tokenValidity); + } + + /** + * Initialize the SignerSecretProvider against an attribute store. + *

+ * The default implementation throws, because a provider has to implement one + * of the two initialization methods. It is never reached by a provider that + * overrides the deprecated + * {@link #init(Properties, ServletContext, long)}, since callers go through + * that method and its override does not delegate here. + * + * @param config configuration properties + * @param context the attribute store to initialize against + * @param tokenValidity The amount of time a token is valid for + * @throws Exception thrown if an error occurred + */ + public void initialize(Properties config, SecretProviderContext context, + long tokenValidity) throws Exception { + throw new UnsupportedOperationException(getClass().getName() + + " implements neither initialize(Properties, SecretProviderContext," + + " long) nor the deprecated init(Properties, ServletContext, long)"); + } + /** * Will be called on shutdown; subclasses should perform any cleanup here. */ diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java index b0604c85c39f42..853cffbfa9e412 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java @@ -18,7 +18,6 @@ import java.security.SecureRandom; import java.util.Properties; import java.util.Random; -import javax.servlet.ServletContext; import org.apache.curator.framework.CuratorFramework; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; @@ -100,9 +99,11 @@ public class ZKSignerSecretProvider extends RolloverSignerSecretProvider { CONFIG_PREFIX + "disconnect.on.shutdown"; /** - * Constant for the ServletContext attribute that can be used for providing a - * custom CuratorFramework client. If set ZKSignerSecretProvider will use this - * Curator client instead of creating a new one. The providing class is + * Constant for the {@link SecretProviderContext} attribute that can be used + * for providing a custom CuratorFramework client. In a servlet container that + * is the ServletContext attribute of the same name, so the way it is set is + * unchanged. If set ZKSignerSecretProvider will use this Curator client + * instead of creating a new one. The providing class is * responsible for creating and configuring the Curator client (including * security and ACLs) in this case. */ @@ -159,16 +160,16 @@ public ZKSignerSecretProvider(long seed) { } @Override - public void init(Properties config, ServletContext servletContext, + public void initialize(Properties config, SecretProviderContext context, long tokenValidity) throws Exception { - Object curatorClientObj = servletContext.getAttribute( + Object curatorClientObj = context.getAttribute( ZOOKEEPER_SIGNER_SECRET_PROVIDER_CURATOR_CLIENT_ATTRIBUTE); if (curatorClientObj != null && curatorClientObj instanceof CuratorFramework) { client = (CuratorFramework) curatorClientObj; } else { client = createCuratorClient(config); - servletContext.setAttribute( + context.setAttribute( ZOOKEEPER_SIGNER_SECRET_PROVIDER_CURATOR_CLIENT_ATTRIBUTE, client); } this.tokenValidity = tokenValidity; diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java index 9d857640bcf482..ed5587d7a699d4 100644 --- a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java @@ -15,7 +15,6 @@ import java.nio.charset.StandardCharsets; import java.util.Properties; -import javax.servlet.ServletContext; import org.apache.hadoop.classification.VisibleForTesting; import org.apache.hadoop.classification.InterfaceStability; @@ -34,7 +33,7 @@ class StringSignerSecretProvider extends SignerSecretProvider { public StringSignerSecretProvider() {} @Override - public void init(Properties config, ServletContext servletContext, + public void initialize(Properties config, SecretProviderContext context, long tokenValidity) throws Exception { String signatureSecret = config.getProperty( AuthenticationFilter.SIGNATURE_SECRET, null); diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestSignerSecretProviderCompatibility.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestSignerSecretProviderCompatibility.java new file mode 100644 index 00000000000000..768df500e1e7d7 --- /dev/null +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestSignerSecretProviderCompatibility.java @@ -0,0 +1,200 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. See accompanying LICENSE file. + */ +package org.apache.hadoop.security.authentication.util; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.Map; +import java.util.Properties; + +import javax.servlet.ServletContext; + +import org.junit.jupiter.api.Test; + +/** + * A SignerSecretProvider written against the deprecated + * {@link SignerSecretProvider#init(Properties, ServletContext, long)} must keep + * working without being recompiled, because implementations of it live outside + * this tree - signer.secret.provider takes a classname. + *

+ * These tests pin that contract. They are the reason + * {@link SignerSecretProvider#init(Properties, ServletContext, long)} is a + * concrete bridge rather than a changed abstract method, and they go when it + * does. + */ +@SuppressWarnings("deprecation") +public class TestSignerSecretProviderCompatibility { + + private static final byte[] SECRET = + "secret".getBytes(StandardCharsets.UTF_8); + + /** + * A provider as it would have been written before SecretProviderContext + * existed: it overrides init and takes a ServletContext. + */ + private static final class LegacyProvider extends SignerSecretProvider { + private ServletContext seen; + private boolean initCalled; + + @Override + public void init(Properties config, ServletContext servletContext, + long tokenValidity) { + this.seen = servletContext; + this.initCalled = true; + } + + @Override + public byte[] getCurrentSecret() { + return SECRET; + } + + @Override + public byte[][] getAllSecrets() { + return new byte[][]{SECRET}; + } + } + + /** + * The same, extending RolloverSignerSecretProvider and chaining to super as + * such a provider is expected to. + */ + private static final class LegacyRolloverProvider + extends RolloverSignerSecretProvider { + private boolean initCalled; + + @Override + public void init(Properties config, ServletContext servletContext, + long tokenValidity) throws Exception { + this.initCalled = true; + super.init(config, servletContext, tokenValidity); + } + + @Override + protected byte[] generateNewSecret() { + return SECRET; + } + } + + /** A provider written against the servlet-free method. */ + private static final class ContextProvider extends SignerSecretProvider { + private SecretProviderContext seen; + + @Override + public void initialize(Properties config, SecretProviderContext context, + long tokenValidity) { + this.seen = context; + } + + @Override + public byte[] getCurrentSecret() { + return SECRET; + } + + @Override + public byte[][] getAllSecrets() { + return new byte[][]{SECRET}; + } + } + + /** A provider that implements neither method. */ + private static final class UninitializableProvider extends SignerSecretProvider { + @Override + public byte[] getCurrentSecret() { + return SECRET; + } + + @Override + public byte[][] getAllSecrets() { + return new byte[][]{SECRET}; + } + } + + @Test + public void testLegacyProviderStillReceivesTheServletContext() + throws Exception { + ServletContext servletContext = mock(ServletContext.class); + LegacyProvider provider = new LegacyProvider(); + + provider.init(new Properties(), servletContext, 1000); + + assertTrue(provider.initCalled, "the legacy override should have run"); + assertSame(servletContext, provider.seen); + } + + @Test + public void testLegacyRolloverProviderIsStillRolledOver() throws Exception { + LegacyRolloverProvider provider = new LegacyRolloverProvider(); + try { + provider.init(new Properties(), mock(ServletContext.class), 100000); + + // The override ran, and chaining to super still reached + // RolloverSignerSecretProvider.initialize, which seeds the secret. Were + // the bridge dispatched statically, this would silently be null. + assertTrue(provider.initCalled, "the legacy override should have run"); + assertArrayEquals(SECRET, provider.getCurrentSecret()); + } finally { + provider.destroy(); + } + } + + @Test + public void testContextProviderIsReachedThroughTheDeprecatedEntryPoint() + throws Exception { + Map attributes = new HashMap<>(); + ServletContext servletContext = mock(ServletContext.class); + when(servletContext.getAttribute("a")) + .thenAnswer(invocation -> attributes.get("a")); + + ContextProvider provider = new ContextProvider(); + provider.init(new Properties(), servletContext, 1000); + + assertNotNull(provider.seen, "initialize should have been called"); + + // Writes reach the real ServletContext, so an object a provider shares + // this way is still found there by everything that looks for it. + provider.seen.setAttribute("a", "value"); + org.mockito.Mockito.verify(servletContext).setAttribute("a", "value"); + } + + @Test + public void testNullServletContextYieldsAUsableStore() throws Exception { + ContextProvider provider = new ContextProvider(); + provider.init(new Properties(), null, 1000); + + assertNotNull(provider.seen, + "a null ServletContext should still yield a store"); + provider.seen.setAttribute("a", "value"); + assertSame("value", provider.seen.getAttribute("a")); + assertFalse(provider.seen instanceof ServletSecretProviderContext); + } + + @Test + public void testProviderImplementingNeitherMethodFailsLoudly() { + UninitializableProvider provider = new UninitializableProvider(); + + UnsupportedOperationException e = + assertThrows(UnsupportedOperationException.class, + () -> provider.init(new Properties(), null, 1000)); + assertTrue(e.getMessage().contains(UninitializableProvider.class.getName())); + } +} From 47fc7165f57a1608ef009b1ede683d55290db58b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 24 Aug 2026 19:49:53 +0000 Subject: [PATCH 08/30] HADOOP-19971. Stop ApiServiceClient naming a Jetty type. It used org.eclipse.jetty.util.UrlEncoded to percent-encode a user name. java.net.URLEncoder does the same job from the JDK, so the class no longer needs Jetty. Co-Authored-By: Claude --- .../hadoop/yarn/service/client/ApiServiceClient.java | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java index 57d14efcc5a9c8..ca9c14d59563a8 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java @@ -21,6 +21,8 @@ import java.io.File; import java.io.IOException; import java.net.URI; +import java.net.URLEncoder; +import java.nio.charset.StandardCharsets; import java.text.MessageFormat; import java.util.List; import java.util.Map; @@ -62,7 +64,6 @@ import org.apache.hadoop.yarn.service.conf.RestApiConstants; import org.apache.hadoop.yarn.service.utils.ServiceApiUtil; import org.apache.hadoop.yarn.util.RMHAUtils; -import org.eclipse.jetty.util.UrlEncoded; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -226,8 +227,8 @@ private void appendUserNameIfRequired(StringBuilder builder) .equalsIgnoreCase("simple")) { String username = UserGroupInformation.getCurrentUser() .getShortUserName(); - builder.append("?user.name=").append(UrlEncoded - .encodeString(username)); + builder.append("?user.name=").append( + URLEncoder.encode(username, StandardCharsets.UTF_8)); } } From 5a75ac9d875b28548529decd5f9b311c92e38df3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 24 Aug 2026 19:50:08 +0000 Subject: [PATCH 09/30] HADOOP-19971. Stop the shuffle handler naming a Jetty type. ShuffleChannelHandler is a Netty handler. It imported org.eclipse.jetty.http.HttpHeader only to name the Connection and Keep-Alive headers. That was the module's last Jetty reference, so the jetty-http declaration HADOOP-19970 added to its pom goes too. The names are spelled as constants rather than taken from Netty's HttpHeaderNames and HttpHeaderValues, whose constants are lowercase AsciiStrings: using those would have sent "connection: keep-alive" where this handler has always sent "Connection: Keep-Alive". The shuffle response is read by other projects, so no byte of it changes here. Co-Authored-By: Claude --- .../hadoop-mapreduce-client-shuffle/pom.xml | 5 ----- .../hadoop/mapred/ShuffleChannelHandler.java | 17 ++++++++++++----- .../mapred/TestShuffleChannelHandler.java | 8 ++++---- 3 files changed, 16 insertions(+), 14 deletions(-) diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml index cb3428316b65ae..f54f4162a8ddb0 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml @@ -55,11 +55,6 @@ ${leveldbjni.group} leveldbjni-all - - - org.eclipse.jetty - jetty-http - ch.qos.logback logback-classic diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleChannelHandler.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleChannelHandler.java index d0d0e74c9970b0..41f3e906b104e2 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleChannelHandler.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleChannelHandler.java @@ -61,7 +61,6 @@ import org.apache.hadoop.io.SecureIOUtils; import org.apache.hadoop.mapreduce.security.SecureShuffleUtils; import org.apache.hadoop.mapreduce.task.reduce.ShuffleHeader; -import org.eclipse.jetty.http.HttpHeader; import static io.netty.buffer.Unpooled.wrappedBuffer; import static io.netty.handler.codec.http.HttpHeaderNames.CONTENT_TYPE; @@ -128,6 +127,15 @@ * */ public class ShuffleChannelHandler extends SimpleChannelInboundHandler { + /** + * Spelled out rather than taken from a library so that the bytes on the wire + * do not depend on which library supplies the constant: Netty's + * HttpHeaderNames and HttpHeaderValues are lowercase AsciiStrings, and these + * headers have been sent capitalised since this handler was written. + */ + static final String CONNECTION = "Connection"; + static final String KEEP_ALIVE = "Keep-Alive"; + private final ShuffleChannelHandlerContext handlerCtx; ShuffleChannelHandler(ShuffleChannelHandlerContext ctx) { @@ -420,11 +428,10 @@ protected void populateHeaders(List mapIds, String jobId, protected void setResponseHeaders(HttpResponse response, boolean keepAliveParam, long contentLength) { if (!handlerCtx.connectionKeepAliveEnabled && !keepAliveParam) { - response.headers().set(HttpHeader.CONNECTION.asString(), CONNECTION_CLOSE); + response.headers().set(CONNECTION, CONNECTION_CLOSE); } else { - response.headers().set(HttpHeader.CONNECTION.asString(), - HttpHeader.KEEP_ALIVE.asString()); - response.headers().set(HttpHeader.KEEP_ALIVE.asString(), + response.headers().set(CONNECTION, KEEP_ALIVE); + response.headers().set(KEEP_ALIVE, "timeout=" + handlerCtx.connectionKeepAliveTimeOut); } diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleChannelHandler.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleChannelHandler.java index 07ec12a42b8868..e8260337073037 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleChannelHandler.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleChannelHandler.java @@ -81,7 +81,6 @@ import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; import org.apache.hadoop.security.token.SecretManager; import org.apache.hadoop.security.token.Token; -import org.eclipse.jetty.http.HttpHeader; import org.junit.jupiter.api.Test; import org.slf4j.LoggerFactory; @@ -371,10 +370,11 @@ public DefaultHttpResponse getExpectedHttpResponse( headers.set(ShuffleHeader.HTTP_HEADER_NAME, ShuffleHeader.DEFAULT_HTTP_HEADER_NAME); headers.set(ShuffleHeader.HTTP_HEADER_VERSION, ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION); if (keepAlive) { - headers.set(HttpHeader.CONNECTION.asString(), HttpHeader.KEEP_ALIVE.asString()); - headers.set(HttpHeader.KEEP_ALIVE.asString(), "timeout=" + ctx.connectionKeepAliveTimeOut); + headers.set(ShuffleChannelHandler.CONNECTION, + ShuffleChannelHandler.KEEP_ALIVE); + headers.set(ShuffleChannelHandler.KEEP_ALIVE, "timeout=" + ctx.connectionKeepAliveTimeOut); } else { - response.headers().set(HttpHeader.CONNECTION.asString(), CONNECTION_CLOSE); + response.headers().set(ShuffleChannelHandler.CONNECTION, CONNECTION_CLOSE); } HttpUtil.setContentLength(response, contentLength); return response; From b592e5d255b61a02ebefc88dd5b7aec89f5ff580 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 16:57:43 +0000 Subject: [PATCH 10/30] HADOOP-19971. Answer the review of the servlet-free provider changes. Four things the review of this branch raised, none of which changed what the branch does, only what it says and how loudly it says it. parseRSAPublicKey now reports the CertificateException the parse raised rather than the one toRSAPublicKey wraps it in. Routing the deprecated method through the new one had added a level to the cause chain a caller walks: it was ServletException -> CertificateException -> IOException before and had become ServletException -> CertificateException -> CertificateException -> IOException. The chain is what it always was again. A provider given no ServletContext gets a store private to itself, and now says so when something is written to it. Only a provider that shares an object through the store writes there - ZKSignerSecretProvider and its Curator client is the one case here - and there is nothing to share it with, so a second provider in the same JVM builds a second client rather than finding the first. That used to be a NullPointerException, which was at least loud. SignerSecretProvider documents what the move to initialize costs code that reflects over these classes: the providers here no longer declare init, so getDeclaredMethod("init", ...) on a subclass raises NoSuchMethodException where it used to succeed. Calls and overrides resolve as before, and getMethod still finds it. ApiServiceClient records that URLEncoder is not byte-for-byte what Jetty's UrlEncoded produced - they disagree on '~' and '*' - and that both forms decode to the same user name. Tests pin the cause chain and the unshared store. Co-Authored-By: Claude --- .../authentication/util/CertificateUtil.java | 7 +++- .../util/ServletSecretProviderContext.java | 24 +++++++++++++- .../util/SignerSecretProvider.java | 11 +++++++ .../util/TestCertificateUtil.java | 33 +++++++++++++++++++ ...TestSignerSecretProviderCompatibility.java | 28 ++++++++++++++++ .../yarn/service/client/ApiServiceClient.java | 5 +++ 6 files changed, 106 insertions(+), 2 deletions(-) diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/CertificateUtil.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/CertificateUtil.java index 068d7a3c383f94..13840948ab349f 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/CertificateUtil.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/CertificateUtil.java @@ -50,7 +50,12 @@ public static RSAPublicKey parseRSAPublicKey(String pem) try { return toRSAPublicKey(pem); } catch (CertificateException ce) { - throw new ServletException(ce.getMessage(), ce); + // Report the exception toRSAPublicKey wrapped, not the wrapper, so that + // the cause a caller sees is the one this method has always reported: + // the CertificateException the parse itself raised. Wrapping the wrapper + // would add a level to the chain that was not there before. + Throwable cause = ce.getCause() == null ? ce : ce.getCause(); + throw new ServletException(ce.getMessage(), cause); } } diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ServletSecretProviderContext.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ServletSecretProviderContext.java index e7f576f7fc59ae..317eeb541453e9 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ServletSecretProviderContext.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ServletSecretProviderContext.java @@ -18,6 +18,9 @@ import javax.servlet.ServletContext; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.classification.VisibleForTesting; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * Presents a ServletContext to a {@link SignerSecretProvider} as a @@ -38,6 +41,9 @@ @InterfaceAudience.Private final class ServletSecretProviderContext implements SecretProviderContext { + private static final Logger LOG = + LoggerFactory.getLogger(ServletSecretProviderContext.class); + private final ServletContext servletContext; private ServletSecretProviderContext(ServletContext servletContext) { @@ -77,7 +83,8 @@ public void setAttribute(String name, Object value) { * Attributes live as long as the provider does and are seen by nothing else, * which is what passing a null ServletContext already meant. */ - private static final class MapSecretProviderContext + @VisibleForTesting + static final class MapSecretProviderContext implements SecretProviderContext { private final Map attributes = new HashMap<>(); @@ -87,8 +94,23 @@ public Object getAttribute(String name) { return attributes.get(name); } + /** + * Stores the attribute, and says so. + *

+ * A provider only writes here to share an object with the rest of the + * application - the CuratorFramework client of + * {@link ZKSignerSecretProvider} is the one case in this tree. There is + * nothing to share it with: this store is private to the provider, so a + * second provider in the same JVM builds a second client rather than + * finding this one. Before there was a store at all a null ServletContext + * threw, which was at least loud; this keeps it visible. + */ @Override public void setAttribute(String name, Object value) { + LOG.warn("SignerSecretProvider was initialized without a ServletContext," + + " so the attribute {} is being stored in a map private to this" + + " provider. Anything that expects to share it - including another" + + " provider in this JVM - will not find it there.", name); attributes.put(name, value); } } diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SignerSecretProvider.java index f187abc90a0512..ff5acc210f7805 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SignerSecretProvider.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/SignerSecretProvider.java @@ -32,6 +32,17 @@ * implementation overrides it, so providers written against it keep working * unchanged; it is deprecated and will be removed with the move to the jakarta * servlet namespace. + *

+ * One consequence for code that reflects over these classes: the providers + * shipped here override + * {@link #initialize(Properties, SecretProviderContext, long)} rather than + * {@link #init(Properties, ServletContext, long)}, so init is no longer among + * their declared methods. Calls and overrides are unaffected - init is + * inherited from this class and resolves as it always did - but + * Class#getDeclaredMethod("init", ...) on a subclass such as + * {@link FileSignerSecretProvider} now raises NoSuchMethodException where it + * used to succeed. Class#getMethod, which searches superclasses, still finds + * it. */ @InterfaceStability.Unstable @InterfaceAudience.Private diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestCertificateUtil.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestCertificateUtil.java index 5ffbad88cd5031..92851654e4e041 100644 --- a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestCertificateUtil.java +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestCertificateUtil.java @@ -18,6 +18,7 @@ package org.apache.hadoop.security.authentication.util; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.junit.jupiter.api.Assertions.fail; @@ -114,4 +115,36 @@ public void testDeprecatedEntryPointStillThrowsServletException() { } } + /** + * The deprecated entry point reports the CertificateException the parse + * raised, not the one toRSAPublicKey wraps it in. Routing through the new + * method must not lengthen the chain a caller walks. + */ + @Test + @SuppressWarnings("deprecation") + public void testDeprecatedEntryPointDoesNotDeepenTheCauseChain() { + try { + CertificateUtil.parseRSAPublicKey("not a certificate"); + fail("Should not have thrown ServletException"); + } catch (ServletException se) { + Throwable cause = se.getCause(); + assertNotNull(cause, "the ServletException should carry a cause"); + assertTrue(cause instanceof CertificateException, + "expected a CertificateException, got " + cause.getClass()); + assertFalse(cause.getCause() instanceof CertificateException, + "the wrapper toRSAPublicKey adds must not appear in the chain;" + + " chain was " + chainOf(se)); + } + } + + private static String chainOf(Throwable t) { + StringBuilder sb = new StringBuilder(); + for (Throwable c = t; c != null; c = c.getCause()) { + if (sb.length() > 0) { + sb.append(" -> "); + } + sb.append(c.getClass().getSimpleName()); + } + return sb.toString(); + } } diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestSignerSecretProviderCompatibility.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestSignerSecretProviderCompatibility.java index 768df500e1e7d7..2a515889da46b0 100644 --- a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestSignerSecretProviderCompatibility.java +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestSignerSecretProviderCompatibility.java @@ -16,6 +16,7 @@ import static org.junit.jupiter.api.Assertions.assertArrayEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertSame; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -188,6 +189,33 @@ public void testNullServletContextYieldsAUsableStore() throws Exception { assertFalse(provider.seen instanceof ServletSecretProviderContext); } + /** + * A provider that shares an object through the store - which in this tree + * means ZKSignerSecretProvider and its Curator client - gets a store private + * to itself when there is no ServletContext, so a second provider does not + * find what the first one put there. That is a real change from the + * NullPointerException a null ServletContext used to raise, so it is pinned + * here rather than left to be discovered. + */ + @Test + public void testNullServletContextDoesNotShareBetweenProviders() + throws Exception { + ContextProvider first = new ContextProvider(); + ContextProvider second = new ContextProvider(); + first.init(new Properties(), null, 1000); + second.init(new Properties(), null, 1000); + + first.seen.setAttribute("shared", "from-first"); + + assertNull(second.seen.getAttribute("shared"), + "a store backed by no ServletContext is private to its provider"); + assertTrue( + first.seen instanceof ServletSecretProviderContext + .MapSecretProviderContext, + "expected the private map store, got " + + first.seen.getClass().getName()); + } + @Test public void testProviderImplementingNeitherMethodFailsLoudly() { UninitializableProvider provider = new UninitializableProvider(); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java index ca9c14d59563a8..b85169b6460165 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java @@ -227,6 +227,11 @@ private void appendUserNameIfRequired(StringBuilder builder) .equalsIgnoreCase("simple")) { String username = UserGroupInformation.getCurrentUser() .getShortUserName(); + // Not byte-for-byte what Jetty's UrlEncoded produced: it left '~' + // literal and escaped '*', and URLEncoder does the opposite. Both forms + // decode to the same user name, so what the server reads is unchanged; + // every other character a short user name can hold, '@' and non-ASCII + // among them, encodes identically. builder.append("?user.name=").append( URLEncoder.encode(username, StandardCharsets.UTF_8)); } From 83a51ffc04970015892452fff695fd2501f27e5d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 24 Aug 2026 20:12:43 +0000 Subject: [PATCH 11/30] HADOOP-19972. Upgrade to Jetty 12 on the ee8 environment, keeping javax.servlet. Jetty 9.4 is end of life and 12 is the supported line. Jetty 12 ships the servlet container as a pluggable EE environment; ee8 is the one that serves javax.servlet, so the tree moves to 12 without touching the namespace, Jersey 2, or any public signature. Dependency management moves to 12.0.37. The core artifacts keep their coordinates; the servlet container moves under org.eclipse.jetty.ee8, and the HTTP/2 artifacts the catalog webapp pins gain the jetty- prefix Jetty 12 gave them. jetty-servlet-tester goes: it does not exist on the 12.x line and no module declared it. The servlet API becomes org.eclipse.jetty.toolchain:jetty-servlet-api. It publishes the same javax.servlet packages as the jakarta.servlet-api it replaces, so the namespace and HADOOP-19970's one-API-per-classpath rule both hold, but it is the only one of the two that also carries the web.xml schemas, which ee8 resolves from whichever archive javax.servlet.Servlet came from. Grizzly's own jakarta.servlet-api is excluded for the same reason. HttpServer2, HttpServer2Metrics, the YARN WebApps builder, the WebSocket code and the remaining embedded servers are ported to the ee8 API. The shaded client artifacts and LICENSE-binary follow the new coordinates. Four things fall out of the move: * hadoop-client-minicluster excluded the servlet API from the Hadoop modules it pulls back in by naming jakarta.servlet:jakarta.servlet-api. Those exclusions stop matching, which puts the 85 javax.servlet classes in both shaded jars - caught by BanDuplicateClasses in hadoop-client-check-test-invariants. They name the new coordinate too. * Jetty 12's DefaultServlet defaults dirAllowed to true, where 9.4's init parameter had it off for this context. It is set explicitly, and asserted, so no directory listing is served from /static. * Jetty 9.4's WebSocketServlet bootstrapped itself from init(). Jetty 12's JettyWebSocketServlet expects the WebSocket components on the context already and throws otherwise, so the node manager's container shell endpoint answered 500 on upgrade. What installs them is a ServletContainerInitializer, discovered only by AnnotationConfiguration, which lives in jetty-ee8-annotations - excluded here because it drags in an asm the enforcer bans. Calling the initializer directly does the same work without the scan, and has to happen while the context is still stopped, so the WebApps builder gains a window between build and start. TestNMContainerWebSocket caught none of this because it wrapped the exchange in catch (Throwable) and logged; it now asserts the upgrade produced a session. * hadoop-common's exclusion of javax.servlet-api from jetty-util matches no edge, because Jetty 12's jetty-util depends on nothing but slf4j, and the Router pom's comment described jetty-servlet and jakarta.servlet-api, neither of which this tree has any more. Co-Authored-By: Claude --- LICENSE-binary | 37 +++--- NOTICE-binary | 1 - .../hadoop-client-minicluster/pom.xml | 35 +++++- .../hadoop-client-runtime/pom.xml | 15 ++- hadoop-common-project/hadoop-auth/pom.xml | 4 +- .../server/AuthenticationFilter.java | 8 +- .../client/AuthenticatorTestCase.java | 6 +- .../client/TestPseudoAuthenticator.java | 20 +++- hadoop-common-project/hadoop-common/pom.xml | 14 +-- .../hadoop/http/AdminAuthorizedServlet.java | 2 +- .../org/apache/hadoop/http/HttpServer2.java | 107 +++++++++++------ .../hadoop/http/HttpServer2Metrics.java | 46 +++---- .../hadoop/http/ProfileOutputServlet.java | 2 +- .../org/apache/hadoop/http/WebServlet.java | 2 +- .../http/RestCsrfPreventionFilter.java | 7 +- .../DelegationTokenAuthenticationHandler.java | 10 +- .../hadoop/fs/FSMainOperationsBaseTest.java | 8 +- .../fs/viewfs/ViewFileSystemTestSetup.java | 14 ++- .../hadoop/fs/viewfs/ViewFsTestSetup.java | 14 ++- .../http/TestAuthenticationSessionCookie.java | 10 +- .../apache/hadoop/http/TestHttpServer.java | 65 ++++++++-- .../web/TestWebDelegationToken.java | 6 +- hadoop-common-project/hadoop-kms/pom.xml | 4 +- .../kms/server/KMSAuthenticationFilter.java | 14 +-- .../hadoop-hdfs-httpfs/pom.xml | 4 +- .../fs/http/client/BaseTestHttpFSWith.java | 2 +- .../server/TestHttpFSAccessControlled.java | 2 +- .../fs/http/server/TestHttpFSServer.java | 2 +- .../http/server/TestHttpFSServerNoACLs.java | 2 +- .../http/server/TestHttpFSServerNoXAttrs.java | 2 +- .../http/server/TestHttpFSWithKerberos.java | 2 +- .../apache/hadoop/test/TestHFSTestCase.java | 2 +- .../org/apache/hadoop/test/TestHTestCase.java | 2 +- .../hdfs/server/namenode/ImageServlet.java | 7 +- .../hadoop/mapred/NotificationTestCase.java | 4 +- hadoop-project/pom.xml | 113 ++++++++++-------- hadoop-tools/hadoop-sls/pom.xml | 16 ++- .../apache/hadoop/yarn/sls/web/SLSWebApp.java | 14 ++- .../pom.xml | 52 +++++++- .../hadoop-yarn-services-api/pom.xml | 8 +- .../yarn/service/webapp/ApiServerWebApp.java | 2 +- .../service/client/TestApiServiceClient.java | 4 +- .../client/TestSecureApiServiceClient.java | 4 +- .../hadoop-yarn/hadoop-yarn-client/pom.xml | 4 +- .../client/api/ContainerShellWebSocket.java | 40 +++++-- .../yarn/client/api/impl/YarnClientImpl.java | 8 +- .../yarn/client/api/impl/TestAMRMClient.java | 10 +- .../hadoop/yarn/client/cli/TestYarnCLI.java | 9 +- .../apache/hadoop/yarn/webapp/WebApps.java | 29 ++++- .../pom.xml | 8 +- .../ApplicationHistoryServer.java | 6 +- .../TestRollingLevelDBTimelineStore.java | 10 +- .../hadoop-yarn-server-nodemanager/pom.xml | 14 ++- .../amrmproxy/FederationInterceptor.java | 3 +- .../webapp/ContainerShellWebSocket.java | 31 +++-- .../ContainerShellWebSocketServlet.java | 8 +- .../nodemanager/webapp/TerminalServlet.java | 2 +- .../server/nodemanager/webapp/WebServer.java | 11 ++ .../docker/TestDockerCommandExecutor.java | 6 +- .../TestLogAggregationService.java | 4 +- .../ContainerShellClientSocketTest.java | 6 +- .../webapp/TestNMContainerWebSocket.java | 41 ++++--- .../resourcemanager/ResourceManager.java | 2 +- .../yarn/server/resourcemanager/MockNM.java | 8 +- .../hadoop-yarn-server-router/pom.xml | 23 ++-- .../hadoop/yarn/server/router/Router.java | 2 +- .../hadoop/yarn/server/router/TestRouter.java | 6 +- .../router/webapp/TestRouterWebAppProxy.java | 4 +- .../pom.xml | 8 +- .../webproxy/TestWebAppProxyServlet.java | 4 +- .../webproxy/TestWebAppProxyServletFed.java | 4 +- .../webproxy/amfilter/TestAmFilter.java | 4 +- 72 files changed, 659 insertions(+), 351 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index c3c0a92ca4d292..5f30fc70a08f0c 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -411,21 +411,29 @@ org.apache.yetus:audience-annotations:0.5.0 org.apache.zookeeper:zookeeper:3.8.6 org.codehaus.jettison:jettison:1.5.4 org.conscrypt:conscrypt-openjdk-uber:2.5.2 -org.eclipse.jetty:jetty-annotations:9.4.58.v20250814 -org.eclipse.jetty:jetty-http:9.4.58.v20250814 -org.eclipse.jetty:jetty-io:9.4.58.v20250814 -org.eclipse.jetty:jetty-jndi:9.4.58.v20250814 -org.eclipse.jetty:jetty-plus:9.4.58.v20250814 -org.eclipse.jetty:jetty-security:9.4.58.v20250814 -org.eclipse.jetty:jetty-server:9.4.58.v20250814 -org.eclipse.jetty:jetty-servlet:9.4.58.v20250814 -org.eclipse.jetty:jetty-util:9.4.58.v20250814 -org.eclipse.jetty:jetty-util-ajax:9.4.58.v20250814 -org.eclipse.jetty:jetty-webapp:9.4.58.v20250814 -org.eclipse.jetty:jetty-xml:9.4.58.v20250814 +org.eclipse.jetty:jetty-alpn-client:12.0.37 +org.eclipse.jetty:jetty-client:12.0.37 +org.eclipse.jetty:jetty-ee:12.0.37 +org.eclipse.jetty:jetty-http:12.0.37 +org.eclipse.jetty:jetty-io:12.0.37 +org.eclipse.jetty:jetty-security:12.0.37 +org.eclipse.jetty:jetty-server:12.0.37 +org.eclipse.jetty:jetty-session:12.0.37 +org.eclipse.jetty:jetty-util:12.0.37 +org.eclipse.jetty:jetty-xml:12.0.37 +org.eclipse.jetty.ee8:jetty-ee8-nested:12.0.37 +org.eclipse.jetty.ee8:jetty-ee8-security:12.0.37 +org.eclipse.jetty.ee8:jetty-ee8-servlet:12.0.37 +org.eclipse.jetty.ee8:jetty-ee8-webapp:12.0.37 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-api:12.0.37 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-client:12.0.37 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-common:12.0.37 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-server:12.0.37 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-servlet:12.0.37 org.eclipse.jetty.toolchain:jetty-servlet-api:4.0.9 -org.eclipse.jetty.websocket:javax-websocket-client-impl:9.4.58.v20250814 -org.eclipse.jetty.websocket:javax-websocket-server-impl:9.4.58.v20250814 +org.eclipse.jetty.websocket:jetty-websocket-core-client:12.0.37 +org.eclipse.jetty.websocket:jetty-websocket-core-common:12.0.37 +org.eclipse.jetty.websocket:jetty-websocket-core-server:12.0.37 org.ehcache:ehcache:3.8.2 org.ini4j:ini4j:0.5.4 org.objenesis:objenesis:2.6 @@ -627,7 +635,6 @@ com.sun.xml.bind:jaxb-impl:2.2.3-1 javax.annotation:javax.annotation-api:1.3.2 javax.cache:cache-api:1.1.1 javax.servlet.jsp:jsp-api:2.1 -javax.websocket:javax.websocket-api:1.0 Eclipse Distribution License (EDL) 1.0 -------------------------- diff --git a/NOTICE-binary b/NOTICE-binary index b3f7359b96ae2d..6ac4e1918ef97d 100644 --- a/NOTICE-binary +++ b/NOTICE-binary @@ -465,7 +465,6 @@ https://glassfish.dev.java.net/nonav/public/CDDL+GPL.html * javax.annotation:javax.annotation-api * javax.transaction:javax.transaction-api - * javax.websocket:javax.websocket-api ------ Oracle OpenJDK diff --git a/hadoop-client-modules/hadoop-client-minicluster/pom.xml b/hadoop-client-modules/hadoop-client-minicluster/pom.xml index e2e9588a668048..9ddc124fb1fd0e 100644 --- a/hadoop-client-modules/hadoop-client-minicluster/pom.xml +++ b/hadoop-client-modules/hadoop-client-minicluster/pom.xml @@ -127,6 +127,10 @@ org.eclipse.jetty.toolchain jetty-servlet-api + + org.eclipse.jetty.toolchain + jetty-servlet-api + jakarta.ws.rs jakarta.ws.rs-api @@ -479,6 +483,10 @@ org.eclipse.jetty.toolchain jetty-servlet-api + + org.eclipse.jetty.toolchain + jetty-servlet-api + jakarta.ws.rs jakarta.ws.rs-api @@ -554,6 +562,10 @@ org.eclipse.jetty.toolchain jetty-servlet-api + + org.eclipse.jetty.toolchain + jetty-servlet-api + jakarta.ws.rs jakarta.ws.rs-api @@ -623,6 +635,10 @@ jakarta.servlet jakarta.servlet-api + + org.eclipse.jetty.toolchain + jetty-servlet-api + junit-jupiter org.junit.jupiter @@ -634,6 +650,15 @@ org.glassfish.jersey.test-framework.providers jersey-test-framework-provider-jdk-http + + + jakarta.servlet + jakarta.servlet-api + + + org.eclipse.jetty.toolchain + jetty-servlet-api + jakarta.ws.rs jakarta.ws.rs-api @@ -707,8 +732,14 @@ junit:junit com.google.code.findbugs:jsr305 ch.qos.reload4j:reload4j - org.eclipse.jetty.websocket:websocket-common - org.eclipse.jetty.websocket:websocket-api + + org.eclipse.jetty.websocket:jetty-websocket-core-common + org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-api + org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-common org.bouncycastle:* diff --git a/hadoop-client-modules/hadoop-client-runtime/pom.xml b/hadoop-client-modules/hadoop-client-runtime/pom.xml index 40d2c9e883a855..3b9e6a7d133d47 100644 --- a/hadoop-client-modules/hadoop-client-runtime/pom.xml +++ b/hadoop-client-modules/hadoop-client-runtime/pom.xml @@ -161,15 +161,26 @@ com.google.code.findbugs:jsr305 io.netty:* io.dropwizard.metrics:metrics-core - org.eclipse.jetty:jetty-servlet org.eclipse.jetty:jetty-security org.eclipse.jetty:jetty-util - org.eclipse.jetty:jetty-util-ajax org.eclipse.jetty:jetty-server + org.eclipse.jetty:jetty-session + org.eclipse.jetty.ee8:jetty-ee8-servlet + org.eclipse.jetty.ee8:jetty-ee8-nested + org.eclipse.jetty.ee8:jetty-ee8-security org.ow2.asm:* org.bouncycastle:* diff --git a/hadoop-common-project/hadoop-auth/pom.xml b/hadoop-common-project/hadoop-auth/pom.xml index 4588a15f8cd8d0..9bab9991929123 100644 --- a/hadoop-common-project/hadoop-auth/pom.xml +++ b/hadoop-common-project/hadoop-auth/pom.xml @@ -57,8 +57,8 @@ test - org.eclipse.jetty - jetty-servlet + org.eclipse.jetty.ee8 + jetty-ee8-servlet test diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java index 7cc70c493c0f66..3549ef1ed9c42a 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java @@ -621,8 +621,11 @@ && getMaxInactiveInterval() > 0) { KerberosAuthenticator.WWW_AUTHENTICATE.toLowerCase()))) { errCode = HttpServletResponse.SC_FORBIDDEN; } - // After Jetty 9.4.21, sendError() no longer allows a custom message. - // use setStatus() to set a custom message. + // The reason phrase is not a place to put this any more. Jetty 12 + // stores what ee8's Response.setStatusWithReason is given and never + // writes it to the wire, and setStatus(int, String) has ignored its + // message since Servlet 3.0 deprecated it. sendError puts the detail + // in the response body, which is where a client can still read it. String reason; if (authenticationEx == null) { reason = "Authentication required"; @@ -630,7 +633,6 @@ && getMaxInactiveInterval() > 0) { reason = authenticationEx.getMessage(); } - httpResponse.setStatus(errCode, reason); httpResponse.sendError(errCode, reason); } } diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java index 14538b873c803c..c6137b4ee08cb9 100644 --- a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java @@ -35,9 +35,9 @@ import org.eclipse.jetty.server.Connector; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.servlet.FilterHolder; -import org.eclipse.jetty.servlet.ServletContextHandler; -import org.eclipse.jetty.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.FilterHolder; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; import javax.servlet.DispatcherType; import javax.servlet.FilterConfig; diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestPseudoAuthenticator.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestPseudoAuthenticator.java index 43f38f2594e263..b944857656c872 100644 --- a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestPseudoAuthenticator.java +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestPseudoAuthenticator.java @@ -20,9 +20,12 @@ import org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler; import org.junit.jupiter.api.Test; +import java.io.InputStream; import java.net.HttpURLConnection; import java.net.URL; +import java.nio.charset.StandardCharsets; import java.util.Properties; +import java.util.Scanner; public class TestPseudoAuthenticator { @@ -67,7 +70,9 @@ public void testAnonymousDisallowed() throws Exception { conn.connect(); assertEquals(HttpURLConnection.HTTP_UNAUTHORIZED, conn.getResponseCode()); assertTrue(conn.getHeaderFields().containsKey("WWW-Authenticate")); - assertEquals("Authentication required", conn.getResponseMessage()); + // Jetty 12 sends the canonical reason phrase whatever the server sets, + // so the detail is read out of the body instead. + assertTrue(readBody(conn).contains("Authentication required")); } finally { auth.stop(); } @@ -105,4 +110,17 @@ public void testAuthenticationAnonymousDisallowedWithPost() throws Exception { auth._testAuthentication(new PseudoAuthenticator(), true); } + private static String readBody(HttpURLConnection conn) { + InputStream in = conn.getErrorStream() != null + ? conn.getErrorStream() : null; + if (in == null) { + return ""; + } + try (Scanner scanner = new Scanner(in, StandardCharsets.UTF_8.name())) { + // \A matches only at the start of input, so the whole body is one token. + scanner.useDelimiter("\\A"); + return scanner.hasNext() ? scanner.next() : ""; + } + } + } diff --git a/hadoop-common-project/hadoop-common/pom.xml b/hadoop-common-project/hadoop-common/pom.xml index 3f50fd7d47ac68..23d5db114d2d87 100644 --- a/hadoop-common-project/hadoop-common/pom.xml +++ b/hadoop-common-project/hadoop-common/pom.xml @@ -141,21 +141,15 @@ org.eclipse.jetty jetty-util compile - - - javax.servlet-api - javax.servlet - - - org.eclipse.jetty - jetty-servlet + org.eclipse.jetty.ee8 + jetty-ee8-servlet compile - org.eclipse.jetty - jetty-webapp + org.eclipse.jetty.ee8 + jetty-ee8-webapp compile diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/AdminAuthorizedServlet.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/AdminAuthorizedServlet.java index a4b05a1d534b85..8b17db7558f462 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/AdminAuthorizedServlet.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/AdminAuthorizedServlet.java @@ -23,7 +23,7 @@ import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; -import org.eclipse.jetty.servlet.DefaultServlet; +import org.eclipse.jetty.ee8.servlet.DefaultServlet; /** * General servlet which is admin-authorized. diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java index d17874d78aca82..0d54e2b2e77d55 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java @@ -89,6 +89,15 @@ import org.apache.hadoop.util.ReflectionUtils; import org.apache.hadoop.util.Shell; import org.apache.hadoop.util.StringUtils; +import org.eclipse.jetty.ee8.nested.ErrorHandler; +import org.eclipse.jetty.ee8.nested.SessionHandler; +import org.eclipse.jetty.ee8.servlet.FilterHolder; +import org.eclipse.jetty.ee8.servlet.FilterMapping; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.ServletMapping; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import org.eclipse.jetty.http.HttpVersion; import org.eclipse.jetty.server.ConnectionFactory; import org.eclipse.jetty.server.Connector; @@ -102,21 +111,11 @@ import org.eclipse.jetty.server.SslConnectionFactory; import org.eclipse.jetty.server.SymlinkAllowedResourceAliasChecker; import org.eclipse.jetty.server.handler.ContextHandlerCollection; -import org.eclipse.jetty.server.handler.HandlerCollection; -import org.eclipse.jetty.server.handler.RequestLogHandler; import org.eclipse.jetty.server.handler.StatisticsHandler; -import org.eclipse.jetty.server.session.SessionHandler; -import org.eclipse.jetty.servlet.FilterHolder; -import org.eclipse.jetty.servlet.FilterMapping; -import org.eclipse.jetty.servlet.ServletContextHandler; -import org.eclipse.jetty.servlet.ServletHandler; -import org.eclipse.jetty.servlet.ServletHolder; -import org.eclipse.jetty.servlet.ServletMapping; import org.eclipse.jetty.util.ArrayUtil; -import org.eclipse.jetty.util.MultiException; +import org.eclipse.jetty.util.ExceptionUtil; import org.eclipse.jetty.util.ssl.SslContextFactory; import org.eclipse.jetty.util.thread.QueuedThreadPool; -import org.eclipse.jetty.webapp.WebAppContext; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -173,6 +172,19 @@ public final class HttpServer2 implements FilterContainer { = "hadoop.http.sni.host.check.enabled"; public static final boolean HTTP_SNI_HOST_CHECK_ENABLED_DEFAULT = false; + /** + * Prefix under which a context init parameter reaches Jetty's DefaultServlet. + *

+ * This is not the package the servlet lives in: ee8 moved the class to + * org.eclipse.jetty.ee8.servlet but DefaultServlet#getInitParameter still + * reads the 9.4 name. Spelling it the other way costs nothing at startup and + * silently drops the setting - dirAllowed then falls back to its default of + * true and /static starts listing directories - so the prefix is written + * once here rather than at each call site. + */ + private static final String DEFAULT_SERVLET_INIT_PREFIX = + "org.eclipse.jetty.servlet.Default."; + // The ServletContext attribute where the daemon Configuration // gets stored. public static final String CONF_CONTEXT_ATTRIBUTE = "hadoop.conf"; @@ -186,7 +198,7 @@ public final class HttpServer2 implements FilterContainer { protected final Server webServer; - private final HandlerCollection handlers; + private final Handler.Sequence handlers; private final List listeners = Lists.newArrayList(); @@ -739,7 +751,7 @@ private HttpServer2(final Builder b) throws IOException { final String appDir = getWebAppsPath(b.name); this.webServer = new Server(); this.adminsAcl = b.adminsAcl; - this.handlers = new HandlerCollection(); + this.handlers = new Handler.Sequence(); this.webAppContext = createWebAppContext(b, adminsAcl, appDir); this.xFrameOptionIsEnabled = b.xFrameEnabled; this.xFrameOption = b.xFrameOption; @@ -767,7 +779,19 @@ private void initializeWebServer(String name, String hostName, throws IOException { Preconditions.checkNotNull(webAppContext); - webAppContext.getErrorHandler().setShowStacks(LOG.isTraceEnabled()); + // Jetty only builds an error page for GET, HEAD and POST, so an error on a + // PUT or a DELETE goes back with no body at all. That did not show on 9.4 + // because the detail also travelled in the reason phrase, which Jetty 12 + // no longer puts on the wire, and losing both leaves a client with nothing + // but the status code. + ErrorHandler errorHandler = new ErrorHandler() { + @Override + public boolean errorPageForMethod(String method) { + return true; + } + }; + errorHandler.setShowStacks(LOG.isTraceEnabled()); + webAppContext.setErrorHandler(errorHandler); int maxThreads = conf.getInt(HTTP_MAX_THREADS_KEY, -1); // If HTTP_MAX_THREADS is not configured, QueueThreadPool() will use the @@ -788,10 +812,13 @@ private void initializeWebServer(String name, String hostName, handlers.addHandler(contexts); if (requestLog != null) { - RequestLogHandler requestLogHandler = new RequestLogHandler(); - requestLogHandler.setRequestLog(requestLog); - handlers.addHandler(requestLogHandler); + // Jetty 12 removed RequestLogHandler. A request log is now a property of + // the server and covers every request it handles, which is what wrapping + // the whole handler list amounted to before. + webServer.setRequestLog(requestLog); } + // An ee8 context is not a core Handler; it supplies one. Handler.Collection + // takes the Supplier directly and adds the core handler behind it. handlers.addHandler(webAppContext); final String appDir = getWebAppsPath(name); addDefaultApps(contexts, appDir, conf); @@ -804,9 +831,8 @@ private void initializeWebServer(String name, String hostName, // The tree might look like this: // // - StatisticsHandler (for all requests) - // - HandlerList + // - Handler.Sequence // - ContextHandlerCollection - // - RequestLogHandler (if enabled) // - WebAppContext // - SessionHandler // - Servlets @@ -971,6 +997,16 @@ protected void addDefaultApps(ContextHandlerCollection parent, boolean logsEnabled = conf.getBoolean( CommonConfigurationKeys.HADOOP_HTTP_LOGS_ENABLED, CommonConfigurationKeys.HADOOP_HTTP_LOGS_ENABLED_DEFAULT); + // Jetty 9.4 let a context start on a base resource that was not there and + // served 404s from it. Jetty 12 rejects it in ContextHandler#doStart, which + // turns a missing log directory into a server that will not come up at all, + // so the context is left out rather than allowed to fail the daemon. The + // endpoint answers 404 either way. + if (logDir != null && logsEnabled && !new File(logDir).isDirectory()) { + LOG.warn("Not adding the /logs context: hadoop.log.dir is set to {}," + + " which is not a directory.", logDir); + logsEnabled = false; + } if (logDir != null && logsEnabled) { ServletContextHandler logContext = new ServletContextHandler(parent, "/logs"); @@ -981,14 +1017,15 @@ protected void addDefaultApps(ContextHandlerCollection parent, CommonConfigurationKeys.DEFAULT_HADOOP_JETTY_LOGS_SERVE_ALIASES)) { @SuppressWarnings("unchecked") Map params = logContext.getInitParams(); - params.put("org.eclipse.jetty.servlet.Default.aliases", "true"); + params.put(DEFAULT_SERVLET_INIT_PREFIX + "aliases", "true"); } logContext.setDisplayName("logs"); SessionHandler handler = new SessionHandler(); handler.setHttpOnly(true); handler.getSessionCookieConfig().setSecure(true); logContext.setSessionHandler(handler); - logContext.addAliasCheck(new SymlinkAllowedResourceAliasChecker(logContext)); + logContext.addAliasCheck( + new SymlinkAllowedResourceAliasChecker(logContext.getCoreContextHandler())); setContextAttributes(logContext, conf); addNoCacheFilter(logContext); defaultContexts.put(logContext, true); @@ -1001,13 +1038,14 @@ protected void addDefaultApps(ContextHandlerCollection parent, staticContext.setDisplayName("static"); @SuppressWarnings("unchecked") Map params = staticContext.getInitParams(); - params.put("org.eclipse.jetty.servlet.Default.dirAllowed", "false"); - params.put("org.eclipse.jetty.servlet.Default.gzip", "true"); + params.put(DEFAULT_SERVLET_INIT_PREFIX + "dirAllowed", "false"); + params.put(DEFAULT_SERVLET_INIT_PREFIX + "gzip", "true"); SessionHandler handler = new SessionHandler(); handler.setHttpOnly(true); handler.getSessionCookieConfig().setSecure(true); staticContext.setSessionHandler(handler); - staticContext.addAliasCheck(new SymlinkAllowedResourceAliasChecker(staticContext)); + staticContext.addAliasCheck(new SymlinkAllowedResourceAliasChecker( + staticContext.getCoreContextHandler())); setContextAttributes(staticContext, conf); defaultContexts.put(staticContext, true); } @@ -1208,8 +1246,8 @@ public void addInternalServlet(String name, String pathSpec, * @param handler The handler to add */ public void addHandlerAtFront(Handler handler) { - Handler[] h = ArrayUtil.prependToArray( - handler, this.handlers.getHandlers(), Handler.class); + List h = new ArrayList<>(this.handlers.getHandlers()); + h.add(0, handler); handlers.setHandlers(h); } @@ -1464,13 +1502,9 @@ public void start() throws IOException { } catch (IOException ex) { LOG.info("HttpServer.start() threw a non Bind IOException", ex); throw ex; - } catch (MultiException ex) { - LOG.info("HttpServer.start() threw a MultiException", ex); - throw ex; } // Make sure there is no handler failures. - Handler[] hs = webServer.getHandlers(); - for (Handler handler : hs) { + for (Handler handler : webServer.getHandlers()) { if (handler.isFailed()) { throw new IOException( "Problem in starting http server. Server handlers failed"); @@ -1619,7 +1653,7 @@ void openListeners() throws Exception { * @throws Exception exception. */ public void stop() throws Exception { - MultiException exception = null; + ExceptionUtil.MultiException exception = null; if (this.configurationChangeMonitor.isPresent()) { try { this.configurationChangeMonitor.get().cancel(); @@ -1670,9 +1704,10 @@ public void stop() throws Exception { } - private MultiException addMultiException(MultiException exception, Exception e) { + private ExceptionUtil.MultiException addMultiException( + ExceptionUtil.MultiException exception, Exception e) { if(exception == null){ - exception = new MultiException(); + exception = new ExceptionUtil.MultiException(); } exception.add(e); return exception; @@ -1960,9 +1995,7 @@ public void doFilter(ServletRequest request, */ private String inferMimeType(ServletRequest request) { String path = ((HttpServletRequest)request).getRequestURI(); - ServletContextHandler.Context sContext = - (ServletContextHandler.Context)config.getServletContext(); - String mime = sContext.getMimeType(path); + String mime = config.getServletContext().getMimeType(path); return (mime == null) ? null : mime; } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2Metrics.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2Metrics.java index f99476fc4d148d..6cdb3d5b0f0737 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2Metrics.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2Metrics.java @@ -30,6 +30,14 @@ /** * This class collects all the metrics of Jetty's StatisticsHandler * and expose them as Hadoop Metrics. + * + * Jetty 12 rebuilt StatisticsHandler around the core request lifecycle. The + * dispatch counters it used to publish are now handle counters measuring the + * same thing under a different name, and are read as such here so the metric + * names Hadoop emits do not move. Its four async counters and its count of + * expired async requests have no counterpart, because the core no longer sees + * servlet async activity, so those five metrics are no longer emitted rather + * than reported as a constant. */ @InterfaceAudience.Private @InterfaceStability.Unstable @@ -42,53 +50,33 @@ public class HttpServer2Metrics { private final int acceptorThreads; private final int selectorThreads; - @Metric("number of requested that have been asynchronously dispatched") - public int asyncDispatches() { - return handler.getAsyncDispatches(); - } - @Metric("total number of async requests") - public int asyncRequests() { - return handler.getAsyncRequests(); - } - @Metric("currently waiting async requests") - public int asyncRequestsWaiting() { - return handler.getAsyncRequestsWaiting(); - } - @Metric("maximum number of waiting async requests") - public int asyncRequestsWaitingMax() { - return handler.getAsyncRequestsWaitingMax(); - } @Metric("number of dispatches") public int dispatched() { - return handler.getDispatched(); + return handler.getHandleTotal(); } @Metric("number of dispatches currently active") public int dispatchedActive() { - return handler.getDispatchedActive(); + return handler.getHandleActive(); } @Metric("maximum number of active dispatches being handled") public int dispatchedActiveMax() { - return handler.getDispatchedActiveMax(); + return handler.getHandleActiveMax(); } @Metric("maximum time spend in dispatch handling (in ms)") public long dispatchedTimeMax() { - return handler.getDispatchedTimeMax(); + return handler.getHandleTimeMax(); } @Metric("mean time spent in dispatch handling (in ms)") public double dispatchedTimeMean() { - return handler.getDispatchedTimeMean(); + return handler.getHandleTimeMean(); } @Metric("standard deviation for dispatch handling (in ms)") public double dispatchedTimeStdDev() { - return handler.getDispatchedTimeStdDev(); + return handler.getHandleTimeStdDev(); } @Metric("total time spent in dispatch handling (in ms)") public long dispatchedTimeTotal() { - return handler.getDispatchedTimeTotal(); - } - @Metric("number of async requests requests that have expired") - public int expires() { - return handler.getExpires(); + return handler.getHandleTimeTotal(); } @Metric("number of requests") public int requests() { @@ -140,11 +128,11 @@ public int responses5xx() { } @Metric("total number of bytes across all responses") public long responsesBytesTotal() { - return handler.getResponsesBytesTotal(); + return handler.getBytesWritten(); } @Metric("time in milliseconds stats have been collected for") public long statsOnMs() { - return handler.getStatsOnMs(); + return handler.getStatisticsDuration().toMillis(); } @Metric("maximum number of threads in the pool") public int maxThreads() { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/ProfileOutputServlet.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/ProfileOutputServlet.java index 1ecc21f3753ceb..3c69eca6e6ae2f 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/ProfileOutputServlet.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/ProfileOutputServlet.java @@ -25,7 +25,7 @@ import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; -import org.eclipse.jetty.servlet.DefaultServlet; +import org.eclipse.jetty.ee8.servlet.DefaultServlet; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/WebServlet.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/WebServlet.java index 2eb6c2beb16a62..765709625e5110 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/WebServlet.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/WebServlet.java @@ -22,7 +22,7 @@ import javax.servlet.ServletException; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; -import org.eclipse.jetty.servlet.DefaultServlet; +import org.eclipse.jetty.ee8.servlet.DefaultServlet; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/RestCsrfPreventionFilter.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/RestCsrfPreventionFilter.java index 7363ca0ba64505..7c3a6fa5a8d259 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/RestCsrfPreventionFilter.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/RestCsrfPreventionFilter.java @@ -37,7 +37,6 @@ import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.conf.Configuration; -import org.eclipse.jetty.server.Response; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -272,10 +271,8 @@ public void proceed() throws IOException, ServletException { @Override public void sendError(int code, String message) throws IOException { - if (httpResponse instanceof Response) { - ((Response)httpResponse).setStatusWithReason(code, message); - } - + // Jetty 12 never puts a reason phrase on the wire, so the detail is + // left to sendError, which writes it into the response body. httpResponse.sendError(code, message); } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java index f4ede6f35edb0c..29e311739e18f4 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java @@ -302,7 +302,15 @@ public boolean managementOperation(AuthenticationToken token, requestUgi.getShortUserName()); map = Collections.singletonMap("long", expirationTime); } catch (IOException ex) { - throw new AuthenticationException(ex.toString(), ex); + // Reported the same way as the authorization failure above, + // rather than thrown for AuthenticationFilter to turn into a + // bare status. Letting it out used to be survivable because + // the reason phrase carried the detail; Jetty 12 does not + // send one, and the container's error page is HTML, which is + // not what the client on the other end parses. + HttpExceptionUtils.createServletExceptionResponse(response, + HttpServletResponse.SC_FORBIDDEN, ex); + requestContinues = false; } } break; diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java index be1baa97c88109..a1b67a697078bb 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java @@ -36,7 +36,8 @@ import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; -import org.eclipse.jetty.util.log.Log; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** *

@@ -58,6 +59,9 @@ *

*/ public abstract class FSMainOperationsBaseTest extends FileSystemTestHelper { + private static final Logger LOG = + LoggerFactory.getLogger(FSMainOperationsBaseTest.class); + private static String TEST_DIR_AAA2 = "test/hadoop2/aaa"; private static String TEST_DIR_AAA = "test/hadoop/aaa"; @@ -803,7 +807,7 @@ public void testRenameNonExistentPath() throws Exception { rename(src, dst, false, false, false, Rename.NONE); fail("Should throw FileNotFoundException"); } catch (IOException e) { - Log.getLog().info("XXX", e); + LOG.info("XXX", e); assertTrue(unwrapException(e) instanceof FileNotFoundException); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java index 5713f532be7e81..6482dc5936a6bd 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java @@ -26,7 +26,8 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.viewfs.ConfigUtil; import org.apache.hadoop.util.Shell; -import org.eclipse.jetty.util.log.Log; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** @@ -46,6 +47,9 @@ * We also set the view file system's wd to point to the wd. */ public class ViewFileSystemTestSetup { + private static final Logger LOG = + LoggerFactory.getLogger(ViewFileSystemTestSetup.class); + static public String ViewFSTestDir = "/testDir"; @@ -83,7 +87,7 @@ static public FileSystem setupForViewFileSystem(Configuration conf, FileSystemTe FileSystem fsView = FileSystem.get(FsConstants.VIEWFS_URI, conf); fsView.setWorkingDirectory(new Path(wdDir)); // in case testdir relative to wd. - Log.getLog().info("Working dir is: " + fsView.getWorkingDirectory()); + LOG.info("Working dir is: " + fsView.getWorkingDirectory()); return fsView; } @@ -117,12 +121,12 @@ static void setUpHomeDir(Configuration conf, FileSystem fsTarget) { } else { // home dir is at root. Just link the home dir itse URI linkTarget = fsTarget.makeQualified(new Path(homeDir)).toUri(); ConfigUtil.addLink(conf, homeDir, linkTarget); - Log.getLog().info("Added link for home dir " + homeDir + "->" + linkTarget); + LOG.info("Added link for home dir " + homeDir + "->" + linkTarget); } // Now set the root of the home dir for viewfs String homeDirRoot = fsTarget.getHomeDirectory().getParent().toUri().getPath(); ConfigUtil.setHomeDirConf(conf, homeDirRoot); - Log.getLog().info("Home dir base for viewfs" + homeDirRoot); + LOG.info("Home dir base for viewfs" + homeDirRoot); } /* @@ -137,7 +141,7 @@ static void linkUpFirstComponents(Configuration conf, String path, FileSystem fs String firstComponent = path.substring(0, indexOfEnd); URI linkTarget = fsTarget.makeQualified(new Path(firstComponent)).toUri(); ConfigUtil.addLink(conf, firstComponent, linkTarget); - Log.getLog().info("Added link for " + info + " " + LOG.info("Added link for " + info + " " + firstComponent + "->" + linkTarget); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java index 8f9ac895946e0d..688825ac89ce67 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java @@ -32,7 +32,8 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme.ChildFsGetter; import org.apache.hadoop.util.Shell; -import org.eclipse.jetty.util.log.Log; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** @@ -53,6 +54,9 @@ */ public class ViewFsTestSetup { + private static final Logger LOG = + LoggerFactory.getLogger(ViewFsTestSetup.class); + static public String ViewFSTestDir = "/testDir"; @@ -88,7 +92,7 @@ static public FileContext setupForViewFsLocalFs(FileContextTestHelper helper) th FileContext fc = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf); fc.setWorkingDirectory(new Path(wdDir)); // in case testdir relative to wd. - Log.getLog().info("Working dir is: " + fc.getWorkingDirectory()); + LOG.info("Working dir is: " + fc.getWorkingDirectory()); //System.out.println("SRCOfTests = "+ getTestRootPath(fc, "test")); //System.out.println("TargetOfTests = "+ targetOfTests.toUri()); return fc; @@ -113,12 +117,12 @@ static void setUpHomeDir(Configuration conf, FileContext fsTarget) { } else { // home dir is at root. Just link the home dir itse URI linkTarget = fsTarget.makeQualified(new Path(homeDir)).toUri(); ConfigUtil.addLink(conf, homeDir, linkTarget); - Log.getLog().info("Added link for home dir " + homeDir + "->" + linkTarget); + LOG.info("Added link for home dir " + homeDir + "->" + linkTarget); } // Now set the root of the home dir for viewfs String homeDirRoot = fsTarget.getHomeDirectory().getParent().toUri().getPath(); ConfigUtil.setHomeDirConf(conf, homeDirRoot); - Log.getLog().info("Home dir base for viewfs" + homeDirRoot); + LOG.info("Home dir base for viewfs" + homeDirRoot); } /* @@ -134,7 +138,7 @@ static void linkUpFirstComponents(Configuration conf, String path, String firstComponent = path.substring(0, indexOfEnd); URI linkTarget = fsTarget.makeQualified(new Path(firstComponent)).toUri(); ConfigUtil.addLink(conf, firstComponent, linkTarget); - Log.getLog().info("Added link for " + info + " " + LOG.info("Added link for " + info + " " + firstComponent + "->" + linkTarget); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java index 545b273701fcf0..79137f25ab9e24 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestAuthenticationSessionCookie.java @@ -21,7 +21,6 @@ import org.apache.hadoop.test.GenericTestUtils; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Test; -import org.eclipse.jetty.util.log.Log; import javax.servlet.*; import javax.servlet.http.HttpServletResponse; @@ -37,8 +36,13 @@ import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; public class TestAuthenticationSessionCookie { + private static final Logger LOG = + LoggerFactory.getLogger(TestAuthenticationSessionCookie.class); + private static final String BASEDIR = GenericTestUtils.getTempPath(TestHttpCookieFlag.class.getSimpleName()); private static boolean isCookiePersistent; @@ -152,7 +156,7 @@ public void testSessionCookie() throws IOException { String header = conn.getHeaderField("Set-Cookie"); List cookies = HttpCookie.parse(header); assertTrue(!cookies.isEmpty()); - Log.getLog().info(header); + LOG.info(header); assertFalse(header.contains("; Expires=")); assertTrue("token".equals(cookies.get(0).getValue())); } @@ -174,7 +178,7 @@ public void testPersistentCookie() throws IOException { String header = conn.getHeaderField("Set-Cookie"); List cookies = HttpCookie.parse(header); assertTrue(!cookies.isEmpty()); - Log.getLog().info(header); + LOG.info(header); assertTrue(header.contains("; Expires=")); assertTrue("token".equals(cookies.get(0).getValue())); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java index d8cf9ff0bdec3f..0cf5a82ffd9622 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java @@ -34,6 +34,8 @@ import com.fasterxml.jackson.core.type.TypeReference; import org.eclipse.jetty.server.HttpConfiguration; import org.eclipse.jetty.server.ServerConnector; +import org.apache.hadoop.test.GenericTestUtils; +import org.eclipse.jetty.server.Handler; import org.eclipse.jetty.server.handler.StatisticsHandler; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; @@ -277,6 +279,29 @@ public void testAcceptorSelectorConfigurability() throws Exception { conn.getContentType()); } + /** + * /static must never list what is in it. The setting that stops it is a + * context init parameter, which the DefaultServlet reads under a prefix of + * its own choosing: get the prefix wrong and the parameter is not rejected, + * it is ignored, and dirAllowed falls back to its default of true. That is + * silent, so the endpoint is asserted rather than the setting. + */ + @Test + public void testStaticContextDoesNotListDirectories() throws Exception { + URL staticUrl = new URL(baseUrl, "/static/"); + HttpURLConnection conn = (HttpURLConnection) staticUrl.openConnection(); + conn.connect(); + assertEquals(HttpServletResponse.SC_FORBIDDEN, conn.getResponseCode(), + "/static served a directory listing"); + + // The context is otherwise working, so the 403 above is dirAllowed doing + // its job and not the whole context being broken. + URL cssUrl = new URL(baseUrl, "/static/test.css"); + conn = (HttpURLConnection) cssUrl.openConnection(); + conn.connect(); + assertEquals(HttpServletResponse.SC_OK, conn.getResponseCode()); + } + @Test public void testHttpServer2Metrics() throws Exception { final HttpServer2Metrics metrics = server.getMetrics(); @@ -286,8 +311,10 @@ public void testHttpServer2Metrics() throws Exception { (HttpURLConnection)servletUrl.openConnection(); conn.connect(); assertThat(conn.getResponseCode()).isEqualTo(200); - final int after = metrics.responses2xx(); - assertThat(after).isGreaterThan(before); + // Jetty 12 books the response when the exchange completes on the server, + // which can be after the client has read the status line, so the counter + // is given a moment rather than read straight away. + GenericTestUtils.waitFor(() -> metrics.responses2xx() > before, 50, 10000); } @Test @@ -323,9 +350,12 @@ public void testHttpServer2ThreadPoolMetrics() throws Exception { } /** - * Jetty StatisticsHandler must be inserted via Server#insertHandler - * instead of Server#setHandler. The server fails to start if - * the handler is added by setHandler. + * Jetty StatisticsHandler must be inserted via Server#insertHandler instead + * of Server#setHandler. On 9.4 the difference showed up as a server that + * refused to start, so the test could assert the failure; Jetty 12 starts a + * childless handler quite happily and serves 404s from it, which is worse. + * So the assertion is that the server still serves after the handler goes + * in - the reason to prefer insertHandler in the first place. */ @Test public void testSetStatisticsHandler() throws Exception { @@ -334,11 +364,30 @@ public void testSetStatisticsHandler() throws Exception { conf.setBoolean( CommonConfigurationKeysPublic.HADOOP_HTTP_METRICS_ENABLED, false); final HttpServer2 testServer = createTestServer(conf); - testServer.webServer.setHandler(new StatisticsHandler()); + testServer.addServlet("echo", "/echo", EchoServlet.class); + + final Handler tree = testServer.webServer.getHandler(); + assertThat(tree).isNotNull(); + final StatisticsHandler statistics = new StatisticsHandler(); + testServer.webServer.insertHandler(statistics); + assertThat(statistics.getHandler()) + .as("insertHandler keeps the handler tree underneath") + .isSameAs(tree); + try { testServer.start(); - fail("IOException should be thrown."); - } catch (IOException ignore) { + final URL echoUrl = new URL(getServerURL(testServer), "/echo?a=b"); + final HttpURLConnection conn = + (HttpURLConnection) echoUrl.openConnection(); + conn.connect(); + assertThat(conn.getResponseCode()) + .as("the webapp stopped serving once StatisticsHandler was inserted") + .isEqualTo(HttpServletResponse.SC_OK); + // Booked when the exchange completes on the server, which can be after + // the client has read the status line - see testHttpServer2Metrics. + GenericTestUtils.waitFor(() -> statistics.getRequests() > 0, 50, 10000); + } finally { + testServer.stop(); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java index 25756dd4277944..c39296c4d958c8 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java @@ -35,12 +35,12 @@ import org.apache.hadoop.test.GenericTestUtils; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; -import org.eclipse.jetty.servlet.FilterHolder; -import org.eclipse.jetty.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.FilterHolder; +import org.eclipse.jetty.ee8.servlet.ServletHolder; import org.slf4j.event.Level; import javax.security.auth.Subject; diff --git a/hadoop-common-project/hadoop-kms/pom.xml b/hadoop-common-project/hadoop-kms/pom.xml index 8ccf377c5cd5f8..42c3654367ab6d 100644 --- a/hadoop-common-project/hadoop-kms/pom.xml +++ b/hadoop-common-project/hadoop-kms/pom.xml @@ -82,8 +82,8 @@ runtime - org.eclipse.jetty - jetty-webapp + org.eclipse.jetty.ee8 + jetty-ee8-webapp org.apache.hadoop diff --git a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java index 30f55737c62f00..dc7a5a3cb56453 100644 --- a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java +++ b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSAuthenticationFilter.java @@ -28,7 +28,6 @@ import org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler; import org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticationHandler; import org.apache.hadoop.security.token.delegation.web.PseudoDelegationTokenAuthenticationHandler; -import org.eclipse.jetty.server.Response; import javax.servlet.FilterChain; import javax.servlet.FilterConfig; @@ -115,17 +114,8 @@ public void sendError(int sc, String msg) throws IOException { statusCode = sc; this.msg = msg; - ServletResponse response = getResponse(); - - // After Jetty 9.4.21, sendError() no longer allows a custom message. - // use setStatusWithReason() to set a custom message. - if (response instanceof Response) { - ((Response) response).setStatusWithReason(sc, msg); - } else { - KMS.LOG.warn("The wrapped response object is instance of {}" + - ", not org.eclipse.jetty.server.Response. Can't set custom error " + - "message", response.getClass()); - } + // Jetty 12 never puts a reason phrase on the wire, so the detail is + // left to sendError, which writes it into the response body. super.sendError(sc, HtmlQuoting.quoteHtmlChars(msg)); } diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml b/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml index 14e46f62bc52a0..539f29d7a99836 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml @@ -87,8 +87,8 @@ runtime - org.eclipse.jetty - jetty-webapp + org.eclipse.jetty.ee8 + jetty-ee8-webapp org.apache.hadoop diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java index 41ba89d9517cc0..a414c975e87539 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java @@ -82,7 +82,7 @@ import org.json.simple.parser.ContainerFactory; import org.json.simple.parser.JSONParser; import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.MethodSource; diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSAccessControlled.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSAccessControlled.java index 99dfb3e281d597..e7334e88809f4d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSAccessControlled.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSAccessControlled.java @@ -31,7 +31,7 @@ import org.apache.hadoop.test.TestJettyHelper; import org.junit.jupiter.api.Test; import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import java.io.File; import java.io.FileOutputStream; diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java index d9c1b8813f49fe..d5a82a4613fd25 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java @@ -111,7 +111,7 @@ import org.json.simple.parser.JSONParser; import org.junit.jupiter.api.Test; import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import org.apache.hadoop.thirdparty.com.google.common.collect.Maps; import java.util.Properties; diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerNoACLs.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerNoACLs.java index ff78afabe1bdd7..3cb15048d522ed 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerNoACLs.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerNoACLs.java @@ -32,7 +32,7 @@ import org.apache.hadoop.test.TestJettyHelper; import org.junit.jupiter.api.Test; import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import java.io.BufferedReader; import java.io.File; diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerNoXAttrs.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerNoXAttrs.java index a7dcffe06fbd1c..bd7c9dff6f00bc 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerNoXAttrs.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerNoXAttrs.java @@ -33,7 +33,7 @@ import org.apache.hadoop.test.TestJettyHelper; import org.junit.jupiter.api.Test; import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import java.io.BufferedReader; import java.io.File; diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java index 44da554a849e15..349664ab108e84 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSWithKerberos.java @@ -41,7 +41,7 @@ import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Test; import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import java.io.File; import java.io.FileOutputStream; diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestHFSTestCase.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestHFSTestCase.java index 737940908cf3e8..ceae8b88ffced6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestHFSTestCase.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestHFSTestCase.java @@ -35,7 +35,7 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.util.Time; -import org.eclipse.jetty.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; import org.junit.jupiter.api.Test; import org.eclipse.jetty.server.Server; diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestHTestCase.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestHTestCase.java index 7354def8b05bca..1b60324658e855 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestHTestCase.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/TestHTestCase.java @@ -30,7 +30,7 @@ import javax.servlet.http.HttpServletResponse; import org.apache.hadoop.util.Time; -import org.eclipse.jetty.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; import org.junit.jupiter.api.Test; import org.eclipse.jetty.server.Server; diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java index 442c1aba95b1cf..b6d7568d008ad2 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java @@ -42,7 +42,6 @@ import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.DFSUtilClient; import org.apache.hadoop.security.SecurityUtil; -import org.eclipse.jetty.server.Response; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.hadoop.classification.InterfaceAudience; @@ -703,10 +702,8 @@ public Void run() throws Exception { private void sendError(HttpServletResponse response, int code, String message) throws IOException { - if (response instanceof Response) { - ((Response)response).setStatusWithReason(code, message); - } - + // Jetty 12 never puts a reason phrase on the wire, so the detail is left + // to sendError, which writes it into the response body. response.sendError(code, message); } diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java index 26feba37a0672f..5e044c8003bd1a 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java @@ -20,8 +20,8 @@ import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.servlet.ServletContextHandler; -import org.eclipse.jetty.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.io.Text; diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index d134d39745162b..d44a9e8fb78e13 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -36,11 +36,12 @@ true true - 9.4.58.v20250814 + 12.0.37 4.0.9 _ @@ -910,24 +911,6 @@ org.eclipse.jetty jetty-server ${jetty.version} - - - org.eclipse.jetty - javax.servlet-api - - - - javax.servlet - javax.servlet-api - - org.eclipse.jetty @@ -939,39 +922,72 @@ jetty-util ${jetty.version} - - org.eclipse.jetty - jetty-servlet + + + org.eclipse.jetty.ee8 + jetty-ee8-nested ${jetty.version} - org.eclipse.jetty - jetty-webapp + org.eclipse.jetty.ee8 + jetty-ee8-servlet ${jetty.version} - org.eclipse.jetty.websocket - javax-websocket-server-impl + org.eclipse.jetty.ee8 + jetty-ee8-webapp + ${jetty.version} + + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-server ${jetty.version} - org.ow2.asm - asm - - - org.eclipse.jetty - jetty-webapp - - - - javax.servlet - javax.servlet-api + org.eclipse.jetty.ee8 + jetty-ee8-annotations - org.eclipse.jetty.websocket - websocket-client + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-client ${jetty.version} @@ -1092,11 +1108,6 @@ ${commons-io.version} - - org.eclipse.jetty - jetty-servlet-tester - ${jetty.version} - commons-logging commons-logging @@ -2210,6 +2221,14 @@ + org.glassfish.jersey.test-framework.providers jersey-test-framework-provider-jdk-http diff --git a/hadoop-tools/hadoop-sls/pom.xml b/hadoop-tools/hadoop-sls/pom.xml index af9a37b3cd61a9..2e9be81530d371 100644 --- a/hadoop-tools/hadoop-sls/pom.xml +++ b/hadoop-tools/hadoop-sls/pom.xml @@ -52,12 +52,16 @@ org.eclipse.jetty jetty-server provided - - - org.eclipse.jetty - javax.servlet-api - - + + + + org.eclipse.jetty.ee8 + jetty-ee8-nested + provided org.eclipse.jetty diff --git a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java index ce36854bca2fde..ada2f83be51790 100644 --- a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java +++ b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java @@ -41,12 +41,13 @@ import org.apache.hadoop.yarn.sls.scheduler.SchedulerWrapper; import org.apache.hadoop.yarn.sls.utils.NodeUsageRanges; +import org.eclipse.jetty.ee8.nested.AbstractHandler; +import org.eclipse.jetty.ee8.nested.ContextHandler; +import org.eclipse.jetty.ee8.nested.Handler; +import org.eclipse.jetty.ee8.nested.Request; +import org.eclipse.jetty.ee8.nested.ResourceHandler; import org.eclipse.jetty.http.MimeTypes; -import org.eclipse.jetty.server.Handler; -import org.eclipse.jetty.server.Request; import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.server.handler.AbstractHandler; -import org.eclipse.jetty.server.handler.ResourceHandler; import com.codahale.metrics.Counter; import com.codahale.metrics.Gauge; import com.codahale.metrics.Histogram; @@ -179,8 +180,11 @@ public void handle(String target, Request baseRequest, } }; + ContextHandler context = new ContextHandler("/"); + context.setHandler(handler); + server = new Server(port); - server.setHandler(handler); + server.setHandler(context); server.start(); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml index 18740d02a851dd..d766394eaf12ad 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml @@ -87,24 +87,30 @@ test + org.eclipse.jetty.http2 - http2-hpack + jetty-http2-hpack ${jetty.version} org.eclipse.jetty.http2 - http2-client + jetty-http2-client ${jetty.version} org.eclipse.jetty.http2 - http2-common + jetty-http2-common ${jetty.version} org.eclipse.jetty.http2 - http2-http-client-transport + jetty-http2-client-transport ${jetty.version} @@ -195,6 +201,10 @@ weight. It also lags Hadoop's managed Jetty by several releases, which is what put two Jetty releases on this module's test classpath. Excluding it leaves only ${jetty.version}. + + Maven has no way to share an exclusion set, so this block is + repeated verbatim on the other Solr dependency in this file. + Keep the two in step. --> org.eclipse.jetty @@ -228,6 +238,21 @@ org.eclipse.jetty jetty-servlets + + + org.eclipse.jetty + jetty-servlet + + + org.eclipse.jetty + jetty-webapp + org.eclipse.jetty.http2 http2-server @@ -295,6 +320,10 @@ weight. It also lags Hadoop's managed Jetty by several releases, which is what put two Jetty releases on this module's test classpath. Excluding it leaves only ${jetty.version}. + + Maven has no way to share an exclusion set, so this block is + repeated verbatim on the other Solr dependency in this file. + Keep the two in step. --> org.eclipse.jetty @@ -328,6 +357,21 @@ org.eclipse.jetty jetty-servlets + + + org.eclipse.jetty + jetty-servlet + + + org.eclipse.jetty + jetty-webapp + org.eclipse.jetty.http2 http2-server diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/pom.xml index 1aa02a539893bd..e27e7eadd670e8 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/pom.xml @@ -124,8 +124,8 @@ slf4j-api - org.eclipse.jetty - jetty-webapp + org.eclipse.jetty.ee8 + jetty-ee8-webapp com.google.inject @@ -173,8 +173,8 @@ jetty-util - org.eclipse.jetty - jetty-servlet + org.eclipse.jetty.ee8 + jetty-ee8-servlet org.mockito diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServerWebApp.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServerWebApp.java index f4acd942cc9fb0..a9c1bdfeadbcc4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServerWebApp.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServerWebApp.java @@ -27,7 +27,7 @@ import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.webapp.GenericExceptionHandler; import org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider; -import org.eclipse.jetty.webapp.Configuration; +import org.eclipse.jetty.ee8.webapp.Configuration; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestApiServiceClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestApiServiceClient.java index fe9c081ed64a51..e62bdc8e16ec2e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestApiServiceClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestApiServiceClient.java @@ -32,8 +32,8 @@ import org.apache.hadoop.yarn.exceptions.YarnException; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.servlet.ServletContextHandler; -import org.eclipse.jetty.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; import org.eclipse.jetty.util.thread.QueuedThreadPool; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestSecureApiServiceClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestSecureApiServiceClient.java index 60c06e9aa75f73..fb0b2a84cbedb7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestSecureApiServiceClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestSecureApiServiceClient.java @@ -46,8 +46,8 @@ import org.apache.log4j.Logger; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.servlet.ServletContextHandler; -import org.eclipse.jetty.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; import org.eclipse.jetty.util.thread.QueuedThreadPool; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml index 88c5b1ac392b82..1ac9f6cf165fa2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml @@ -70,8 +70,8 @@ reload4j - org.eclipse.jetty.websocket - websocket-client + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-client diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/ContainerShellWebSocket.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/ContainerShellWebSocket.java index 5656484fca126b..7b83d72099e276 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/ContainerShellWebSocket.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/ContainerShellWebSocket.java @@ -20,15 +20,17 @@ import java.io.IOException; import java.io.OutputStream; +import java.net.InetSocketAddress; +import java.net.SocketAddress; import java.nio.charset.StandardCharsets; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; -import org.eclipse.jetty.websocket.api.Session; -import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose; -import org.eclipse.jetty.websocket.api.annotations.OnWebSocketConnect; -import org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage; -import org.eclipse.jetty.websocket.api.annotations.WebSocket; +import org.eclipse.jetty.ee8.websocket.api.Session; +import org.eclipse.jetty.ee8.websocket.api.annotations.OnWebSocketClose; +import org.eclipse.jetty.ee8.websocket.api.annotations.OnWebSocketConnect; +import org.eclipse.jetty.ee8.websocket.api.annotations.OnWebSocketMessage; +import org.eclipse.jetty.ee8.websocket.api.annotations.WebSocket; import org.jline.terminal.Terminal; import org.jline.terminal.TerminalBuilder; import org.jline.reader.LineReader; @@ -69,16 +71,16 @@ public void onText(Session session, String message) throws IOException { @OnWebSocketConnect public void onConnect(Session s) { initTerminal(s); - LOG.info("{} connected!", s.getRemoteAddress().getHostString()); + LOG.info("{} connected!", remoteHost(s)); } @OnWebSocketClose public void onClose(Session session, int status, String reason) { if (status==1000) { - LOG.info("{} closed, status: {}", session.getRemoteAddress().getHostString(), status); + LOG.info("{} closed, status: {}", remoteHost(session), status); } else { LOG.warn("{} closed, status:" + - " {} Reason: {}.", session.getRemoteAddress().getHostString(), status, reason); + " {} Reason: {}.", remoteHost(session), status, reason); } } @@ -101,11 +103,8 @@ public void run() { } inputThread.join(); } catch (IOException | InterruptedException e) { - try { - mySession.disconnect(); - } catch (IOException e1) { - LOG.error("Error closing connection: ", e1); - } + // Session.disconnect() no longer declares IOException as of Jetty 12. + mySession.disconnect(); } } @@ -159,4 +158,19 @@ public void run() { } } } + + /** + * Jetty 12 widened Session.getRemoteAddress() from InetSocketAddress to + * SocketAddress, so the host is read back out here for logging. + * + * @param session the WebSocket session + * @return the remote host, or the address as written if it is not an IP socket + */ + private static String remoteHost(Session session) { + SocketAddress remote = session.getRemoteAddress(); + return remote instanceof InetSocketAddress + ? ((InetSocketAddress) remote).getHostString() + : String.valueOf(remote); + } + } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java index ca30ff865bf08e..538d0a8a9a226a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java @@ -143,10 +143,10 @@ import org.apache.hadoop.yarn.util.Records; import org.apache.hadoop.yarn.util.resource.ResourceUtils; import org.apache.hadoop.yarn.util.timeline.TimelineUtils; -import org.eclipse.jetty.websocket.api.Session; -import org.eclipse.jetty.websocket.api.WebSocketException; -import org.eclipse.jetty.websocket.client.ClientUpgradeRequest; -import org.eclipse.jetty.websocket.client.WebSocketClient; +import org.eclipse.jetty.ee8.websocket.api.Session; +import org.eclipse.jetty.ee8.websocket.api.exceptions.WebSocketException; +import org.eclipse.jetty.ee8.websocket.client.ClientUpgradeRequest; +import org.eclipse.jetty.ee8.websocket.client.WebSocketClient; import org.apache.hadoop.classification.VisibleForTesting; import org.slf4j.Logger; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java index 2da782ea6db1c0..524f5240d7579a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java @@ -87,14 +87,18 @@ import org.mockito.ArgumentCaptor; import org.mockito.invocation.InvocationOnMock; import org.mockito.stubbing.Answer; -import org.eclipse.jetty.util.log.Log; import static org.junit.jupiter.api.Assumptions.assumeTrue; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * Test application master client class to resource manager. */ public class TestAMRMClient extends BaseAMRMClientTest{ + private static final Logger LOG = + LoggerFactory.getLogger(TestAMRMClient.class); + private final static int DEFAULT_ITERATION = 15; @@ -555,7 +559,7 @@ public void testAMRMClientMatchStorage(String pSchedulerName, int iterationsLeft = 15; while (allocatedContainerCount < 2 && iterationsLeft-- > 0) { - Log.getLog().info("Allocated " + allocatedContainerCount + " containers" + LOG.info("Allocated " + allocatedContainerCount + " containers" + " with " + iterationsLeft + " iterations left"); AllocateResponse allocResponse = amClient.allocate(0.1f); assertEquals(0, amClient.ask.size()); @@ -752,7 +756,7 @@ private int getAllocatedContainersNumber( throws YarnException, IOException { int allocatedContainerCount = 0; while (iterationsLeft-- > 0) { - Log.getLog().info("Allocated " + allocatedContainerCount + " containers" + LOG.info("Allocated " + allocatedContainerCount + " containers" + " with " + iterationsLeft + " iterations left"); AllocateResponse allocResponse = amClient.allocate(0.1f); assertEquals(0, amClient.ask.size()); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java index 1848a921222e0b..a984c80bb69534 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java @@ -106,7 +106,6 @@ import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Timeout; -import org.eclipse.jetty.util.log.Log; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -385,11 +384,11 @@ public void testGetContainers() throws Exception { "http://host:2345", ""); pw.close(); String appReportStr = baos.toString("UTF-8"); - Log.getLog().info("ExpectedOutput"); - Log.getLog().info("["+appReportStr+"]"); - Log.getLog().info("OutputFrom command"); + LOG.info("ExpectedOutput"); + LOG.info("["+appReportStr+"]"); + LOG.info("OutputFrom command"); String actualOutput = sysOutStream.toString("UTF-8"); - Log.getLog().info("["+actualOutput+"]"); + LOG.info("["+actualOutput+"]"); assertEquals(appReportStr, actualOutput); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java index 27600f24211de5..b25751d3cec708 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java @@ -29,6 +29,7 @@ import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.function.Consumer; import javax.servlet.http.HttpServlet; @@ -44,7 +45,7 @@ import org.apache.hadoop.yarn.api.ApplicationClientProtocol; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.webapp.util.WebAppUtils; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import org.glassfish.jersey.server.ResourceConfig; import org.glassfish.jersey.servlet.ServletProperties; import org.slf4j.Logger; @@ -108,6 +109,8 @@ static class ServletStruct { private final HashMap attributes = new HashMap<>(); private ApplicationClientProtocol appClientProtocol; private ResourceConfig config; + private final List> serverConfigurers = + new ArrayList<>(); Builder(String name, Class api, T application, String wsName) { this.name = name; this.api = api; @@ -223,6 +226,24 @@ public Builder withCSRFProtection(String prefix) { return this; } + /** + * Register work to run against the built server before it is started. + *

+ * Some servlets have to be set up while their context is still stopped - + * a Jetty WebSocket servlet needs the WebSocket components installed on + * the context, and installing them once the context has started throws. + * Builder gives no other window between build and start, and the work is + * specific enough to a single webapp that pulling its dependencies into + * this module would be the wrong trade. + * + * @param configurer work to run on the server before it starts + * @return this builder + */ + public Builder withServerConfigurer(Consumer configurer) { + this.serverConfigurers.add(configurer); + return this; + } + /** * Enable the XFS filter. * @param prefix The config prefix that identifies the @@ -492,11 +513,15 @@ public WebApp start(WebApp webapp, WebAppContext... additionalContexts) { for (WebAppContext context : additionalContexts) { if (context != null) { addFiltersForNewContext(context); - httpServer.addHandlerAtFront(context); + httpServer.addHandlerAtFront(context.getCoreContextHandler()); } } } + for (Consumer configurer : serverConfigurers) { + configurer.accept(httpServer); + } + try { httpServer.start(); LOG.info("Web app {} started at {}.", name, httpServer.getConnectorAddress(0).getPort()); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml index 69df71c598633b..adfef58d0f285d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml @@ -58,14 +58,14 @@ org.eclipse.jetty.servlet.FilterHolder. Declared provided, matching the scope they already resolve at through the provided hadoop-common above. --> - org.eclipse.jetty - jetty-webapp + org.eclipse.jetty.ee8 + jetty-ee8-webapp provided - org.eclipse.jetty - jetty-servlet + org.eclipse.jetty.ee8 + jetty-ee8-servlet provided diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java index 06f21f4ae747ba..e2a75b32788d49 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java @@ -64,8 +64,8 @@ import org.apache.hadoop.yarn.webapp.WebApps; import org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider; import org.apache.hadoop.yarn.webapp.util.WebAppUtils; -import org.eclipse.jetty.servlet.FilterHolder; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.servlet.FilterHolder; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import org.apache.hadoop.classification.VisibleForTesting; import org.glassfish.jersey.internal.inject.AbstractBinder; @@ -323,7 +323,7 @@ private void startWebApp() { } } LOG.info("Hosting {} from {} at {}.", name, onDiskPath, webPath); - httpServer.addHandlerAtFront(uiWebAppContext); + httpServer.addHandlerAtFront(uiWebAppContext.getCoreContextHandler()); } httpServer.start(); conf.updateConnectAddr(YarnConfiguration.TIMELINE_SERVICE_BIND_HOST, diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java index 6d37a9730e1d51..71cc77d4bf5ce9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java @@ -21,7 +21,6 @@ import java.io.FilenameFilter; import java.io.IOException; -import org.eclipse.jetty.util.log.Log; import org.fusesource.leveldbjni.JniDBFactory; import org.iq80.leveldb.Options; import org.junit.jupiter.api.AfterEach; @@ -50,11 +49,16 @@ import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.junit.jupiter.api.Assertions.fail; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** Test class to verify RollingLevelDBTimelineStore. */ @InterfaceAudience.Private @InterfaceStability.Unstable public class TestRollingLevelDBTimelineStore extends TimelineStoreTestUtils { + private static final Logger LOG = + LoggerFactory.getLogger(TestRollingLevelDBTimelineStore.class); + private FileContext fsContext; private File fsPath; private Configuration config = new YarnConfiguration(); @@ -375,7 +379,7 @@ public void testStorePerformance() throws IOException { long start = System.currentTimeMillis(); int num = 1000000; - Log.getLog().info("Start test for " + num); + LOG.info("Start test for " + num); final String tezTaskAttemptId = "TEZ_TA"; final String tezEntityId = "attempt_1429158534256_0001_1_00_000000_"; @@ -421,7 +425,7 @@ public void testStorePerformance() throws IOException { } long duration = System.currentTimeMillis() - start; - Log.getLog().info("Duration for " + num + ": " + duration); + LOG.info("Duration for " + num + ": " + duration); } /** diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml index 17fafd3e5b71ec..be795b3e228bee 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml @@ -82,8 +82,18 @@ jetty-util - org.eclipse.jetty.websocket - javax-websocket-server-impl + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-server + + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-client + test org.apache.hadoop.thirdparty diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java index 86d78f2fc351ae..18376bc939549e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java @@ -99,7 +99,6 @@ import org.apache.hadoop.yarn.util.ConverterUtils; import org.apache.hadoop.yarn.util.MonotonicClock; import org.apache.hadoop.yarn.util.resource.Resources; -import org.eclipse.jetty.util.ConcurrentHashSet; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -290,7 +289,7 @@ public FederationInterceptor() { this.finishAMCalled = false; this.lastSCResponseTime = new ConcurrentHashMap<>(); this.lastAMHeartbeatTime = this.clock.getTime(); - this.nmTokenMapFromRegisterSecondaryCluster = new ConcurrentHashSet<>(); + this.nmTokenMapFromRegisterSecondaryCluster = ConcurrentHashMap.newKeySet(); } /** diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocket.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocket.java index 175ee09f8db889..461e85eebffe65 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocket.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocket.java @@ -20,6 +20,8 @@ import java.io.IOException; import java.net.URI; +import java.net.InetSocketAddress; +import java.net.SocketAddress; import java.nio.charset.StandardCharsets; import java.util.List; import java.util.Map; @@ -33,11 +35,11 @@ import org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor; import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; import org.apache.hadoop.yarn.server.nodemanager.executor.ContainerExecContext; -import org.eclipse.jetty.websocket.api.Session; -import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose; -import org.eclipse.jetty.websocket.api.annotations.OnWebSocketConnect; -import org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage; -import org.eclipse.jetty.websocket.api.annotations.WebSocket; +import org.eclipse.jetty.ee8.websocket.api.Session; +import org.eclipse.jetty.ee8.websocket.api.annotations.OnWebSocketClose; +import org.eclipse.jetty.ee8.websocket.api.annotations.OnWebSocketConnect; +import org.eclipse.jetty.ee8.websocket.api.annotations.OnWebSocketMessage; +import org.eclipse.jetty.ee8.websocket.api.annotations.WebSocket; import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair; import org.apache.hadoop.security.HadoopKerberosName; import org.apache.hadoop.security.UserGroupInformation; @@ -120,7 +122,7 @@ public void onConnect(Session session) { session.close(1003, "Nonsecure mode is unsupported."); return; } - LOG.info(session.getRemoteAddress().getHostString() + " connected!"); + LOG.info(remoteHost(session) + " connected!"); LOG.info( "Making interactive connection to running docker container with ID: " + cId); @@ -140,7 +142,7 @@ public void onConnect(Session session) { @OnWebSocketClose public void onClose(Session session, int status, String reason) { try { - LOG.info(session.getRemoteAddress().getHostString() + " closed!"); + LOG.info(remoteHost(session) + " closed!"); String exit = "exit\r\n"; pair.out.write(exit.getBytes(StandardCharsets.UTF_8)); pair.out.flush(); @@ -195,4 +197,19 @@ private boolean checkInsecureSetup() { } return limitUsers; } + + /** + * Jetty 12 widened Session.getRemoteAddress() from InetSocketAddress to + * SocketAddress, so the host is read back out here for logging. + * + * @param session the WebSocket session + * @return the remote host, or the address as written if it is not an IP socket + */ + private static String remoteHost(Session session) { + SocketAddress remote = session.getRemoteAddress(); + return remote instanceof InetSocketAddress + ? ((InetSocketAddress) remote).getHostString() + : String.valueOf(remote); + } + } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocketServlet.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocketServlet.java index 8a8d6d102abf70..d82454b15249e0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocketServlet.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocketServlet.java @@ -20,17 +20,17 @@ import javax.servlet.annotation.WebServlet; -import org.eclipse.jetty.websocket.servlet.WebSocketServlet; -import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory; +import org.eclipse.jetty.ee8.websocket.server.JettyWebSocketServlet; +import org.eclipse.jetty.ee8.websocket.server.JettyWebSocketServletFactory; /** * Container shell web socket interface. */ @WebServlet(urlPatterns="/container/container/*") -public class ContainerShellWebSocketServlet extends WebSocketServlet{ +public class ContainerShellWebSocketServlet extends JettyWebSocketServlet { @Override - public void configure(WebSocketServletFactory factory) { + protected void configure(JettyWebSocketServletFactory factory) { factory.register(ContainerShellWebSocket.class); } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TerminalServlet.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TerminalServlet.java index ec1f9be13fb005..ac6be36d4b8913 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TerminalServlet.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TerminalServlet.java @@ -24,7 +24,7 @@ import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; -import org.eclipse.jetty.servlet.DefaultServlet; +import org.eclipse.jetty.ee8.servlet.DefaultServlet; /** * TerminalServlet host the static html and javascript for diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java index eb7436b6a109d7..b920d42c8464bb 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java @@ -40,6 +40,7 @@ import org.apache.hadoop.yarn.webapp.WebApps; import org.apache.hadoop.yarn.webapp.YarnWebParams; import org.apache.hadoop.yarn.webapp.util.WebAppUtils; +import org.eclipse.jetty.ee8.websocket.server.config.JettyWebSocketServletContainerInitializer; import javax.servlet.Filter; import org.glassfish.jersey.internal.inject.AbstractBinder; @@ -138,6 +139,16 @@ protected void serviceStart() throws Exception { .at(bindAddress) .withServlet("ContainerShellWebSocket", "/container/*", ContainerShellWebSocketServlet.class, params, false) + // Jetty 9.4 let a WebSocketServlet bootstrap itself from init(). + // Jetty 12 keeps the WebSocket components on the context and expects + // a ServletContainerInitializer to have put them there, which only + // runs if jetty-ee8-annotations is on the classpath - it is excluded + // here because it drags in a banned asm. Installing them directly is + // the same work without the scan, and has to happen while the + // context is still stopped. + .withServerConfigurer(server -> + JettyWebSocketServletContainerInitializer.configure( + server.getWebAppContext(), null)) .withServlet("Terminal", "/terminal/*", TerminalServlet.class, terminalParams, false) .with(conf) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java index 5c2ba9c0942a46..20913b208c5def 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java @@ -46,7 +46,6 @@ import static org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.LinuxContainerRuntimeConstants.CONTAINER_ID_STR; import static org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.docker.DockerCommandExecutor.DockerContainerStatus; -import static org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.LOG; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -55,11 +54,16 @@ import static org.mockito.ArgumentMatchers.eq; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * Test common docker commands. */ public class TestDockerCommandExecutor { + private static final Logger LOG = + LoggerFactory.getLogger(TestDockerCommandExecutor.class); + private static final String MOCK_CONTAINER_ID = "container_e11_1861047502093_13763105_01_000001"; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java index a90172a096007c..a77d2346c59c90 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java @@ -156,7 +156,7 @@ import org.junit.jupiter.api.Timeout; import org.mockito.ArgumentCaptor; import org.mockito.Mockito; -import org.eclipse.jetty.util.MultiException; +import org.eclipse.jetty.util.ExceptionUtil; import java.util.function.Supplier; import org.slf4j.LoggerFactory; @@ -1456,7 +1456,7 @@ void checkEvents(EventHandler eventHandler, List actualEvents = eventCaptor.getAllValues(); // batch up exceptions so junit presents them as one - MultiException failures = new MultiException(); + ExceptionUtil.MultiException failures = new ExceptionUtil.MultiException(); try { assertEquals(expectedEvents.length, actualEvents.size(), "expected events"); } catch (Throwable e) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellClientSocketTest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellClientSocketTest.java index e059de90b7fdd3..9e4484e69e39cd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellClientSocketTest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellClientSocketTest.java @@ -18,9 +18,9 @@ package org.apache.hadoop.yarn.server.nodemanager.webapp; -import org.eclipse.jetty.websocket.api.Session; -import org.eclipse.jetty.websocket.api.WebSocketAdapter; -import org.eclipse.jetty.websocket.api.annotations.WebSocket; +import org.eclipse.jetty.ee8.websocket.api.Session; +import org.eclipse.jetty.ee8.websocket.api.WebSocketAdapter; +import org.eclipse.jetty.ee8.websocket.api.annotations.WebSocket; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMContainerWebSocket.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMContainerWebSocket.java index 87c1f782112b08..31a1bb21ba6c38 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMContainerWebSocket.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMContainerWebSocket.java @@ -28,14 +28,15 @@ import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; import org.apache.hadoop.yarn.server.nodemanager.health.NodeHealthCheckerService; import org.apache.hadoop.yarn.server.security.ApplicationACLsManager; -import org.eclipse.jetty.websocket.api.Session; -import org.eclipse.jetty.websocket.api.UpgradeRequest; -import org.eclipse.jetty.websocket.client.WebSocketClient; +import org.eclipse.jetty.ee8.websocket.api.Session; +import org.eclipse.jetty.ee8.websocket.api.UpgradeRequest; +import org.eclipse.jetty.ee8.websocket.client.WebSocketClient; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.junit.jupiter.api.Assertions.fail; import static org.mockito.Mockito.mock; @@ -48,6 +49,7 @@ import java.util.List; import java.util.Map; import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; /** * Test class for Node Manager Container Web Socket. @@ -125,31 +127,36 @@ private NodeHealthCheckerService createNodeHealthCheckerService() { return new NodeHealthCheckerService(dirsHandler); } + /** + * The container shell endpoint completes a WebSocket upgrade. + *

+ * What is asserted is the handshake, not a working shell: there is no + * container "abc" here, so ContainerShellWebSocket closes the session + * shortly after it opens. Getting a Session at all is the interesting part, + * because it is the whole Jetty upgrade path - and it is what breaks, with a + * 500, when the WebSocket components are missing from the context. + */ @Test - public void testWebServerWithServlet() { + public void testWebServerWithServlet() throws Exception { int port = startNMWebAppServer("0.0.0.0"); LOG.info("bind to port: " + port); - StringBuilder sb = new StringBuilder(); - sb.append("ws://localhost:").append(port).append("/container/abc/"); - String dest = sb.toString(); + String dest = "ws://localhost:" + port + "/container/abc/"; WebSocketClient client = new WebSocketClient(); try { - ContainerShellClientSocketTest socket = new ContainerShellClientSocketTest(); + ContainerShellClientSocketTest socket = + new ContainerShellClientSocketTest(); client.start(); - URI echoUri = new URI(dest); - Future future = client.connect(socket, echoUri); - Session session = future.get(); - session.getRemote().sendString("hello world"); + Future future = client.connect(socket, new URI(dest)); + Session session = future.get(30, TimeUnit.SECONDS); + assertNotNull(session, "WebSocket upgrade produced no session"); + assertTrue(socket.getLatch().await(30, TimeUnit.SECONDS), + "the client was never told the WebSocket had connected"); session.close(); - client.stop(); - } catch (Throwable t) { - LOG.error("Failed to connect WebSocket and send message to server", t); } finally { try { client.stop(); + } finally { server.close(); - } catch (Exception e) { - LOG.error("Failed to close client", e); } } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java index 7b1042d101ff5d..c6d7d366c2767f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java @@ -139,7 +139,7 @@ import org.apache.hadoop.yarn.webapp.WebApps.Builder; import org.apache.hadoop.yarn.webapp.util.WebAppUtils; import org.apache.zookeeper.server.auth.DigestAuthenticationProvider; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import java.io.IOException; import java.io.InputStream; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java index d433753701b25e..7b3b3e51daff2d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java @@ -55,9 +55,13 @@ import org.apache.hadoop.yarn.util.Records; import org.apache.hadoop.yarn.util.YarnVersionInfo; import org.apache.hadoop.yarn.util.resource.Resources; -import org.eclipse.jetty.util.log.Log; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; public class MockNM { + private static final Logger LOG = + LoggerFactory.getLogger(MockNM.class); + private int responseId; private NodeId nodeId; @@ -238,7 +242,7 @@ public NodeHeartbeatResponse nodeHeartbeat(ApplicationAttemptId attemptId, ArrayList containerStatusList = new ArrayList(1); containerStatusList.add(containerStatus); - Log.getLog().info("ContainerStatus: " + containerStatus); + LOG.info("ContainerStatus: " + containerStatus); return nodeHeartbeat(containerStatusList, Collections.emptyList(), true, responseId); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml index bef0086d1ef2ee..67efbc2a97d1e4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml @@ -60,18 +60,17 @@ hadoop-yarn-server-common - - - org.eclipse.jetty - jetty-webapp + + + org.eclipse.jetty.ee8 + jetty-ee8-webapp diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java index 8f3c4d0fe577e0..ffeaca543a6bc2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java @@ -67,7 +67,7 @@ import org.apache.hadoop.yarn.webapp.WebApps.Builder; import org.apache.hadoop.yarn.webapp.util.WebAppUtils; import org.apache.hadoop.yarn.webapp.util.WebServiceClient; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/TestRouter.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/TestRouter.java index 10ef06c8125f56..6da068f9dcb695 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/TestRouter.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/TestRouter.java @@ -33,9 +33,9 @@ import org.apache.hadoop.test.LambdaTestUtils; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.webapp.WebApp; -import org.eclipse.jetty.servlet.FilterHolder; -import org.eclipse.jetty.servlet.ServletHandler; -import org.eclipse.jetty.webapp.WebAppContext; +import org.eclipse.jetty.ee8.servlet.FilterHolder; +import org.eclipse.jetty.ee8.servlet.ServletHandler; +import org.eclipse.jetty.ee8.webapp.WebAppContext; import org.junit.jupiter.api.Test; import javax.servlet.FilterChain; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/TestRouterWebAppProxy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/TestRouterWebAppProxy.java index d6b87225065ee0..f319b0f0942cab 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/TestRouterWebAppProxy.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/TestRouterWebAppProxy.java @@ -41,8 +41,8 @@ import org.apache.hadoop.yarn.server.webproxy.FedAppReportFetcher; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.servlet.ServletContextHandler; -import org.eclipse.jetty.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; import org.eclipse.jetty.util.thread.QueuedThreadPool; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml index 7ee63ba3edcef4..cbd9a0909c4302 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml @@ -460,13 +460,13 @@ dependency --> - org.eclipse.jetty - jetty-servlet + org.eclipse.jetty.ee8 + jetty-ee8-servlet test - org.eclipse.jetty - jetty-webapp + org.eclipse.jetty.ee8 + jetty-ee8-webapp test diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java index b1349df26de2b9..90086d0ae57c71 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java @@ -45,8 +45,8 @@ import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.servlet.ServletContextHandler; -import org.eclipse.jetty.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; import org.eclipse.jetty.util.thread.QueuedThreadPool; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java index d3cc76951527ef..3906b7581b1d6a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServletFed.java @@ -54,8 +54,8 @@ import org.apache.hadoop.yarn.webapp.util.WebAppUtils; import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.servlet.ServletContextHandler; -import org.eclipse.jetty.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; import org.eclipse.jetty.util.thread.QueuedThreadPool; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/amfilter/TestAmFilter.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/amfilter/TestAmFilter.java index 1b3545c767d966..ad9d2a74551c58 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/amfilter/TestAmFilter.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/amfilter/TestAmFilter.java @@ -46,8 +46,8 @@ import org.eclipse.jetty.server.Server; import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.servlet.ServletContextHandler; -import org.eclipse.jetty.servlet.ServletHolder; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; import org.eclipse.jetty.util.thread.QueuedThreadPool; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Timeout; From 16c6719eaca7883da44c858b0968dfa1f6d127cf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 24 Aug 2026 20:12:51 +0000 Subject: [PATCH 12/30] HADOOP-19972. Assert the servlet contract Hadoop offers the projects that embed it. HBase, Hive, Spark, Ozone and Knox compile against hadoop-auth's filters and HttpServer2. Moving to Jetty 12 must not change what they see: the servlet types on that surface stay javax.servlet, and nothing on it may name a jakarta.servlet type. A test asserts it rather than leaving it to review. It fails the day the jakarta rename reaches this surface, which is the point - that day belongs to a major release, not to this change. Co-Authored-By: Claude --- .../TestDownstreamServletCompatibility.java | 422 ++++++++++++++++++ 1 file changed, 422 insertions(+) create mode 100644 hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestDownstreamServletCompatibility.java diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestDownstreamServletCompatibility.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestDownstreamServletCompatibility.java new file mode 100644 index 00000000000000..ea8e238d3f5ba1 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestDownstreamServletCompatibility.java @@ -0,0 +1,422 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.http; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.lang.reflect.Modifier; +import java.lang.reflect.Type; +import java.net.HttpURLConnection; +import java.net.URL; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Properties; +import java.util.Scanner; +import java.util.Set; +import java.util.concurrent.atomic.AtomicBoolean; + +import javax.servlet.Filter; +import javax.servlet.FilterChain; +import javax.servlet.FilterConfig; +import javax.servlet.ServletException; +import javax.servlet.ServletRequest; +import javax.servlet.ServletResponse; +import javax.servlet.http.HttpServlet; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.security.authentication.server.AuthenticationHandler; +import org.apache.hadoop.security.authentication.server.AuthenticationToken; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +/** + * Guards the contract Hadoop offers the projects that embed it - HBase, Hive, + * Spark, Ozone, Knox and the rest - across a Jetty upgrade. + * + * Those projects implement Hadoop's servlet-facing extension points and embed + * HttpServer2, and they do it against javax.servlet. Two things have to stay + * true for them, and neither is checked by anything else in the tree: + * + * - the extension points keep speaking javax.servlet, never jakarta.servlet, + * - Jetty stays out of the signatures, so nobody downstream has to compile + * against the container Hadoop happens to embed, or care which one it is. + * + * The reflective half of this test would have failed had the Jetty 12 port let + * an org.eclipse.jetty type into a public signature, and will fail on the day + * the jakarta rename reaches one, which is the point: that day belongs to a + * major release and should not arrive by accident. + */ +public class TestDownstreamServletCompatibility + extends HttpServerFunctionalTest { + + /** + * The servlet-facing surface a downstream project compiles against. Types + * are named rather than discovered so that dropping one from the list is a + * deliberate edit and shows up in review. + */ + private static final Class[] PUBLIC_SURFACE = { + org.apache.hadoop.security.authentication.server.AuthenticationHandler.class, + org.apache.hadoop.security.authentication.server.AuthenticationFilter.class, + org.apache.hadoop.security.authentication.server.AuthenticationToken.class, + org.apache.hadoop.security.authentication.client.Authenticator.class, + org.apache.hadoop.security.http.RestCsrfPreventionFilter.class, + org.apache.hadoop.security.http.XFrameOptionsFilter.class, + org.apache.hadoop.http.FilterContainer.class, + org.apache.hadoop.http.FilterInitializer.class, + org.apache.hadoop.http.HttpServer2.class, + }; + + private static final String JAKARTA_SERVLET = "jakarta.servlet"; + private static final String JETTY = "org.eclipse.jetty"; + + /** + * How many of PUBLIC_SURFACE are not InterfaceAudience.Private, and so are + * held to the no-Jetty rule. Asserted as an exact count so that marking one + * of them Private has to be a deliberate edit here too. + */ + private static final int EXPECTED_JETTY_FREE_CLASSES = 7; + + /** + * Nothing on that surface may name a jakarta.servlet type. This is the + * promise the ee8 environment exists to keep, and the one that would break + * every downstream implementation at once. + */ + @Test + public void testPublicSurfaceNeverNamesJakartaServlet() { + List offences = new ArrayList<>(); + int javaxReferences = 0; + for (Class clazz : PUBLIC_SURFACE) { + for (String type : referencedTypes(clazz)) { + if (type.startsWith(JAKARTA_SERVLET)) { + offences.add(clazz.getName() + " exposes " + type + + "; downstream code is written against javax.servlet"); + } + if (type.startsWith("javax.servlet")) { + javaxReferences++; + } + } + } + // Without this the test would also pass if referencedTypes stopped + // reporting servlet types at all, which is the failure it exists to catch. + assertTrue(javaxReferences > 0, + "no javax.servlet type was found anywhere on the surface; the scan is" + + " not reading signatures and this test proves nothing"); + assertTrue(offences.isEmpty(), + "public API has moved to the jakarta namespace:\n " + + String.join("\n ", offences)); + } + + /** + * The classes downstream projects are expected to reach for on that surface + * - everything not marked InterfaceAudience.Private - may not name a Jetty + * type, so that nobody downstream has to compile against the container + * Hadoop embeds, or track which one it is. + *

+ * Absence of an annotation is not a licence to leak. Several of these are + * the oldest classes in hadoop-auth and carry no audience marker at all, + * which does not make them less public in practice; only an explicit + * Private marker excuses a class, and only two here have one - + * AuthenticationFilter, and HttpServer2, which names Jetty deliberately + * (addHandlerAtFront takes a Handler) and is documented as internal. + */ + @Test + public void testPublicApiNeverNamesJetty() { + List offences = new ArrayList<>(); + int checked = 0; + for (Class clazz : PUBLIC_SURFACE) { + if (clazz.getAnnotation(InterfaceAudience.Private.class) != null) { + continue; + } + checked++; + for (String type : referencedTypes(clazz)) { + if (type.startsWith(JETTY)) { + offences.add(clazz.getName() + " exposes " + type + + "; the embedded container must not reach a public signature"); + } + } + } + // A count, not a floor: marking one of these Private would shrink the + // guarded set silently, and that is a decision for review, not a + // side effect. + assertEquals(EXPECTED_JETTY_FREE_CLASSES, checked, + "the set of classes guarded against naming Jetty has changed; if that" + + " is deliberate, update EXPECTED_JETTY_FREE_CLASSES to match"); + assertTrue(offences.isEmpty(), + "public API leaks the servlet container:\n " + + String.join("\n ", offences)); + } + + /** + * Every type named by a public or protected member of the class, and of the + * nested classes it publishes. HttpServer2.Builder is the reason for the + * second half: it is how a downstream project actually configures an + * embedded server, and scanning only the outer class would miss it. + */ + private static Set referencedTypes(Class clazz) { + Set types = declaredTypes(clazz); + for (Class nested : clazz.getDeclaredClasses()) { + if (isVisibleDownstream(nested.getModifiers())) { + types.addAll(declaredTypes(nested)); + } + } + return types; + } + + /** Every type named by a public or protected member declared on the class. */ + private static Set declaredTypes(Class clazz) { + Set types = new LinkedHashSet<>(); + for (Method m : clazz.getDeclaredMethods()) { + if (!isVisibleDownstream(m.getModifiers())) { + continue; + } + collect(types, m.getGenericReturnType()); + for (Type t : m.getGenericParameterTypes()) { + collect(types, t); + } + for (Class t : m.getExceptionTypes()) { + collect(types, t); + } + } + for (Constructor c : clazz.getDeclaredConstructors()) { + if (!isVisibleDownstream(c.getModifiers())) { + continue; + } + for (Type t : c.getGenericParameterTypes()) { + collect(types, t); + } + } + for (Field f : clazz.getDeclaredFields()) { + if (isVisibleDownstream(f.getModifiers())) { + collect(types, f.getGenericType()); + } + } + return types; + } + + private static boolean isVisibleDownstream(int modifiers) { + return Modifier.isPublic(modifiers) || Modifier.isProtected(modifiers); + } + + private static void collect(Set into, Type type) { + if (type == null) { + return; + } + if (type instanceof Class) { + Class c = (Class) type; + while (c.isArray()) { + c = c.getComponentType(); + } + if (!c.isPrimitive()) { + into.add(c.getName()); + } + return; + } + if (type instanceof java.lang.reflect.ParameterizedType) { + java.lang.reflect.ParameterizedType p = + (java.lang.reflect.ParameterizedType) type; + collect(into, p.getRawType()); + for (Type arg : p.getActualTypeArguments()) { + collect(into, arg); + } + return; + } + if (type instanceof java.lang.reflect.GenericArrayType) { + collect(into, + ((java.lang.reflect.GenericArrayType) type).getGenericComponentType()); + } + } + + /** + * An AuthenticationHandler written the way a downstream project writes one: + * javax.servlet types, no Hadoop-internal or Jetty types anywhere. That this + * class compiles at all is most of the assertion. + */ + public static class DownstreamAuthenticationHandler + implements AuthenticationHandler { + + static final String TYPE = "downstream"; + private boolean initialised; + + @Override + public String getType() { + return TYPE; + } + + @Override + public void init(Properties config) throws ServletException { + initialised = true; + } + + @Override + public void destroy() { + initialised = false; + } + + @Override + public boolean managementOperation(AuthenticationToken token, + HttpServletRequest request, HttpServletResponse response) { + return true; + } + + @Override + public AuthenticationToken authenticate(HttpServletRequest request, + HttpServletResponse response) { + if (!initialised) { + return null; + } + return new AuthenticationToken(request.getParameter("user"), "p", TYPE); + } + } + + /** + * The extension point still takes and returns javax.servlet types at run + * time, not only at compile time. + */ + @Test + public void testAuthenticationHandlerStillTakesJavaxServlet() + throws Exception { + AuthenticationHandler handler = new DownstreamAuthenticationHandler(); + handler.init(new Properties()); + try { + HttpServletRequest request = mock(HttpServletRequest.class); + HttpServletResponse response = mock(HttpServletResponse.class); + when(request.getParameter("user")).thenReturn("alice"); + + AuthenticationToken token = handler.authenticate(request, response); + assertNotNull(token, "handler returned no token"); + assertEquals("alice", token.getUserName()); + assertEquals(DownstreamAuthenticationHandler.TYPE, token.getType()); + assertTrue(handler.managementOperation(token, request, response)); + } finally { + handler.destroy(); + } + } + + /** A servlet a downstream project would write: javax.servlet and nothing else. */ + public static class DownstreamServlet extends HttpServlet { + private static final long serialVersionUID = 1L; + + @Override + protected void doGet(HttpServletRequest request, + HttpServletResponse response) throws IOException { + response.setContentType("text/plain; charset=utf-8"); + response.setStatus(HttpServletResponse.SC_OK); + response.getWriter().print("downstream-ok"); + } + } + + /** Likewise a plain javax.servlet Filter. */ + public static class DownstreamFilter implements Filter { + static final AtomicBoolean RAN = new AtomicBoolean(false); + + @Override + public void init(FilterConfig filterConfig) { + } + + @Override + public void doFilter(ServletRequest request, ServletResponse response, + FilterChain chain) throws IOException, ServletException { + RAN.set(true); + chain.doFilter(request, response); + } + + @Override + public void destroy() { + } + } + + /** + * A downstream servlet and filter, registered through the public HttpServer2 + * API, still serve a request on the embedded container. + */ + @Test + public void testDownstreamServletAndFilterStillServeRequests() + throws Exception { + DownstreamFilter.RAN.set(false); + HttpServer2 server = createTestServer(); + try { + server.addServlet("downstream", "/downstream", DownstreamServlet.class); + server.addFilter("downstream-filter", + DownstreamFilter.class.getName(), null); + server.start(); + + URL url = new URL(getServerURL(server), "/downstream"); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.connect(); + + assertEquals(HttpServletResponse.SC_OK, conn.getResponseCode()); + assertEquals("downstream-ok", body(conn)); + assertTrue(DownstreamFilter.RAN.get(), + "a downstream filter registered through addFilter did not run"); + } finally { + stop(server); + } + } + + private static String body(HttpURLConnection conn) throws IOException { + try (InputStream in = conn.getInputStream(); + Scanner scanner = new Scanner(in, StandardCharsets.UTF_8.name())) { + scanner.useDelimiter("\\A"); + return scanner.hasNext() ? scanner.next().trim() : ""; + } + } + + /** + * The servlet API a downstream project resolves alongside Hadoop is the + * javax one, and it is loadable from Hadoop's own classpath. Guards against + * an upgrade that quietly swaps the namespace out from under an embedder. + */ + @Test + public void testServletApiOnTheClasspathIsJavax() throws Exception { + Class servlet = Class.forName("javax.servlet.Servlet"); + assertEquals("javax.servlet.Servlet", servlet.getName()); + assertTrue(servlet.isInterface()); + + // HttpServer2 is the embedding entry point; what it hands a downstream + // servlet has to be the same javax.servlet.Servlet loaded here. + assertTrue(servlet.isAssignableFrom(DownstreamServlet.class), + "HttpServlet no longer implements the javax.servlet.Servlet on" + + " the classpath"); + + List jakarta = new ArrayList<>(); + for (String name : Arrays.asList("jakarta.servlet.Servlet", + "jakarta.servlet.http.HttpServlet")) { + try { + Class.forName(name); + jakarta.add(name); + } catch (ClassNotFoundException expected) { + // the jakarta namespace must not be resolvable beside the javax one + } + } + assertTrue(jakarta.isEmpty(), + "both servlet namespaces are on the classpath: " + jakarta); + } +} From c6a4bef401ad4b866dd13b6dc59d8404cf0957e9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Mon, 24 Aug 2026 19:52:23 +0000 Subject: [PATCH 13/30] HADOOP-19972. Let a closed listener be reopened again on Jetty 12. Jetty 9.4's ServerConnector.close() released the accept channel, so a listener could re-bind. Jetty 12's does not, and a connector that has been closed cannot be started again. Hand it a fresh channel instead. Found by pjfanning on PR #8653. Co-Authored-By: Claude --- .../org/apache/hadoop/http/HttpServer2.java | 69 ++++++++++++++++++- 1 file changed, 68 insertions(+), 1 deletion(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java index 0d54e2b2e77d55..25818cd28da1e1 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java @@ -28,6 +28,7 @@ import java.net.MalformedURLException; import java.net.URI; import java.net.URL; +import java.nio.channels.ServerSocketChannel; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; @@ -611,7 +612,7 @@ HttpServer2 addConnectors( private ServerConnector createHttpChannelConnector( Server server, HttpConfiguration httpConfig) { - ServerConnector conn = new ServerConnector(server, + ServerConnector conn = new ReopenableServerConnector(server, conf.getInt(HTTP_ACCEPTOR_COUNT_KEY, HTTP_ACCEPTOR_COUNT_DEFAULT), conf.getInt(HTTP_SELECTOR_COUNT_KEY, HTTP_SELECTOR_COUNT_DEFAULT)); ConnectionFactory connFactory = new HttpConnectionFactory(httpConfig); @@ -1545,9 +1546,75 @@ private static void bindListener(ServerConnector listener) throws Exception { // failed to open w/o issuing a close first, even if the port is changed listener.close(); listener.open(); + if (listener.getLocalPort() < 0) { + // open() came back without an exception and without a socket. That is + // not a port conflict, so it must not be retried on the next port: the + // retry would no-op in exactly the same way, forever. See + // ReopenableServerConnector for the one case that used to get here. + throw new IllegalStateException( + "Jetty reported no error but did not bind " + listener); + } LOG.info("Jetty bound to port " + listener.getLocalPort()); } + /** + * A ServerConnector whose listen socket can be reopened after it is closed. + *

+ * Jetty 9.4's ServerConnector#close released the accept channel as well as + * closing it, so a close/open pair rebound the same port. Jetty 12 closes + * the channel but keeps the reference, and ServerConnector#open does nothing + * at all while that reference is set - it neither rebinds nor complains, and + * getLocalPort stays at -2. Only doStop clears it. + *

+ * That is enough for a server that is started, because stopping it clears + * the channel on the way down. It is not enough here: openListeners binds + * before the Server is started, so a caller that opens listeners, stops, and + * opens them again gets a connector that silently never comes back. + *

+ * The public open(ServerSocketChannel) overload assigns the channel + * unconditionally, so handing it a fresh one restores the 9.4 behaviour + * without reaching into Jetty's internals. That is done only after Jetty's + * own open() has been given the first go and turned out to be a no-op, so + * the ordinary first bind runs the stock path unchanged; the replacement + * path does not fire the connector's open callbacks, which is a step up + * from the nothing it does today and the reason to keep it to that one + * case. + */ + private static final class ReopenableServerConnector extends ServerConnector { + + private ReopenableServerConnector(Server server, int acceptors, + int selectors) { + super(server, acceptors, selectors); + } + + @Override + public void open() throws IOException { + // Always give Jetty the first go, so a connector that has never been + // opened takes the stock path and nothing - the open callbacks it fires + // among them - is skipped. + super.open(); + if (isStarted() || getLocalPort() != -2 || isOpen()) { + return; + } + // super.open() returned having bound nothing: the connector is holding + // a channel it closed. isOpen() being false is what says the channel is + // spent - Jetty only closes it from close() when the connector has + // acceptors, and binding a second channel to a port the first one still + // holds would fail and leak the first. + ServerSocketChannel channel = openAcceptChannel(); + try { + open(channel); + } catch (Throwable t) { + try { + channel.close(); + } catch (IOException suppressed) { + t.addSuppressed(suppressed); + } + throw t; + } + } + } + /** * Create bind exception by wrapping the bind exception thrown. * @param listener From 550e353263aea779a332469d868bf879d125ccfc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 22:23:56 +0000 Subject: [PATCH 14/30] HADOOP-19972. Let a path with an empty segment reach the servlet again. WebHDFS answers a bad path with its own JSON RemoteException, and its clients parse that body. On Jetty 12 they were getting a bare 400 with no body at all, because the connector rejects the URI before any servlet runs: //tmp//file trips UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT, which DEFAULT does not allow and Jetty 9.4 did. Reproduced against 12.0.37 with a servlet that answers 400 with a JSON body, driven over a raw socket so the path is sent as written: PUT /webhdfs/v1/tmp/file -> 400 len=-1 application/json {"RemoteException"... PUT /webhdfs/v1//tmp//file -> 400 len=0 ctype=null (empty) The second never reached the servlet, which is why WebHdfsFileSystem.jsonParse returns null - it returns null exactly when Content-Length is 0 - and TestRouterWebHdfsMethods then dies on a NullPointerException rather than reading the exception it asked for. Only that one violation is allowed back. The ambiguities that let a request read as one path to a filter and another to a servlet stay rejected, and the same probe confirms it: with the violation allowed, tmp%2Ffile is still refused and a/../../etc still never reaches the servlet. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../org/apache/hadoop/http/HttpServer2.java | 12 ++++++ .../apache/hadoop/http/TestHttpServer.java | 43 +++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java index 25818cd28da1e1..848dc0af0b31c0 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java @@ -100,6 +100,7 @@ import org.eclipse.jetty.ee8.servlet.ServletMapping; import org.eclipse.jetty.ee8.webapp.WebAppContext; import org.eclipse.jetty.http.HttpVersion; +import org.eclipse.jetty.http.UriCompliance; import org.eclipse.jetty.server.ConnectionFactory; import org.eclipse.jetty.server.Connector; import org.eclipse.jetty.server.Handler; @@ -551,6 +552,17 @@ public HttpServer2 build() throws IOException { httpConfig.setRequestHeaderSize(requestHeaderSize); httpConfig.setResponseHeaderSize(responseHeaderSize); httpConfig.setSendServerVersion(false); + // Jetty 12 rejects a path with an empty segment - //tmp//file - at the + // connector, with a bare 400 and no body, so the request never reaches + // the servlet. WebHDFS has always taken those paths and answered with + // its own JSON RemoteException, which is what its clients parse, and + // Jetty 9.4 let them through to do it. The one violation is allowed back + // so that Hadoop keeps deciding what a path means; every other ambiguity + // DEFAULT rejects - an encoded separator, an ambiguous segment or + // parameter - stays rejected, because those are the ones that let a + // request read as one path to a filter and another to a servlet. + httpConfig.setUriCompliance(UriCompliance.DEFAULT.with( + "hadoop", UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT)); int backlogSize = conf.getInt(HTTP_SOCKET_BACKLOG_SIZE_KEY, HTTP_SOCKET_BACKLOG_SIZE_DEFAULT); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java index 0cf5a82ffd9622..a82619600220e1 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java @@ -111,6 +111,20 @@ public void doGet(HttpServletRequest request, } } + /** Writes back the path info exactly as the container parsed it. */ + @SuppressWarnings("serial") + public static class PathInfoServlet extends HttpServlet { + @Override + public void doGet(HttpServletRequest request, HttpServletResponse response) + throws IOException { + response.setContentType("text/plain; charset=utf-8"); + response.setStatus(HttpServletResponse.SC_OK); + try (PrintWriter out = response.getWriter()) { + out.println(request.getPathInfo()); + } + } + } + @SuppressWarnings("serial") public static class EchoServlet extends HttpServlet { @SuppressWarnings("unchecked") @@ -156,6 +170,7 @@ public static void setup() throws Exception { CommonConfigurationKeysPublic.HADOOP_HTTP_METRICS_ENABLED, true); server = createTestServer(conf); server.addServlet("echo", "/echo", EchoServlet.class); + server.addServlet("pathinfo", "/pathinfo/*", PathInfoServlet.class); server.addServlet("echomap", "/echomap", EchoMapServlet.class); server.addServlet("htmlcontent", "/htmlcontent", HtmlContentServlet.class); server.addServlet("longheader", "/longheader", LongHeaderServlet.class); @@ -302,6 +317,34 @@ public void testStaticContextDoesNotListDirectories() throws Exception { assertEquals(HttpServletResponse.SC_OK, conn.getResponseCode()); } + /** + * A path with an empty segment has to reach the servlet, which is where + * Hadoop decides what it means. Jetty 12 rejects one at the connector by + * default, with a bare 400 and no body - WebHDFS clients parse a JSON + * RemoteException out of that response, and there is nothing there to parse. + * The ambiguities that actually matter stay rejected, so an encoded + * separator is asserted here too. + */ + @Test + public void testEmptyPathSegmentReachesTheServlet() throws Exception { + URL url = new URL(baseUrl, "/pathinfo//tmp//file"); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.connect(); + assertEquals(HttpServletResponse.SC_OK, conn.getResponseCode(), + "an empty path segment was rejected before the servlet ran"); + assertEquals("//tmp//file", readOutput(url).trim(), + "the servlet did not see the path as sent"); + + // An encoded separator is a different question and must still be refused. + URL encoded = new URL(baseUrl, "/pathinfo/tmp%2Ffile"); + HttpURLConnection encodedConn = + (HttpURLConnection) encoded.openConnection(); + encodedConn.connect(); + assertThat(encodedConn.getResponseCode()) + .as("an encoded path separator must not be accepted") + .isNotEqualTo(HttpServletResponse.SC_OK); + } + @Test public void testHttpServer2Metrics() throws Exception { final HttpServer2Metrics metrics = server.getMetrics(); From a3b63624c7cf325bbd98bc1a2abc156b92ca54c5 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 22:24:07 +0000 Subject: [PATCH 15/30] HADOOP-19972. Assert the refusal a renew now comes back with. Reporting the renew failure through HttpExceptionUtils rather than throwing it changed what the caller receives, and these four cases were still asserting the old shape. The client gets the server's own exception back now - an AccessControlException carrying "bar tries to renew a token ... with non-matching renewer foo" - because the JSON error envelope reaches it and HttpExceptionUtils.validateResponse rebuilds the named type. Before, the body was not parseable, so validateResponse fell back to a generic IOException quoting the status line, and assertTrue(msg.contains("403")) was pinning that fallback. The detail the caller actually wanted was being discarded, and the assertion passed anyway. Asserting the type and the reason is the stronger check: the status code only ever appeared in the message that meant the round trip had failed. Verified against trunk, where these four pass on Jetty 9.4, so the old assertions were not wrong then - they were describing a worse response. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../web/TestWebDelegationToken.java | 28 +++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java index c39296c4d958c8..ce3867afd483b1 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java @@ -21,6 +21,7 @@ import org.apache.commons.io.IOUtils; import org.apache.hadoop.io.Text; import org.apache.hadoop.minikdc.MiniKdc; +import org.apache.hadoop.security.AccessControlException; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.authentication.KerberosTestUtils; import org.apache.hadoop.security.authentication.client.AuthenticationException; @@ -78,6 +79,7 @@ import java.util.concurrent.Callable; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -344,6 +346,28 @@ public void testDelegationTokenAuthenticatorCallsWithQueryString() testDelegationTokenAuthenticatorCalls(true); } + /** + * A renewal refused by the server has to reach the caller as the exception + * the server raised, carrying the reason it gave. + *

+ * The handler reports this through HttpExceptionUtils, which writes the + * error as JSON and lets the client rebuild the original type. These calls + * used to assert that the message contained "403" instead, which only held + * while the body was unparseable: the client fell back to a generic + * IOException that quoted the status line. That fallback is what the + * assertion was really pinning, so it went on passing while the detail the + * caller needed was being thrown away. + * + * @param ex the exception the client raised + */ + private static void assertRenewRefused(Exception ex) { + assertInstanceOf(AccessControlException.class, ex, + "renewal refusal did not survive the round trip as the server's own" + + " exception; got " + ex.getClass().getName() + ": " + + ex.getMessage()); + assertTrue(ex.getMessage().contains("renew"), + "refusal lost the server's reason: " + ex.getMessage()); + } private void testDelegationTokenAuthenticatorCalls(final boolean useQS) throws Exception { @@ -394,7 +418,7 @@ private void testDelegationTokenAuthenticatorCalls(final boolean useQS) aUrl.renewDelegationToken(authURL2, token); fail(); } catch (Exception ex) { - assertTrue(ex.getMessage().contains("403")); + assertRenewRefused(ex); } aUrl.getDelegationToken(authURL, token, FOO_USER); @@ -817,7 +841,7 @@ public Void call() throws Exception { aUrl.renewDelegationToken(url, token, doAsUser); fail(); } catch (Exception ex) { - assertTrue(ex.getMessage().contains("403")); + assertRenewRefused(ex); } aUrl.getDelegationToken(url, token, FOO_USER, doAsUser); From 1c1f937c9e866dd870b83ab6956390146421a17c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 22:31:38 +0000 Subject: [PATCH 16/30] HADOOP-19972. Let a path with an encoded percent reach the servlet too. A file whose name contains a '%' is written on the wire as %25, and that trips UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING, which DEFAULT does not allow. Names like that are ordinary in HDFS, and YARN routes carry them: TestWebApp#testEncodedUrl asks for "@;%$", which encodes to %40%3B%25%24 and came back 400 with no body. Isolated by adding one violation at a time to DEFAULT and re-running the same two requests. The encoded semicolon is not the problem - it is the encoded percent: + AMBIGUOUS_PATH_PARAMETER @;%$ -> 400 %2F -> 400 + AMBIGUOUS_PATH_SEGMENT @;%$ -> 400 %2F -> 400 + AMBIGUOUS_PATH_ENCODING @;%$ -> ok %2F -> 400 + SUSPICIOUS_PATH_CHARACTERS @;%$ -> 400 %2F -> 400 Allowing it does not widen anything else. With both violations set, a%2Fb and a%2E%2E%2Fb are still refused, .. still cannot climb out of the context, and a%252Fb still decodes once, to the literal a%2Fb, rather than to a separator. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../org/apache/hadoop/http/HttpServer2.java | 29 +++++++----- .../apache/hadoop/http/TestHttpServer.java | 45 +++++++++++++------ 2 files changed, 50 insertions(+), 24 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java index 848dc0af0b31c0..ef4135c71eb82f 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java @@ -552,17 +552,24 @@ public HttpServer2 build() throws IOException { httpConfig.setRequestHeaderSize(requestHeaderSize); httpConfig.setResponseHeaderSize(responseHeaderSize); httpConfig.setSendServerVersion(false); - // Jetty 12 rejects a path with an empty segment - //tmp//file - at the - // connector, with a bare 400 and no body, so the request never reaches - // the servlet. WebHDFS has always taken those paths and answered with - // its own JSON RemoteException, which is what its clients parse, and - // Jetty 9.4 let them through to do it. The one violation is allowed back - // so that Hadoop keeps deciding what a path means; every other ambiguity - // DEFAULT rejects - an encoded separator, an ambiguous segment or - // parameter - stays rejected, because those are the ones that let a - // request read as one path to a filter and another to a servlet. - httpConfig.setUriCompliance(UriCompliance.DEFAULT.with( - "hadoop", UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT)); + // Jetty 12 rejects two kinds of path at the connector that 9.4 handed to + // the servlet, with a bare 400 and no body, so the request never reaches + // the code that knows what a Hadoop path is: + // + // - an empty segment, //tmp//file. WebHDFS takes those and answers with + // its own JSON RemoteException, which is what its clients parse. + // - an encoded percent, %25, which is how a file whose name contains a + // '%' is written on the wire. Names like that are ordinary in HDFS + // and YARN routes carry them too. + // + // Both are allowed back so that Hadoop keeps deciding what a path means. + // The ambiguities that let a request read as one path to a filter and + // another to a servlet stay rejected: an encoded separator (a%2Fb) and + // an encoded dot-segment (a%2E%2E%2Fb) are still refused, and .. still + // cannot climb out of the context. + httpConfig.setUriCompliance(UriCompliance.DEFAULT.with("hadoop", + UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT, + UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING)); int backlogSize = conf.getInt(HTTP_SOCKET_BACKLOG_SIZE_KEY, HTTP_SOCKET_BACKLOG_SIZE_DEFAULT); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java index a82619600220e1..f4b9ce4834ae3e 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java @@ -326,22 +326,41 @@ public void testStaticContextDoesNotListDirectories() throws Exception { * separator is asserted here too. */ @Test - public void testEmptyPathSegmentReachesTheServlet() throws Exception { - URL url = new URL(baseUrl, "/pathinfo//tmp//file"); + public void testHadoopPathsReachTheServlet() throws Exception { + assertPathInfo("//tmp//file", "/pathinfo//tmp//file"); + // A file whose name contains a '%' arrives as %25. + assertPathInfo("/a%b", "/pathinfo/a%25b"); + assertPathInfo("/@;%$", "/pathinfo/%40%3B%25%24"); + } + + /** + * The other half of the same setting: a path that would read as one thing + * to a filter and another to a servlet still has to be refused. + */ + @Test + public void testAmbiguousPathsAreStillRejected() throws Exception { + assertNotServed("an encoded path separator", "/pathinfo/tmp%2Ffile"); + assertNotServed("an encoded dot-segment", "/pathinfo/a%2E%2E%2Fb"); + } + + private static void assertPathInfo(String expected, String path) + throws Exception { + URL url = new URL(baseUrl, path); HttpURLConnection conn = (HttpURLConnection) url.openConnection(); conn.connect(); assertEquals(HttpServletResponse.SC_OK, conn.getResponseCode(), - "an empty path segment was rejected before the servlet ran"); - assertEquals("//tmp//file", readOutput(url).trim(), - "the servlet did not see the path as sent"); - - // An encoded separator is a different question and must still be refused. - URL encoded = new URL(baseUrl, "/pathinfo/tmp%2Ffile"); - HttpURLConnection encodedConn = - (HttpURLConnection) encoded.openConnection(); - encodedConn.connect(); - assertThat(encodedConn.getResponseCode()) - .as("an encoded path separator must not be accepted") + path + " was rejected before the servlet ran"); + assertEquals(expected, readOutput(url).trim(), + path + " did not reach the servlet as sent"); + } + + private static void assertNotServed(String what, String path) + throws Exception { + URL url = new URL(baseUrl, path); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.connect(); + assertThat(conn.getResponseCode()) + .as(what + " must not be served") .isNotEqualTo(HttpServletResponse.SC_OK); } From 7a450098f7acaab2da37cbb669cf5b71138e049d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 04:19:30 +0000 Subject: [PATCH 17/30] HADOOP-19972. Let an encoded backslash reach the servlet too. TestWebHdfsUrl creates files whose names contain the characters HDFS allows and a URL does not, and two of those tests came back 400 with no body: "Unexpected HTTP response: code=400 != 307, op=CREATE, message=Bad Request". The name is percent-encoded on the wire, so the connector was refusing an escape sequence rather than a character. Which one was measured rather than guessed. Sending each character of that filename on its own, percent-encoded, against the setting this branch already had, exactly one is refused: specialFile ?"\()[]_-=&+;,{}#%'`~!@$^*|<>. ^ %5C Jetty flags an encoded backslash as suspicious because it separates paths on Windows. On HDFS it is a character in a name like any other, and 9.4 passed it through, so SUSPICIOUS_PATH_CHARACTERS joins the two violations already allowed. Re-checked with all three set: a%2Fb and a%2E%2E%2Fb are still refused, .. still cannot climb out of the context, and a%252Fb still decodes once, to the literal a%2Fb rather than to a separator. TestWebHdfsUrl 11/11, TestHttpServer 34/34. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../org/apache/hadoop/http/HttpServer2.java | 19 +++++++++++++------ .../apache/hadoop/http/TestHttpServer.java | 2 ++ 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java index ef4135c71eb82f..b35472054f8daf 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java @@ -561,15 +561,22 @@ public HttpServer2 build() throws IOException { // - an encoded percent, %25, which is how a file whose name contains a // '%' is written on the wire. Names like that are ordinary in HDFS // and YARN routes carry them too. + // - an encoded backslash, %5C. Jetty treats it as suspicious because + // it separates paths on Windows; on HDFS it is just a character in a + // name, and TestWebHdfsUrl creates files containing one. Measured + // rather than assumed: of every character in that test's filename, + // %5C is the only one this violation gates. // - // Both are allowed back so that Hadoop keeps deciding what a path means. - // The ambiguities that let a request read as one path to a filter and - // another to a servlet stay rejected: an encoded separator (a%2Fb) and - // an encoded dot-segment (a%2E%2E%2Fb) are still refused, and .. still - // cannot climb out of the context. + // All three are allowed back so that Hadoop keeps deciding what a path + // means. The ambiguities that let a request read as one path to a filter + // and another to a servlet stay rejected: an encoded separator (a%2Fb) + // and an encoded dot-segment (a%2E%2E%2Fb) are still refused, .. still + // cannot climb out of the context, and a%252Fb still decodes once, to + // the literal a%2Fb rather than to a separator. httpConfig.setUriCompliance(UriCompliance.DEFAULT.with("hadoop", UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT, - UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING)); + UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING, + UriCompliance.Violation.SUSPICIOUS_PATH_CHARACTERS)); int backlogSize = conf.getInt(HTTP_SOCKET_BACKLOG_SIZE_KEY, HTTP_SOCKET_BACKLOG_SIZE_DEFAULT); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java index f4b9ce4834ae3e..64412d6a831211 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java @@ -331,6 +331,8 @@ public void testHadoopPathsReachTheServlet() throws Exception { // A file whose name contains a '%' arrives as %25. assertPathInfo("/a%b", "/pathinfo/a%25b"); assertPathInfo("/@;%$", "/pathinfo/%40%3B%25%24"); + // A '\' is a path separator on Windows and just a character on HDFS. + assertPathInfo("/a\\b", "/pathinfo/a%5Cb"); } /** From 51a0a901802ac7ac931615d213907d6b5ed85253 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 04:26:07 +0000 Subject: [PATCH 18/30] HADOOP-19972. Let a CSRF refusal carry its reason to the caller. TestWebHdfsWithRestCsrfPreventionFilter passes 32/32 on trunk and fails six times here, all on the same assertion: the exception the client raises no longer contains "Missing Required Header". The filter refuses with sendError(400, message). That message used to reach the caller in the reason phrase. Jetty 12 does not send one, and what stands in its place is the container's HTML error page - which WebHdfsFileSystem.jsonParse refuses on its content type, so validateResponse falls back to quoting the status line and the caller is told only "Bad Request". Reported as the JSON envelope HttpExceptionUtils writes instead, which is how the rest of Hadoop's HTTP surface answers a refusal and what its clients already parse. Same move this branch made for the delegation token renewal path, for the same reason. Only the servlet side changes. HttpInteraction has four implementations; the two Netty ones serving the DataNode build their own responses and never went through a reason phrase, so they are left alone. TestRestCsrfPreventionFilter verified that sendError had been called, which pins the mechanism rather than the result - and would have gone on passing while the caller learned nothing. It now asserts the status and that the message is in what the caller can read. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../http/RestCsrfPreventionFilter.java | 19 ++++++++++-- .../http/TestRestCsrfPreventionFilter.java | 29 ++++++++++++++++--- 2 files changed, 41 insertions(+), 7 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/RestCsrfPreventionFilter.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/RestCsrfPreventionFilter.java index 7c3a6fa5a8d259..7c5da6fbd268ff 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/RestCsrfPreventionFilter.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/RestCsrfPreventionFilter.java @@ -36,6 +36,7 @@ import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.util.HttpExceptionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -271,9 +272,21 @@ public void proceed() throws IOException, ServletException { @Override public void sendError(int code, String message) throws IOException { - // Jetty 12 never puts a reason phrase on the wire, so the detail is - // left to sendError, which writes it into the response body. - httpResponse.sendError(code, message); + // Reported as the JSON envelope HttpExceptionUtils writes, rather than + // handed to sendError for the container to render. + // + // The message used to travel in the reason phrase. Jetty 12 does not + // send one, and what sendError leaves in its place is the container's + // HTML error page - which WebHdfsFileSystem refuses on its content type + // and reports as a bare "Bad Request", losing the one thing the caller + // needed to know. The envelope is what the rest of Hadoop's HTTP + // surface already answers refusals with, and what its clients parse. + // + // Only the servlet side moves. The two Netty implementations of + // HttpInteraction that serve the DataNode build their own responses and + // never went through a reason phrase. + HttpExceptionUtils.createServletExceptionResponse(httpResponse, code, + new IOException(message)); } } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestRestCsrfPreventionFilter.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestRestCsrfPreventionFilter.java index f39dd1a103b452..d1402349b94573 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestRestCsrfPreventionFilter.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestRestCsrfPreventionFilter.java @@ -18,10 +18,13 @@ package org.apache.hadoop.security.http; +import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.Mockito.atLeastOnce; import static org.mockito.Mockito.verify; import java.io.IOException; +import java.io.PrintWriter; +import java.io.StringWriter; import javax.servlet.FilterChain; import javax.servlet.FilterConfig; @@ -68,6 +71,9 @@ public void testNoHeaderDefaultConfigBadRequest() // Objects to verify interactions based on request HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + Mockito.when(mockRes.getWriter()) + .thenReturn(new PrintWriter(writer)); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -75,8 +81,7 @@ public void testNoHeaderDefaultConfigBadRequest() filter.init(filterConfig); filter.doFilter(mockReq, mockRes, mockChain); - verify(mockRes, atLeastOnce()).sendError( - HttpServletResponse.SC_BAD_REQUEST, EXPECTED_MESSAGE); + assertRejected(mockRes, writer); verifyZeroInteractions(mockChain); } @@ -103,6 +108,9 @@ public void testNoHeaderCustomAgentConfigBadRequest() // Objects to verify interactions based on request HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + Mockito.when(mockRes.getWriter()) + .thenReturn(new PrintWriter(writer)); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -110,8 +118,7 @@ public void testNoHeaderCustomAgentConfigBadRequest() filter.init(filterConfig); filter.doFilter(mockReq, mockRes, mockChain); - verify(mockRes, atLeastOnce()).sendError( - HttpServletResponse.SC_BAD_REQUEST, EXPECTED_MESSAGE); + assertRejected(mockRes, writer); verifyZeroInteractions(mockChain); } @@ -360,4 +367,18 @@ public void testMissingHeaderMultipleIgnoreMethodsConfigBadRequest() verifyZeroInteractions(mockChain); } + + /** + * A refusal is reported as the JSON envelope HttpExceptionUtils writes, not + * handed to sendError, so what is asserted is the status and the message the + * caller will actually be able to read. + * + * @param res the mocked response the filter wrote to + * @param body what the filter wrote into it + */ + private static void assertRejected(HttpServletResponse res, StringWriter body) { + verify(res, atLeastOnce()).setStatus(HttpServletResponse.SC_BAD_REQUEST); + assertTrue(body.toString().contains(EXPECTED_MESSAGE), + "refusal did not carry the reason: " + body); + } } From e6eef6753ff65c6c7ca338a49c2319a76d5cd193 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 04:28:37 +0000 Subject: [PATCH 19/30] HADOOP-19972. Give every mocked response in the CSRF test a writer. The previous commit stubbed getWriter() only on the two mock responses whose tests read the body back. Three others reach the same refusal path and assert nothing about the response, so they went unstubbed and the filter's write landed on a null writer: testMissingHeaderWithCustomHeaderConfigBadRequest IllegalArgument argument "w" is null testMissingHeaderNoMethodsToIgnoreConfigBadRequest IllegalArgument argument "w" is null testMissingHeaderMultipleIgnoreMethodsConfigBadRequest IllegalArgument argument "w" is null Mine, and pushed before the run that would have caught it had finished. All ten mock responses now come from one factory that always supplies a writer, so a test added later cannot land in the same hole by saying nothing about the body. TestWebHdfsWithRestCsrfPreventionFilter, which the previous commit was for, is 32/32. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../http/TestRestCsrfPreventionFilter.java | 48 +++++++++++++------ 1 file changed, 34 insertions(+), 14 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestRestCsrfPreventionFilter.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestRestCsrfPreventionFilter.java index d1402349b94573..a5e30fe0ef437c 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestRestCsrfPreventionFilter.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/http/TestRestCsrfPreventionFilter.java @@ -70,10 +70,8 @@ public void testNoHeaderDefaultConfigBadRequest() thenReturn(BROWSER_AGENT); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); StringWriter writer = new StringWriter(); - Mockito.when(mockRes.getWriter()) - .thenReturn(new PrintWriter(writer)); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -107,10 +105,8 @@ public void testNoHeaderCustomAgentConfigBadRequest() thenReturn("curl"); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); StringWriter writer = new StringWriter(); - Mockito.when(mockRes.getWriter()) - .thenReturn(new PrintWriter(writer)); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -141,7 +137,8 @@ public void testNoHeaderDefaultConfigNonBrowserGoodRequest() thenReturn(NON_BROWSER); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -169,7 +166,8 @@ public void testHeaderPresentDefaultConfigGoodRequest() thenReturn("valueUnimportant"); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -198,7 +196,8 @@ public void testHeaderPresentCustomHeaderConfigGoodRequest() thenReturn("valueUnimportant"); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -229,7 +228,8 @@ public void testMissingHeaderWithCustomHeaderConfigBadRequest() thenReturn(null); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -261,7 +261,8 @@ public void testMissingHeaderNoMethodsToIgnoreConfigBadRequest() thenReturn("GET"); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -293,7 +294,8 @@ public void testMissingHeaderIgnoreGETMethodConfigGoodRequest() thenReturn("GET"); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -325,7 +327,8 @@ public void testMissingHeaderMultipleIgnoreMethodsConfigGoodRequest() thenReturn("OPTIONS"); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -357,7 +360,8 @@ public void testMissingHeaderMultipleIgnoreMethodsConfigBadRequest() thenReturn("PUT"); // Objects to verify interactions based on request - HttpServletResponse mockRes = Mockito.mock(HttpServletResponse.class); + StringWriter writer = new StringWriter(); + HttpServletResponse mockRes = mockResponse(writer); FilterChain mockChain = Mockito.mock(FilterChain.class); // Object under test @@ -381,4 +385,20 @@ private static void assertRejected(HttpServletResponse res, StringWriter body) { assertTrue(body.toString().contains(EXPECTED_MESSAGE), "refusal did not carry the reason: " + body); } + + /** + * A mock response the filter can write a refusal into. Every caller gets a + * writer whether or not it looks at one, because the filter now reports a + * refusal by writing the error rather than by calling sendError. + * + * @param body collects whatever the filter writes + * @return the mock response + * @throws IOException never, but getWriter declares it + */ + private static HttpServletResponse mockResponse(StringWriter body) + throws IOException { + HttpServletResponse res = Mockito.mock(HttpServletResponse.class); + Mockito.when(res.getWriter()).thenReturn(new PrintWriter(body)); + return res; + } } From c76fa07da9228d862ca7bfe5fb6df5b0461e7869 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 05:17:29 +0000 Subject: [PATCH 20/30] HADOOP-19972. Let a refused image transfer carry its reason. ImageServlet reported why it turned a transfer down through sendError, and the detail reached the caller in the HTTP reason phrase. Jetty 12 does not send a reason phrase, so the detail now lives only in the error page the container renders - and this servlet closes its output stream on the way out of every request, committing an empty response before that page is ever produced. The secondary namenode was left with a bare status code. The servlet now writes the reason into the body itself, which does not depend on how the container renders errors, and the three clients that read a failed transfer - Util#doGetUrl, TransferFsImage's upload, and EditLogFileInputStream - read the body before falling back to the phrase. TestCheckpoint's short-send case expects a different message now. Jetty 12 fails the servlet's own write as soon as it delivers fewer bytes than the Content-Length it announced, so the namenode reports the shortfall rather than putting a truncated body on the wire for the secondary to notice. The checkpoint still fails without corrupting the namenode, which is what the test guards. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../hadoop/hdfs/server/common/Util.java | 74 ++++++++++++++++++- .../namenode/EditLogFileInputStream.java | 3 +- .../hdfs/server/namenode/ImageServlet.java | 23 +++++- .../hdfs/server/namenode/TransferFsImage.java | 2 +- .../hdfs/server/namenode/TestCheckpoint.java | 14 +++- 5 files changed, 106 insertions(+), 10 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java index 5039db6ceb4882..820ae502d0efb6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java @@ -21,12 +21,14 @@ import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; +import java.io.InputStreamReader; import java.net.HttpURLConnection; import java.net.InetSocketAddress; import java.net.URI; import java.net.URISyntaxException; import java.net.URL; import java.net.UnknownHostException; +import java.nio.charset.StandardCharsets; import java.security.DigestInputStream; import java.security.MessageDigest; import java.util.ArrayList; @@ -170,7 +172,7 @@ public static MD5Hash doGetUrl(URL url, List localPaths, if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) { throw new HttpGetFailedException("Image transfer servlet at " + url + " failed with status code " + connection.getResponseCode() + - "\nResponse message:\n" + connection.getResponseMessage(), + "\nResponse message:\n" + getResponseDetail(connection), connection); } @@ -192,6 +194,76 @@ public static MD5Hash doGetUrl(URL url, List localPaths, throttler); } + /** How much of a failed response body is worth quoting back. */ + private static final int MAX_RESPONSE_DETAIL_CHARS = 4096; + + /** + * Describes why a request failed, preferring the response body over the + * HTTP reason phrase. + *

+ * The servlets on the other end of these transfers report their reason + * through {@link javax.servlet.http.HttpServletResponse#sendError}. That + * detail used to travel in the reason phrase, which + * {@link HttpURLConnection#getResponseMessage()} returns. Jetty 12 never + * puts a reason phrase on the wire, so the phrase is now always the + * canonical text for the status code - "Forbidden", "Gone" - and the + * detail is in the body instead. Read the body, and fall back to the + * phrase when there is none. + * + * @param connection a connection whose response status has been read + * @return a description of the failure, never null + */ + public static String getResponseDetail(HttpURLConnection connection) { + String body = ""; + try (InputStream err = connection.getErrorStream()) { + if (err != null) { + body = toPlainText(readCapped(err)); + } + } catch (IOException e) { + LOG.debug("Could not read the error response body from {}", + connection.getURL(), e); + } + if (!body.isEmpty()) { + return body; + } + try { + String phrase = connection.getResponseMessage(); + return phrase == null ? "" : phrase; + } catch (IOException e) { + return ""; + } + } + + private static String readCapped(InputStream in) throws IOException { + InputStreamReader reader = + new InputStreamReader(in, StandardCharsets.UTF_8); + StringBuilder sb = new StringBuilder(); + char[] buf = new char[1024]; + int n; + while (sb.length() < MAX_RESPONSE_DETAIL_CHARS + && (n = reader.read(buf)) != -1) { + sb.append(buf, 0, Math.min(n, MAX_RESPONSE_DETAIL_CHARS - sb.length())); + } + return sb.toString(); + } + + /** + * Reduces a response body to something readable in a log line. A container + * that renders sendError as an HTML page buries the message in markup; strip + * it out rather than quoting the page. + */ + private static String toPlainText(String body) { + String text = body; + if (text.indexOf('<') >= 0) { + text = text.replaceAll("(?s)<(script|style)\\b.*?", " ") + .replaceAll("(?s)<[^>]*>", " "); + } + text = text.replace("<", "<").replace(">", ">") + .replace(""", "\"").replace("'", "'") + .replace("&", "&"); + return text.replaceAll("\\s+", " ").trim(); + } + /** * Receives file at the url location from the input stream and puts them in * the specified destination storage location. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java index ba4f32fd2154dd..d7e13577f1fe75 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java @@ -39,6 +39,7 @@ import org.apache.hadoop.hdfs.server.common.HdfsServerConstants; import org.apache.hadoop.hdfs.server.common.HttpGetFailedException; import org.apache.hadoop.hdfs.server.common.Storage; +import org.apache.hadoop.hdfs.server.common.Util; import org.apache.hadoop.hdfs.web.URLConnectionFactory; import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.security.SecurityUtil; @@ -494,7 +495,7 @@ public InputStream run() throws IOException { throw new HttpGetFailedException( "Fetch of " + url + " failed with status code " + connection.getResponseCode() + - "\nResponse message:\n" + connection.getResponseMessage(), + "\nResponse message:\n" + Util.getResponseDetail(connection), connection); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java index b6d7568d008ad2..02dc270548058f 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java @@ -28,6 +28,7 @@ import static org.apache.hadoop.util.Time.monotonicNow; import java.net.HttpURLConnection; +import java.nio.charset.StandardCharsets; import java.security.PrivilegedExceptionAction; import java.util.*; import java.io.*; @@ -702,9 +703,25 @@ public Void run() throws Exception { private void sendError(HttpServletResponse response, int code, String message) throws IOException { - // Jetty 12 never puts a reason phrase on the wire, so the detail is left - // to sendError, which writes it into the response body. - response.sendError(code, message); + // Write the reason into the body rather than leaving it to sendError. + // + // Jetty 12 no longer puts a reason phrase on the wire, so sendError's + // message survives only in the error page the container renders. This + // servlet closes its output stream on the way out of every request, which + // commits an empty response before that page is ever produced - the + // client then sees a bare status code and no reason at all. Writing the + // body here keeps the detail, and does not depend on how the container + // renders errors. + if (response.isCommitted()) { + LOG.warn("Could not report \"{}\": the response is already committed.", + message); + return; + } + byte[] body = message.getBytes(StandardCharsets.UTF_8); + response.setStatus(code); + response.setContentType("text/plain; charset=utf-8"); + response.setContentLength(body.length); + response.getOutputStream().write(body); } /* diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java index ac31c16c91720f..5eb5c418b981d1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java @@ -317,7 +317,7 @@ private static void uploadImage(URL url, Configuration conf, if (responseCode != HttpURLConnection.HTTP_OK) { throw new HttpPutFailedException(String.format( "Image uploading failed, status: %d, url: %s, message: %s", - responseCode, urlWithParams, connection.getResponseMessage()), + responseCode, urlWithParams, Util.getResponseDetail(connection)), responseCode); } } catch (AuthenticationException | URISyntaxException e) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java index 8be6cbef5106f5..d62fed5a7cbf27 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java @@ -87,7 +87,6 @@ import org.apache.hadoop.util.ExitUtil; import org.apache.hadoop.util.ExitUtil.ExitException; import org.apache.hadoop.util.Lists; -import org.apache.hadoop.util.Shell; import org.apache.hadoop.util.StringUtils; import org.slf4j.event.Level; import org.junit.jupiter.api.AfterEach; @@ -654,9 +653,16 @@ public void testNameNodeImageSendFailWrongSize() Mockito.doReturn(true).when(faultInjector) .shouldSendShortFile(filePathContaining("fsimage")); - String expectedText = Shell.isJavaVersionAtLeast(24) ? "Premature EOF" - : "is not of the advertised size"; - doSendFailTest(expectedText); + // Where a short send is caught decides how it reads. Jetty 12 fails the + // servlet's own write as soon as it delivers fewer bytes than the + // Content-Length it announced, so the namenode reports the shortfall + // ("written 200 < 399 content-length") rather than putting a truncated + // body on the wire for the secondary to notice. Under Jetty 9.4 the + // truncated body did go out, and the secondary reported it - as "is not + // of the advertised size", or "Premature EOF" on JDK 24 and later. The + // checkpoint fails either way without corrupting the namenode, which is + // what this test guards. + doSendFailTest("content-length"); } /** From 9c34e0892c35cccb98877a6e79a26058671d37a9 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 05:17:37 +0000 Subject: [PATCH 21/30] HADOOP-19972. Clear the response charset along with the content type. Every server HttpServer2 builds runs QuotingInputFilter, which sets text/plain; charset=utf-8 before a request reaches its resource, so a JAX-RS resource that picks its own content type clears that first. Fourteen of them did it with setContentType(null). That is no longer enough. Jetty 12 drops the charset but remembers that one had been set explicitly, and appends the memory to the next content type - literally ";charset=null" for a type that carries no charset of its own. WebHDFS answered SETPERMISSION with "application/octet-stream;charset=null"; application/xml is affected the same way, while types that assume a charset, application/json among them, escape it. Clearing the character encoding as well resets that state. The idiom moves to JettyUtils#clearContentType so all fourteen call sites get it, and TestHttpServer asserts the header a resource that clears and re-sets its content type actually sends. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../org/apache/hadoop/http/JettyUtils.java | 22 ++++++++++ .../apache/hadoop/http/TestHttpServer.java | 43 +++++++++++++++++++ .../web/resources/NamenodeWebHdfsMethods.java | 3 +- .../hdfs/web/resources/ExceptionHandler.java | 4 +- .../v2/app/webapp/AMWebServices.java | 3 +- .../mapreduce/v2/hs/webapp/HsWebServices.java | 3 +- .../yarn/webapp/GenericExceptionHandler.java | 4 +- .../timeline/webapp/TimelineWebServices.java | 2 +- .../yarn/server/webapp/LogWebService.java | 3 +- .../yarn/server/webapp/WebServices.java | 4 +- .../nodemanager/webapp/NMWebServices.java | 3 +- .../resourcemanager/webapp/RMWebServices.java | 6 +-- .../router/webapp/RouterWebServices.java | 3 +- .../TimelineCollectorWebService.java | 2 +- .../reader/TimelineReaderWebServices.java | 2 +- 15 files changed, 82 insertions(+), 25 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/JettyUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/JettyUtils.java index 29c0930968ede7..09b08d411725ea 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/JettyUtils.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/JettyUtils.java @@ -18,6 +18,8 @@ package org.apache.hadoop.http; +import javax.servlet.http.HttpServletResponse; + import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; @@ -30,6 +32,26 @@ public final class JettyUtils { public static final String UTF_8 = "charset=utf-8"; public static final int HEADER_SIZE = 1024 * 64; + /** + * Clears the content type a response is carrying, charset included. + *

+ * Every server built by {@link HttpServer2} runs + * {@code QuotingInputFilter}, which sets {@code text/plain; charset=utf-8} + * before the request reaches the resource. A JAX-RS resource that picks its + * own content type has to undo that first, and {@code setContentType(null)} + * alone is not enough on Jetty 12: it drops the charset but remembers that + * one had been set explicitly, so the next content type gets that memory + * appended to it - literally {@code ;charset=null} for a type that carries + * no charset of its own, such as {@code application/octet-stream} or + * {@code application/xml}. Clearing the encoding as well resets that state. + * + * @param response the response to clear + */ + public static void clearContentType(HttpServletResponse response) { + response.setContentType(null); + response.setCharacterEncoding(null); + } + private JettyUtils() { } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java index 64412d6a831211..86264e3f2f6995 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java @@ -162,6 +162,23 @@ public void doGet(HttpServletRequest request, } } + /** + * Stands in for a JAX-RS resource that picks its own content type: it drops + * whatever QuotingInputFilter left on the response, then sets the type the + * way Jersey does, through a header. + */ + @SuppressWarnings("serial") + public static class OwnContentTypeServlet extends HttpServlet { + @Override + public void doGet(HttpServletRequest request, + HttpServletResponse response + ) throws ServletException, IOException { + JettyUtils.clearContentType(response); + response.addHeader("Content-Type", request.getParameter("type")); + response.setStatus(HttpServletResponse.SC_OK); + } + } + @BeforeAll public static void setup() throws Exception { Configuration conf = new Configuration(); @@ -174,6 +191,8 @@ public static void setup() throws Exception { server.addServlet("echomap", "/echomap", EchoMapServlet.class); server.addServlet("htmlcontent", "/htmlcontent", HtmlContentServlet.class); server.addServlet("longheader", "/longheader", LongHeaderServlet.class); + server.addServlet("owncontenttype", "/owncontenttype", + OwnContentTypeServlet.class); server.addJerseyResourcePackage( JerseyResource.class.getPackage().getName(), "/jersey/*"); server.start(); @@ -294,6 +313,30 @@ public void testAcceptorSelectorConfigurability() throws Exception { conn.getContentType()); } + /** + * A resource that clears the content type QuotingInputFilter set, then picks + * its own, must not be given a charset back. + *

+ * Jetty 12 remembers that a charset had been set explicitly even after the + * content type carrying it is cleared, and appends that memory to the next + * content type - as ";charset=null" for a type with no charset of its own. + * Only types that assume a charset, application/json among them, escape it, + * which is why octet-stream and xml are the ones asserted here. + */ + @Test + public void testClearedContentTypeCarriesNoCharset() throws Exception { + for (String type : new String[] {MediaType.APPLICATION_OCTET_STREAM, + MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) { + URL url = new URL(baseUrl, "/owncontenttype?type=" + type); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.connect(); + assertEquals(HttpServletResponse.SC_OK, conn.getResponseCode()); + assertEquals(type, conn.getContentType(), + "the cleared charset came back for " + type); + conn.disconnect(); + } + } + /** * /static must never list what is in it. The setting that stops it is a * context init parameter, which the DefaultServlet reads under a prefix of diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java index 443c1836351ed3..b1f924123bfc72 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java @@ -178,8 +178,7 @@ protected void init(final UserGroupInformation ugi, + Param.toSortedString(", ", parameters)); } - //clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); } private static NamenodeProtocols getRPCServer(NameNode namenode) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExceptionHandler.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExceptionHandler.java index 038f9275f57c88..e911755099ac89 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExceptionHandler.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ExceptionHandler.java @@ -30,6 +30,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.hadoop.hdfs.web.JsonUtil; +import org.apache.hadoop.http.JettyUtils; import org.apache.hadoop.ipc.RemoteException; import org.apache.hadoop.ipc.StandbyException; import org.apache.hadoop.security.authorize.AuthorizationException; @@ -75,8 +76,7 @@ public Response toResponse(Exception e) { LOG.trace("GOT EXCEPITION", e); } - //clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); //Convert exception if (e instanceof ParamException) { diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java index 2aa3dcb6441eff..549726a4537260 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java @@ -115,8 +115,7 @@ Boolean hasAccess(Job job, HttpServletRequest request) { } private void init() { - //clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); } public static Job getJobFromContainerIdString(String cid, AppContext appCtx) diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java index 3efa09ec0d2e51..fcb70552f01fd3 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java @@ -150,8 +150,7 @@ private boolean isMRJobContainer(String containerIdStr) { } private void init() { - //clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); } @VisibleForTesting diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/GenericExceptionHandler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/GenericExceptionHandler.java index 00b725e06cf571..acc6ffad4ee2ff 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/GenericExceptionHandler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/GenericExceptionHandler.java @@ -32,6 +32,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.http.JettyUtils; import org.apache.hadoop.ipc.RemoteException; import org.apache.hadoop.security.authorize.AuthorizationException; @@ -61,8 +62,7 @@ public Response toResponse(Exception e) { if (e instanceof javax.ws.rs.NotFoundException) { return ((javax.ws.rs.NotFoundException) e).getResponse(); } - // clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); // Convert exception if (e instanceof RemoteException) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/webapp/TimelineWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/webapp/TimelineWebServices.java index 7a38c59b002e0a..bcd49c501a77cc 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/webapp/TimelineWebServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/webapp/TimelineWebServices.java @@ -360,7 +360,7 @@ public TimelineDomains getDomains( } private void init(HttpServletResponse response) { - response.setContentType(null); + JettyUtils.clearContentType(response); } private static UserGroupInformation getUser(HttpServletRequest req) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogWebService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogWebService.java index c537df226a0248..e8783866b49072 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogWebService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogWebService.java @@ -122,8 +122,7 @@ private Client createTimelineWebClient() { } private void initForReadableEndpoints(HttpServletResponse response) { - // clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); } /** diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java index f62a490dd2d0b6..cbf9031f725946 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java @@ -32,6 +32,7 @@ import javax.ws.rs.WebApplicationException; import org.apache.commons.lang3.Range; +import org.apache.hadoop.http.JettyUtils; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.authorize.AuthorizationException; import org.apache.hadoop.util.StringUtils; @@ -418,8 +419,7 @@ public ContainerInfo getContainer(HttpServletRequest req, } protected void initForReadableEndpoints(HttpServletResponse response) { - // clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); } public static Set diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java index 25a7c91a020882..5e5227d5ba20c7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java @@ -141,8 +141,7 @@ public NMWebServices(final Context nm, final ResourceView view, } private void init() { - //clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); } @GET diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java index b6ea2d438c0b41..8a6c52bd170dfc 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java @@ -328,8 +328,7 @@ protected Boolean hasAccess(RMApp app, HttpServletRequest hsr) { * initForReadableEndpoints does the init for all readable REST end points. */ private void initForReadableEndpoints() { - // clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); } /** @@ -345,8 +344,7 @@ private void initForReadableEndpoints() { */ private void initForWritableEndpoints(UserGroupInformation callerUGI, boolean doAdminACLsCheck) throws AuthorizationException { - // clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); if (callerUGI == null) { String msg = "Unable to obtain user name, user not authenticated"; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServices.java index 1266598ff0e27e..e53c92d1cc2d82 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServices.java @@ -138,8 +138,7 @@ public RouterWebServices(final @Named("router") Router router, } private void init() { - // clear content type - response.setContentType(null); + JettyUtils.clearContentType(response); } @VisibleForTesting diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java index 6f95f3ed9edf3f..1fcfc46a9f1c32 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java @@ -279,7 +279,7 @@ private static ApplicationId parseApplicationId(String appId) { } private static void init(HttpServletResponse response) { - response.setContentType(null); + JettyUtils.clearContentType(response); } private static UserGroupInformation getUser(HttpServletRequest req) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java index a1c2e194216a1b..7f5d334a0c8320 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java @@ -98,7 +98,7 @@ protected DateFormat initialValue() { }; private void init(HttpServletResponse response) { - response.setContentType(null); + JettyUtils.clearContentType(response); } private static final class DateRange { From bbb4fa087e70f47c5b4fb9b239305fb4c67a1854 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 09:19:28 +0000 Subject: [PATCH 22/30] HADOOP-19972. Read a refusal's reason from the body, everywhere it is read. An audit of every sendError, two-argument setStatus and getResponseMessage call in the tree turned up two more clients that still read the detail from the HTTP reason phrase, which Jetty 12 no longer sends: WebHdfsFileSystem, on a 401, threw AccessControlException carrying only the phrase - now always "Unauthorized" rather than what the server said. AuthenticatedURL reported an authentication failure the same way, so an operator saw "message: Forbidden" instead of the reason. Both peers already put the reason in the body: AuthenticationFilter reports through sendError, which this branch changed it to. The body reader moves from Util to HttpExceptionUtils, next to the JSON envelope reader it complements, so hadoop-hdfs and hadoop-hdfs-client share one copy. hadoop-auth sits below hadoop-common and cannot reach it, so AuthenticatedURL keeps a small private one. Two findings from the same audit are deliberately left alone. NetworkTopologyServlet loses its reason the same way, but a probe against both containers shows Jetty 9.4 losing it identically - a failed topology dump answers 200 there too - so it is not this migration's regression and belongs in its own change. KMSClientProvider matches the phrase against two literals; the match is dead now, but its retry still works through the arm that does not read the phrase. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../client/AuthenticatedURL.java | 44 ++++++++++- .../hadoop/util/HttpExceptionUtils.java | 75 ++++++++++++++++++ .../hadoop/util/TestHttpExceptionUtils.java | 44 +++++++++++ .../hadoop/hdfs/web/WebHdfsFileSystem.java | 8 +- .../hadoop/hdfs/server/common/Util.java | 76 +------------------ .../namenode/EditLogFileInputStream.java | 5 +- .../hdfs/server/namenode/TransferFsImage.java | 4 +- 7 files changed, 178 insertions(+), 78 deletions(-) diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java index cb7d36368aa393..288e4b737ca132 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java @@ -19,11 +19,13 @@ import java.io.FileNotFoundException; import java.io.IOException; +import java.io.InputStream; import java.net.CookieHandler; import java.net.HttpCookie; import java.net.HttpURLConnection; import java.net.URI; import java.net.URL; +import java.nio.charset.StandardCharsets; import java.util.Arrays; import java.util.Collections; import java.util.HashMap; @@ -75,6 +77,9 @@ public class AuthenticatedURL { */ public static final String AUTH_COOKIE = "hadoop.auth"; + /** How much of a failed response body is worth quoting back. */ + private static final int MAX_RESPONSE_DETAIL_BYTES = 4096; + // a lightweight cookie handler that will be attached to url connections. // client code is not required to extract or inject auth cookies. private static class AuthCookieHandler extends CookieHandler { @@ -403,7 +408,44 @@ public static void extractToken(HttpURLConnection conn, Token token) throws IOEx throw new AuthenticationException("Authentication failed" + ", URL: " + conn.getURL() + ", status: " + conn.getResponseCode() + - ", message: " + conn.getResponseMessage()); + ", message: " + responseDetail(conn)); + } + } + + /** + * Why the server turned the request down. + *

+ * {@link AuthenticationFilter} reports its reason through + * {@link javax.servlet.http.HttpServletResponse#sendError}, which used to + * reach the caller in the HTTP reason phrase. Jetty 12 does not put a reason + * phrase on the wire, so {@link HttpURLConnection#getResponseMessage()} now + * only ever returns the canonical text for the status code and the reason is + * in the body. Read the body, and fall back to the phrase when it is empty. + */ + private static String responseDetail(HttpURLConnection conn) { + try (InputStream es = conn.getErrorStream()) { + if (es != null) { + byte[] body = new byte[MAX_RESPONSE_DETAIL_BYTES]; + int read = 0, n; + while (read < body.length + && (n = es.read(body, read, body.length - read)) != -1) { + read += n; + } + // A container renders sendError as an HTML page; the reason is in + // there among the markup, which is no use in a one-line message. + String text = new String(body, 0, read, StandardCharsets.UTF_8) + .replaceAll("(?s)<[^>]*>", " ").replaceAll("\\s+", " ").trim(); + if (!text.isEmpty()) { + return text; + } + } + } catch (IOException ex) { + // nothing to add: fall through to the reason phrase + } + try { + return conn.getResponseMessage(); + } catch (IOException ex) { + return null; } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java index 43441a5560a338..d3cf7df08fdcab 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java @@ -25,11 +25,13 @@ import javax.ws.rs.core.Response; import java.io.IOException; import java.io.InputStream; +import java.io.InputStreamReader; import java.io.Writer; import java.lang.invoke.MethodHandle; import java.lang.invoke.MethodHandles; import java.lang.invoke.MethodType; import java.net.HttpURLConnection; +import java.nio.charset.StandardCharsets; import java.util.Collections; import java.util.LinkedHashMap; import java.util.Map; @@ -190,4 +192,77 @@ public static void validateResponse(HttpURLConnection conn, } } + /** How much of a failed response body is worth quoting back. */ + private static final int MAX_RESPONSE_DETAIL_CHARS = 4096; + + /** + * Describes why a request failed, preferring the response body over the HTTP + * reason phrase. + *

+ * A servlet reports its reason through + * {@link HttpServletResponse#sendError}, and that detail used to reach the + * caller in the reason phrase, which + * {@link HttpURLConnection#getResponseMessage()} returns. Jetty 12 never + * puts a reason phrase on the wire: the phrase is now always the canonical + * text for the status code - "Forbidden", "Gone" - and the detail is in the + * body instead. Read the body, and fall back to the phrase when there is + * none. + *

+ * For a response that carries the JSON envelope this class writes, prefer + * {@link #validateResponse}, which rebuilds the original exception. This is + * for everything else: a container's error page, or a plain-text reason. + * + * @param conn a connection whose response status has been read + * @return a description of the failure, never null + */ + public static String getResponseDetail(HttpURLConnection conn) { + String body = ""; + try (InputStream es = conn.getErrorStream()) { + if (es != null) { + body = toPlainText(readCapped(es)); + } + } catch (IOException ex) { + // nothing to add: fall through to the reason phrase + } + if (!body.isEmpty()) { + return body; + } + try { + String phrase = conn.getResponseMessage(); + return phrase == null ? "" : phrase; + } catch (IOException ex) { + return ""; + } + } + + private static String readCapped(InputStream in) throws IOException { + InputStreamReader reader = + new InputStreamReader(in, StandardCharsets.UTF_8); + StringBuilder sb = new StringBuilder(); + char[] buf = new char[1024]; + int n; + while (sb.length() < MAX_RESPONSE_DETAIL_CHARS + && (n = reader.read(buf)) != -1) { + sb.append(buf, 0, Math.min(n, MAX_RESPONSE_DETAIL_CHARS - sb.length())); + } + return sb.toString(); + } + + /** + * Reduces a response body to something readable in a log line. A container + * that renders sendError as an HTML page buries the message in markup; strip + * it out rather than quoting the page. + */ + private static String toPlainText(String body) { + String text = body; + if (text.indexOf('<') >= 0) { + text = text.replaceAll("(?s)<(script|style)\\b.*?", " ") + .replaceAll("(?s)<[^>]*>", " "); + } + text = text.replace("<", "<").replace(">", ">") + .replace(""", "\"").replace("'", "'") + .replace("&", "&"); + return text.replaceAll("\\s+", " ").trim(); + } + } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java index f3f0e43b394194..fa294626e0a215 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java @@ -37,6 +37,8 @@ import static org.junit.jupiter.api.Assertions.assertArrayEquals; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.Mockito.eq; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.verify; @@ -178,4 +180,46 @@ public void testValidateResponseJsonErrorNonException() throws Exception { "java.lang.String", "EX"), () -> HttpExceptionUtils.validateResponse(conn, HttpURLConnection.HTTP_CREATED)); } + + private static HttpURLConnection connectionReturning(String body, + String phrase) throws IOException { + HttpURLConnection conn = mock(HttpURLConnection.class); + when(conn.getErrorStream()).thenReturn(body == null ? null + : new ByteArrayInputStream(body.getBytes(StandardCharsets.UTF_8))); + when(conn.getResponseMessage()).thenReturn(phrase); + return conn; + } + + @Test + public void testResponseDetailPrefersTheBody() throws Exception { + assertEquals("the real reason", HttpExceptionUtils.getResponseDetail( + connectionReturning("the real reason", "Forbidden"))); + } + + @Test + public void testResponseDetailStripsAnErrorPage() throws Exception { + // what a container renders for sendError(403, "the real reason") + String page = "\n\nError 403 the real reason\n" + + "\n

HTTP ERROR 403

\n" + + "
MESSAGE:the real reason
\n" + + "\n\n"; + String detail = HttpExceptionUtils.getResponseDetail( + connectionReturning(page, "Forbidden")); + assertTrue(detail.contains("the real reason"), detail); + assertFalse(detail.contains("<"), "markup survived: " + detail); + } + + @Test + public void testResponseDetailFallsBackToThePhrase() throws Exception { + assertEquals("Forbidden", HttpExceptionUtils.getResponseDetail( + connectionReturning(null, "Forbidden"))); + assertEquals("Forbidden", HttpExceptionUtils.getResponseDetail( + connectionReturning(" \n ", "Forbidden"))); + } + + @Test + public void testResponseDetailIsNeverNull() throws Exception { + assertEquals("", HttpExceptionUtils.getResponseDetail( + connectionReturning(null, null))); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java index a92cea31363c4a..371dbacb08c9de 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java @@ -130,6 +130,7 @@ import org.apache.hadoop.thirdparty.com.google.common.net.HttpHeaders; import org.apache.hadoop.util.JsonSerialization; import org.apache.hadoop.util.KMSUtil; +import org.apache.hadoop.util.HttpExceptionUtils; import org.apache.hadoop.util.Lists; import org.apache.hadoop.util.Progressable; import org.apache.hadoop.util.StringUtils; @@ -514,7 +515,12 @@ private Path makeAbsolute(Path f) { // server is demanding an authentication we don't support if (code == HttpURLConnection.HTTP_UNAUTHORIZED) { // match hdfs/rpc exception - throw new AccessControlException(conn.getResponseMessage()); + // + // The reason comes from the body, not the reason phrase: what + // AuthenticationFilter passes to sendError no longer reaches the wire as + // a phrase, and the phrase is now always "Unauthorized". + throw new AccessControlException( + HttpExceptionUtils.getResponseDetail(conn)); } if (code != op.getExpectedHttpResponseCode()) { final Map m; diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java index 820ae502d0efb6..cf3013b94daaca 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java @@ -21,14 +21,12 @@ import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; -import java.io.InputStreamReader; import java.net.HttpURLConnection; import java.net.InetSocketAddress; import java.net.URI; import java.net.URISyntaxException; import java.net.URL; import java.net.UnknownHostException; -import java.nio.charset.StandardCharsets; import java.security.DigestInputStream; import java.security.MessageDigest; import java.util.ArrayList; @@ -51,6 +49,7 @@ import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.authentication.client.AuthenticationException; +import org.apache.hadoop.util.HttpExceptionUtils; import org.apache.hadoop.util.Lists; import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.util.Time; @@ -172,7 +171,8 @@ public static MD5Hash doGetUrl(URL url, List localPaths, if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) { throw new HttpGetFailedException("Image transfer servlet at " + url + " failed with status code " + connection.getResponseCode() + - "\nResponse message:\n" + getResponseDetail(connection), + "\nResponse message:\n" + + HttpExceptionUtils.getResponseDetail(connection), connection); } @@ -194,76 +194,6 @@ public static MD5Hash doGetUrl(URL url, List localPaths, throttler); } - /** How much of a failed response body is worth quoting back. */ - private static final int MAX_RESPONSE_DETAIL_CHARS = 4096; - - /** - * Describes why a request failed, preferring the response body over the - * HTTP reason phrase. - *

- * The servlets on the other end of these transfers report their reason - * through {@link javax.servlet.http.HttpServletResponse#sendError}. That - * detail used to travel in the reason phrase, which - * {@link HttpURLConnection#getResponseMessage()} returns. Jetty 12 never - * puts a reason phrase on the wire, so the phrase is now always the - * canonical text for the status code - "Forbidden", "Gone" - and the - * detail is in the body instead. Read the body, and fall back to the - * phrase when there is none. - * - * @param connection a connection whose response status has been read - * @return a description of the failure, never null - */ - public static String getResponseDetail(HttpURLConnection connection) { - String body = ""; - try (InputStream err = connection.getErrorStream()) { - if (err != null) { - body = toPlainText(readCapped(err)); - } - } catch (IOException e) { - LOG.debug("Could not read the error response body from {}", - connection.getURL(), e); - } - if (!body.isEmpty()) { - return body; - } - try { - String phrase = connection.getResponseMessage(); - return phrase == null ? "" : phrase; - } catch (IOException e) { - return ""; - } - } - - private static String readCapped(InputStream in) throws IOException { - InputStreamReader reader = - new InputStreamReader(in, StandardCharsets.UTF_8); - StringBuilder sb = new StringBuilder(); - char[] buf = new char[1024]; - int n; - while (sb.length() < MAX_RESPONSE_DETAIL_CHARS - && (n = reader.read(buf)) != -1) { - sb.append(buf, 0, Math.min(n, MAX_RESPONSE_DETAIL_CHARS - sb.length())); - } - return sb.toString(); - } - - /** - * Reduces a response body to something readable in a log line. A container - * that renders sendError as an HTML page buries the message in markup; strip - * it out rather than quoting the page. - */ - private static String toPlainText(String body) { - String text = body; - if (text.indexOf('<') >= 0) { - text = text.replaceAll("(?s)<(script|style)\\b.*?", " ") - .replaceAll("(?s)<[^>]*>", " "); - } - text = text.replace("<", "<").replace(">", ">") - .replace(""", "\"").replace("'", "'") - .replace("&", "&"); - return text.replaceAll("\\s+", " ").trim(); - } - /** * Receives file at the url location from the input stream and puts them in * the specified destination storage location. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java index d7e13577f1fe75..247339225db782 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java @@ -39,7 +39,6 @@ import org.apache.hadoop.hdfs.server.common.HdfsServerConstants; import org.apache.hadoop.hdfs.server.common.HttpGetFailedException; import org.apache.hadoop.hdfs.server.common.Storage; -import org.apache.hadoop.hdfs.server.common.Util; import org.apache.hadoop.hdfs.web.URLConnectionFactory; import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.security.SecurityUtil; @@ -47,6 +46,7 @@ import org.apache.hadoop.security.authentication.client.AuthenticationException; import org.apache.hadoop.classification.VisibleForTesting; +import org.apache.hadoop.util.HttpExceptionUtils; import org.apache.hadoop.util.Preconditions; import org.apache.hadoop.thirdparty.com.google.common.base.Throwables; @@ -495,7 +495,8 @@ public InputStream run() throws IOException { throw new HttpGetFailedException( "Fetch of " + url + " failed with status code " + connection.getResponseCode() + - "\nResponse message:\n" + Util.getResponseDetail(connection), + "\nResponse message:\n" + + HttpExceptionUtils.getResponseDetail(connection), connection); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java index 5eb5c418b981d1..9cee56e13e81eb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java @@ -54,6 +54,7 @@ import org.apache.hadoop.io.MD5Hash; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.authentication.client.AuthenticationException; +import org.apache.hadoop.util.HttpExceptionUtils; import org.apache.hadoop.util.Lists; import org.apache.hadoop.util.Time; import org.apache.http.client.utils.URIBuilder; @@ -317,7 +318,8 @@ private static void uploadImage(URL url, Configuration conf, if (responseCode != HttpURLConnection.HTTP_OK) { throw new HttpPutFailedException(String.format( "Image uploading failed, status: %d, url: %s, message: %s", - responseCode, urlWithParams, Util.getResponseDetail(connection)), + responseCode, urlWithParams, + HttpExceptionUtils.getResponseDetail(connection)), responseCode); } } catch (AuthenticationException | URISyntaxException e) { From 385df0d53b1782e7e9876b30cc08a7b48195c173 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 14:26:54 +0000 Subject: [PATCH 23/30] HADOOP-19972. Answer a failed topology dump with a failure. NetworkTopologyServlet rendered the topology straight to the response stream. Closing that stream commits the response, so by the time the catch block ran the status line was already on the wire and sendError had nothing left to set: a dump that failed half way through went out as 200 OK with a truncated body, and one that failed before writing anything went out as 200 with no body. Rendering now happens into a buffer, and the response is not touched until the whole dump is in hand. The router's subclass had the same shape copied into it, so both now go through one sendTopology. This is not a Jetty 12 regression - a probe against Jetty 9.4.58 loses the failure identically - but it sits in the code this change is already working through, and a silent 200 on failure is worth closing. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../router/RouterNetworkTopologyServlet.java | 21 +-- .../namenode/NetworkTopologyServlet.java | 44 ++++-- .../namenode/TestNetworkTopologyServlet.java | 64 ++++++++ .../pom.xml | 146 +----------------- 4 files changed, 102 insertions(+), 173 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNetworkTopologyServlet.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNetworkTopologyServlet.java index 850f281b2bb120..c4388c3d099e01 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNetworkTopologyServlet.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNetworkTopologyServlet.java @@ -21,13 +21,11 @@ import org.apache.hadoop.hdfs.protocol.HdfsConstants; import org.apache.hadoop.hdfs.server.namenode.NetworkTopologyServlet; import org.apache.hadoop.net.Node; -import org.apache.hadoop.util.StringUtils; import javax.servlet.ServletContext; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.io.IOException; -import java.io.PrintStream; import java.util.Arrays; import java.util.List; @@ -43,13 +41,6 @@ public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException { final ServletContext context = getServletContext(); - String format = parseAcceptHeader(request); - if (FORMAT_TEXT.equals(format)) { - response.setContentType("text/plain; charset=UTF-8"); - } else if (FORMAT_JSON.equals(format)) { - response.setContentType("application/json; charset=UTF-8"); - } - Router router = RouterHttpServer.getRouterFromContext(context); DatanodeInfo[] datanodeReport = null; if (router.getRpcServer().isAsync()) { @@ -66,16 +57,6 @@ public void doGet(HttpServletRequest request, HttpServletResponse response) } List datanodeInfos = Arrays.asList(datanodeReport); - try (PrintStream out = new PrintStream( - response.getOutputStream(), false, "UTF-8")) { - printTopology(out, datanodeInfos, format); - } catch (Throwable t) { - String errMsg = "Print network topology failed. " - + StringUtils.stringifyException(t); - response.sendError(HttpServletResponse.SC_GONE, errMsg); - throw new IOException(errMsg); - } finally { - response.getOutputStream().close(); - } + sendTopology(response, datanodeInfos, parseAcceptHeader(request)); } } \ No newline at end of file diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NetworkTopologyServlet.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NetworkTopologyServlet.java index a6460280835d32..aacdf2f6bc97b0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NetworkTopologyServlet.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NetworkTopologyServlet.java @@ -31,7 +31,9 @@ import javax.servlet.ServletContext; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; +import java.io.ByteArrayOutputStream; import java.io.IOException; +import java.io.OutputStream; import java.io.PrintStream; import java.util.ArrayList; import java.util.Collections; @@ -57,28 +59,48 @@ public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException { final ServletContext context = getServletContext(); - String format = parseAcceptHeader(request); - if (FORMAT_TEXT.equals(format)) { - response.setContentType("text/plain; charset=UTF-8"); - } else if (FORMAT_JSON.equals(format)) { - response.setContentType("application/json; charset=UTF-8"); - } - NameNode nn = NameNodeHttpServer.getNameNodeFromContext(context); BlockManager bm = nn.getNamesystem().getBlockManager(); List leaves = bm.getDatanodeManager().getNetworkTopology() .getLeaves(NodeBase.ROOT); - try (PrintStream out = new PrintStream( - response.getOutputStream(), false, "UTF-8")) { + sendTopology(response, leaves, parseAcceptHeader(request)); + } + + /** + * Renders the topology and sends it, or reports why it could not be sent. + *

+ * The rendering happens before the response is touched. Writing it straight + * to the response stream commits the response as soon as that stream is + * closed, and once the status line is on the wire a later failure cannot be + * reported at all: sendError has nothing left to set, so a topology that + * failed half way through goes out as 200 OK with a truncated body. + * + * @param response the response to send on + * @param leaves the nodes to render + * @param format the response format, from {@link #parseAcceptHeader} + * @throws IOException if the topology could not be rendered or sent + */ + protected void sendTopology(HttpServletResponse response, List leaves, + String format) throws IOException { + ByteArrayOutputStream rendered = new ByteArrayOutputStream(); + try (PrintStream out = new PrintStream(rendered, false, "UTF-8")) { printTopology(out, leaves, format); } catch (Throwable t) { String errMsg = "Print network topology failed. " + StringUtils.stringifyException(t); response.sendError(HttpServletResponse.SC_GONE, errMsg); throw new IOException(errMsg); - } finally { - response.getOutputStream().close(); + } + + if (FORMAT_TEXT.equals(format)) { + response.setContentType("text/plain; charset=UTF-8"); + } else if (FORMAT_JSON.equals(format)) { + response.setContentType("application/json; charset=UTF-8"); + } + response.setContentLength(rendered.size()); + try (OutputStream out = response.getOutputStream()) { + rendered.writeTo(out); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNetworkTopologyServlet.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNetworkTopologyServlet.java index 3a9119d350dfd0..2a52f1eb869459 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNetworkTopologyServlet.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNetworkTopologyServlet.java @@ -23,19 +23,32 @@ import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.io.IOUtils; +import org.apache.hadoop.net.Node; import org.apache.hadoop.net.StaticMapping; import org.junit.jupiter.api.Test; +import org.mockito.Mockito; +import javax.servlet.ServletOutputStream; +import javax.servlet.WriteListener; +import javax.servlet.http.HttpServletResponse; import java.io.ByteArrayOutputStream; import java.io.IOException; +import java.io.PrintStream; import java.net.HttpURLConnection; import java.net.URL; import java.util.ArrayList; +import java.util.Collections; import java.util.Iterator; +import java.util.List; import java.util.Map; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.contains; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; public class TestNetworkTopologyServlet { @@ -202,4 +215,55 @@ public void testPrintTopologyNoDatanodesJsonFormat() throws IOException { assertTrue(topology.contains("No DataNodes")); } } + + /** + * A dump that fails half way through must not go out as a success. + *

+ * Rendering straight to the response stream committed the response before + * the failure was known - sendError then had nothing left to set, and a + * truncated topology was answered 200 OK. The response must stay untouched + * until the whole dump is in hand. + */ + @Test + public void testFailedDumpIsNotAnsweredAsSuccess() throws Exception { + NetworkTopologyServlet servlet = new NetworkTopologyServlet() { + @Override + protected void printTopology(PrintStream stream, List leaves, + String format) throws BadFormatException { + stream.print("half a topology"); + throw new BadFormatException("boom"); + } + }; + // a response that really accepts writes, so the old streaming shape would + // get as far as committing one + ByteArrayOutputStream written = new ByteArrayOutputStream(); + HttpServletResponse response = Mockito.mock(HttpServletResponse.class); + Mockito.when(response.getOutputStream()) + .thenReturn(new ServletOutputStream() { + @Override + public void write(int b) { + written.write(b); + } + + @Override + public boolean isReady() { + return true; + } + + @Override + public void setWriteListener(WriteListener listener) { + } + }); + + IOException thrown = assertThrows(IOException.class, () -> + servlet.sendTopology(response, Collections.emptyList(), "text")); + + assertTrue(thrown.getMessage().contains("boom"), thrown.getMessage()); + verify(response).sendError(eq(HttpServletResponse.SC_GONE), + contains("boom")); + // the half-written dump never reached the wire, so nothing was committed + verify(response, never()).getOutputStream(); + verify(response, never()).setContentType(Mockito.anyString()); + assertEquals(0, written.size(), "a half-written dump reached the wire"); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml index d766394eaf12ad..724a91431c188c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml @@ -87,30 +87,24 @@ test - org.eclipse.jetty.http2 - jetty-http2-hpack + http2-hpack ${jetty.version} org.eclipse.jetty.http2 - jetty-http2-client + http2-client ${jetty.version} org.eclipse.jetty.http2 - jetty-http2-common + http2-common ${jetty.version} org.eclipse.jetty.http2 - jetty-http2-client-transport + http2-http-client-transport ${jetty.version} @@ -195,72 +189,6 @@ org.eclipse.jetty jetty-alpn-client - - - org.eclipse.jetty - jetty-alpn-java-server - - - org.eclipse.jetty - jetty-alpn-server - - - org.eclipse.jetty - jetty-continuation - - - org.eclipse.jetty - jetty-deploy - - - org.eclipse.jetty - jetty-jmx - - - org.eclipse.jetty - jetty-rewrite - - - org.eclipse.jetty - jetty-security - - - org.eclipse.jetty - jetty-servlets - - - - org.eclipse.jetty - jetty-servlet - - - org.eclipse.jetty - jetty-webapp - - - org.eclipse.jetty.http2 - http2-server - - - javax.servlet - javax.servlet-api - commons-collections commons-collections @@ -314,72 +242,6 @@ org.eclipse.jetty jetty-alpn-client - - - org.eclipse.jetty - jetty-alpn-java-server - - - org.eclipse.jetty - jetty-alpn-server - - - org.eclipse.jetty - jetty-continuation - - - org.eclipse.jetty - jetty-deploy - - - org.eclipse.jetty - jetty-jmx - - - org.eclipse.jetty - jetty-rewrite - - - org.eclipse.jetty - jetty-security - - - org.eclipse.jetty - jetty-servlets - - - - org.eclipse.jetty - jetty-servlet - - - org.eclipse.jetty - jetty-webapp - - - org.eclipse.jetty.http2 - http2-server - - - javax.servlet - javax.servlet-api - commons-collections commons-collections From 6b90a2f35c5574f2b965d4336f5400bd23a253d4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 14:26:55 +0000 Subject: [PATCH 24/30] HADOOP-19972. Keep the app catalog on the Jetty that Solr is built against. TestAppCatalogSolrClient failed with NoClassDefFoundError on org.eclipse.jetty.client.api.Request$BeginListener. That package is Jetty 9.4; Jetty 12 moved the class to org.eclipse.jetty.client. Solr 8.11.2 is compiled against Jetty 9.4, and it needs the client even here: EmbeddedSolrServer's own constructor loads HttpShardHandlerFactory, which reaches for that class. The premise this module was migrated on - that EmbeddedSolrServer runs no container so Solr's Jetty is dead weight - does not hold, and excluding Solr's Jetty left it calling a Jetty 12 that has no such API. The two lines cannot share a classpath: they share jetty-http, jetty-io and jetty-util. So this module follows Solr rather than Hadoop. A local dependencyManagement block overrides the inherited pin for the five artifacts hadoop-project manages, the http2 dependencies go back to their 9.4 coordinates at the same version, and the exclusions added on solr-core and solr-solrj are gone. Hadoop's own Jetty 12 still arrives here through hadoop-common's ee8 artifacts, which the war never uses. That is worth a look in review: excluding it from hadoop-common alone does not remove it, because three other Hadoop dependencies carry it too. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- hadoop-project/pom.xml | 2 + .../pom.xml | 54 +++++++++++++++++-- 2 files changed, 52 insertions(+), 4 deletions(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index d44a9e8fb78e13..3e695107d550cd 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -241,6 +241,8 @@ 3.9.0 4.11.0 8.11.2 + + 9.4.44.v20210927 2.2.5.Final 1.0.2 5.4.0 diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml index 724a91431c188c..5b08946504b8ff 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml @@ -39,6 +39,45 @@ *Spec + + + + + org.eclipse.jetty + jetty-server + ${solr.jetty.version} + + + org.eclipse.jetty + jetty-http + ${solr.jetty.version} + + + org.eclipse.jetty + jetty-util + ${solr.jetty.version} + + + org.eclipse.jetty + jetty-io + ${solr.jetty.version} + + + org.eclipse.jetty + jetty-client + ${solr.jetty.version} + + + + io.swagger @@ -87,25 +126,32 @@ test + org.eclipse.jetty.http2 http2-hpack - ${jetty.version} + ${solr.jetty.version} org.eclipse.jetty.http2 http2-client - ${jetty.version} + ${solr.jetty.version} org.eclipse.jetty.http2 http2-common - ${jetty.version} + ${solr.jetty.version} org.eclipse.jetty.http2 http2-http-client-transport - ${jetty.version} + ${solr.jetty.version} From fef99339503e0d0f5fa5e2eb529b20c433f7b69a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 15:10:17 +0000 Subject: [PATCH 25/30] HADOOP-19972. Keep Hadoop's Jetty out of the app catalog war. The previous commit put this module on the Jetty 9.4 that Solr calls, but Hadoop's Jetty 12 still arrived through its ee8 artifacts, leaving two Jetty lines in the war with Jetty 12's ee8 classes sitting on top of Jetty 9.4's jetty-http, jetty-io and jetty-util. Nothing here touches them - the module is packaged as a war and served by someone else's container, so it never starts an HttpServer2 - but that holds only for as long as nothing does. Excluded from all four Hadoop dependencies that carry them. One is not enough: hadoop-common, hadoop-yarn-common, hadoop-yarn-services-core and hadoop-yarn-services-api each bring their own copy, so excluding on any one of them changes nothing. The module now resolves a single Jetty release, which is what HADOOP-19970 set out to give every module - here it is Solr's release rather than Hadoop's. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018SkhAkvYzV8Qwb3NtGZPbW --- .../pom.xml | 84 +++++++++++++++++++ 1 file changed, 84 insertions(+) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml index 5b08946504b8ff..94c26316e7c77b 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml @@ -327,6 +327,36 @@ org.apache.hadoop hadoop-common + + + + org.eclipse.jetty.ee8 + jetty-ee8-servlet + + + org.eclipse.jetty.ee8 + jetty-ee8-webapp + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-server + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-client + + @@ -337,16 +367,70 @@ org.apache.hadoop hadoop-yarn-common + + + org.eclipse.jetty.ee8 + jetty-ee8-servlet + + + org.eclipse.jetty.ee8 + jetty-ee8-webapp + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-server + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-client + + org.apache.hadoop hadoop-yarn-services-core + + + org.eclipse.jetty.ee8 + jetty-ee8-servlet + + + org.eclipse.jetty.ee8 + jetty-ee8-webapp + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-server + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-client + + org.apache.hadoop hadoop-yarn-services-api + + + org.eclipse.jetty.ee8 + jetty-ee8-servlet + + + org.eclipse.jetty.ee8 + jetty-ee8-webapp + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-server + + + org.eclipse.jetty.ee8.websocket + jetty-ee8-websocket-jetty-client + + From 3cd2192887fcecaac2810e3e0447b4bc7e9d3dcc Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 17:26:10 +0000 Subject: [PATCH 26/30] HADOOP-19972. Take Jetty from the 12.1 line rather than 12.0. Jetty has said 12.0.x stops receiving updates during 2026 in favour of 12.1.x, so landing a migration on 12.0 would arrive on a line that is already closing. Both lines carry the ee8 environment this change depends on, and every ee8 artifact named here exists at 12.1.12. No source change goes with it. The property and the LICENSE-binary entries carry the version; jetty-servlet-api stays at 4.0.9, which is what the ee8 parent pins in both lines. Co-Authored-By: Claude --- LICENSE-binary | 44 +++++++++++++++++++++--------------------- hadoop-project/pom.xml | 10 ++++++++-- 2 files changed, 30 insertions(+), 24 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 5f30fc70a08f0c..47d2bef368ae7c 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -411,29 +411,29 @@ org.apache.yetus:audience-annotations:0.5.0 org.apache.zookeeper:zookeeper:3.8.6 org.codehaus.jettison:jettison:1.5.4 org.conscrypt:conscrypt-openjdk-uber:2.5.2 -org.eclipse.jetty:jetty-alpn-client:12.0.37 -org.eclipse.jetty:jetty-client:12.0.37 -org.eclipse.jetty:jetty-ee:12.0.37 -org.eclipse.jetty:jetty-http:12.0.37 -org.eclipse.jetty:jetty-io:12.0.37 -org.eclipse.jetty:jetty-security:12.0.37 -org.eclipse.jetty:jetty-server:12.0.37 -org.eclipse.jetty:jetty-session:12.0.37 -org.eclipse.jetty:jetty-util:12.0.37 -org.eclipse.jetty:jetty-xml:12.0.37 -org.eclipse.jetty.ee8:jetty-ee8-nested:12.0.37 -org.eclipse.jetty.ee8:jetty-ee8-security:12.0.37 -org.eclipse.jetty.ee8:jetty-ee8-servlet:12.0.37 -org.eclipse.jetty.ee8:jetty-ee8-webapp:12.0.37 -org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-api:12.0.37 -org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-client:12.0.37 -org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-common:12.0.37 -org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-server:12.0.37 -org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-servlet:12.0.37 +org.eclipse.jetty:jetty-alpn-client:12.1.12 +org.eclipse.jetty:jetty-client:12.1.12 +org.eclipse.jetty:jetty-ee:12.1.12 +org.eclipse.jetty:jetty-http:12.1.12 +org.eclipse.jetty:jetty-io:12.1.12 +org.eclipse.jetty:jetty-security:12.1.12 +org.eclipse.jetty:jetty-server:12.1.12 +org.eclipse.jetty:jetty-session:12.1.12 +org.eclipse.jetty:jetty-util:12.1.12 +org.eclipse.jetty:jetty-xml:12.1.12 +org.eclipse.jetty.ee8:jetty-ee8-nested:12.1.12 +org.eclipse.jetty.ee8:jetty-ee8-security:12.1.12 +org.eclipse.jetty.ee8:jetty-ee8-servlet:12.1.12 +org.eclipse.jetty.ee8:jetty-ee8-webapp:12.1.12 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-api:12.1.12 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-client:12.1.12 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-common:12.1.12 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-jetty-server:12.1.12 +org.eclipse.jetty.ee8.websocket:jetty-ee8-websocket-servlet:12.1.12 org.eclipse.jetty.toolchain:jetty-servlet-api:4.0.9 -org.eclipse.jetty.websocket:jetty-websocket-core-client:12.0.37 -org.eclipse.jetty.websocket:jetty-websocket-core-common:12.0.37 -org.eclipse.jetty.websocket:jetty-websocket-core-server:12.0.37 +org.eclipse.jetty.websocket:jetty-websocket-core-client:12.1.12 +org.eclipse.jetty.websocket:jetty-websocket-core-common:12.1.12 +org.eclipse.jetty.websocket:jetty-websocket-core-server:12.1.12 org.ehcache:ehcache:3.8.2 org.ini4j:ini4j:0.5.4 org.objenesis:objenesis:2.6 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 3e695107d550cd..02e29bda6e2a7c 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -36,12 +36,18 @@ true true - 12.0.37 + + 12.1.12 4.0.9 _ From 98d8b3f524c941e599453a918efa05b4ef328b53 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Tue, 1 Sep 2026 05:15:43 +0000 Subject: [PATCH 27/30] HADOOP-19972. Report a JSON refusal by its reason phrase, not its envelope. Reading a refusal's reason from the body reached further than it should have. A response that carries the JSON envelope createServletExceptionResponse writes is sent with setStatus, not sendError, so its reason phrase was the canonical text for the status code before Jetty 12 and still is - nothing was lost in the upgrade for those, and there was no reason in the body to recover. Preferring the body anyway replaced a readable "Forbidden" with a line of JSON on every endpoint that refuses that way, the proxy-user check in DelegationTokenAuthenticationFilter among them. TestKMS caught it: testWebHDFSProxyUserKerb and testWebHDFSProxyUserSimple ask for a doAs the ACLs refuse and look for "Forbidden" in what comes back. Both readers now leave a JSON body alone and report the phrase. The body is still preferred everywhere else, which is the case they exist for: what sendError is given no longer reaches the wire as a phrase. Co-Authored-By: Claude --- .../client/AuthenticatedURL.java | 34 ++++++++++++++++ .../hadoop/util/HttpExceptionUtils.java | 27 ++++++++++--- .../hadoop/util/TestHttpExceptionUtils.java | 40 +++++++++++++++++++ 3 files changed, 96 insertions(+), 5 deletions(-) diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java index 288e4b737ca132..1a5ca2a2a5745c 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java @@ -80,6 +80,13 @@ public class AuthenticatedURL { /** How much of a failed response body is worth quoting back. */ private static final int MAX_RESPONSE_DETAIL_BYTES = 4096; + /** + * Content type of the JSON error envelope. Spelled out rather than taken + * from HttpExceptionUtils, which lives in hadoop-common and so is below this + * module rather than above it. + */ + private static final String APPLICATION_JSON_MIME = "application/json"; + // a lightweight cookie handler that will be attached to url connections. // client code is not required to extract or inject auth cookies. private static class AuthCookieHandler extends CookieHandler { @@ -421,8 +428,20 @@ public static void extractToken(HttpURLConnection conn, Token token) throws IOEx * phrase on the wire, so {@link HttpURLConnection#getResponseMessage()} now * only ever returns the canonical text for the status code and the reason is * in the body. Read the body, and fall back to the phrase when it is empty. + *

+ * A JSON body is left alone. That is the envelope + * HttpExceptionUtils#createServletExceptionResponse writes, and a refusal + * that carries it never carried a reason phrase of its own - it is sent with + * setStatus, not sendError, so the phrase was the canonical text for the + * status code before Jetty 12 and still is. Quoting the envelope back as + * free text would replace a readable "Forbidden" with a line of JSON, and + * the callers that want what is inside it parse it with + * HttpExceptionUtils#validateResponse instead. */ private static String responseDetail(HttpURLConnection conn) { + if (isJson(conn.getContentType())) { + return responsePhrase(conn); + } try (InputStream es = conn.getErrorStream()) { if (es != null) { byte[] body = new byte[MAX_RESPONSE_DETAIL_BYTES]; @@ -442,6 +461,21 @@ private static String responseDetail(HttpURLConnection conn) { } catch (IOException ex) { // nothing to add: fall through to the reason phrase } + return responsePhrase(conn); + } + + /** + * Whether the content type names the JSON error envelope. The header can + * carry parameters - "application/json; charset=utf-8" - so this matches a + * prefix rather than the whole value. + */ + private static boolean isJson(String contentType) { + return contentType != null + && contentType.trim().toLowerCase().startsWith(APPLICATION_JSON_MIME); + } + + /** The reason phrase, or null if it cannot be read. */ + private static String responsePhrase(HttpURLConnection conn) { try { return conn.getResponseMessage(); } catch (IOException ex) { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java index d3cf7df08fdcab..9d378c8b69b0cb 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java @@ -211,18 +211,25 @@ public static void validateResponse(HttpURLConnection conn, * For a response that carries the JSON envelope this class writes, prefer * {@link #validateResponse}, which rebuilds the original exception. This is * for everything else: a container's error page, or a plain-text reason. + * A JSON body is therefore left alone here and the phrase reported instead: + * the envelope is sent with {@code setStatus} rather than sendError, so its + * phrase was the canonical text for the status code before Jetty 12 and + * still is, and quoting the envelope back as free text would replace a + * readable "Forbidden" with a line of JSON. * * @param conn a connection whose response status has been read * @return a description of the failure, never null */ public static String getResponseDetail(HttpURLConnection conn) { String body = ""; - try (InputStream es = conn.getErrorStream()) { - if (es != null) { - body = toPlainText(readCapped(es)); + if (!isJson(conn.getContentType())) { + try (InputStream es = conn.getErrorStream()) { + if (es != null) { + body = toPlainText(readCapped(es)); + } + } catch (IOException ex) { + // nothing to add: fall through to the reason phrase } - } catch (IOException ex) { - // nothing to add: fall through to the reason phrase } if (!body.isEmpty()) { return body; @@ -235,6 +242,16 @@ public static String getResponseDetail(HttpURLConnection conn) { } } + /** + * Whether the content type names the JSON error envelope. The header can + * carry parameters - "application/json; charset=utf-8" - so this matches a + * prefix rather than the whole value. + */ + private static boolean isJson(String contentType) { + return contentType != null + && contentType.trim().toLowerCase().startsWith(APPLICATION_JSON_MIME); + } + private static String readCapped(InputStream in) throws IOException { InputStreamReader reader = new InputStreamReader(in, StandardCharsets.UTF_8); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java index fa294626e0a215..981969fa30f699 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java @@ -183,10 +183,16 @@ public void testValidateResponseJsonErrorNonException() throws Exception { private static HttpURLConnection connectionReturning(String body, String phrase) throws IOException { + return connectionReturning(body, phrase, null); + } + + private static HttpURLConnection connectionReturning(String body, + String phrase, String contentType) throws IOException { HttpURLConnection conn = mock(HttpURLConnection.class); when(conn.getErrorStream()).thenReturn(body == null ? null : new ByteArrayInputStream(body.getBytes(StandardCharsets.UTF_8))); when(conn.getResponseMessage()).thenReturn(phrase); + when(conn.getContentType()).thenReturn(contentType); return conn; } @@ -222,4 +228,38 @@ public void testResponseDetailIsNeverNull() throws Exception { assertEquals("", HttpExceptionUtils.getResponseDetail( connectionReturning(null, null))); } + + /** + * A refusal carrying the JSON envelope is reported by its reason phrase, not + * by the envelope. Such a response is sent with setStatus rather than + * sendError - see {@link HttpExceptionUtils#createServletExceptionResponse} + * - so it never had a reason of its own in the phrase, and quoting the JSON + * back would replace a readable "Forbidden" with a line of markup. Callers + * that want what is inside it use validateResponse. + */ + @Test + public void testResponseDetailLeavesTheJsonEnvelopeAlone() throws Exception { + String envelope = "{\"RemoteException\":{\"message\":\"User: client is not" + + " allowed to impersonate foo1\",\"exception\":\"AuthorizationException\"," + + "\"javaClassName\":\"org.apache.hadoop.security.authorize." + + "AuthorizationException\"}}"; + + assertEquals("Forbidden", HttpExceptionUtils.getResponseDetail( + connectionReturning(envelope, "Forbidden", "application/json"))); + // the header may carry parameters + assertEquals("Forbidden", HttpExceptionUtils.getResponseDetail( + connectionReturning(envelope, "Forbidden", + "application/json; charset=utf-8"))); + } + + /** + * A body that is not JSON is still preferred, which is the case the reader + * exists for: Jetty 12 puts what sendError was given in the body and leaves + * the phrase canonical. + */ + @Test + public void testResponseDetailStillPrefersANonJsonBody() throws Exception { + assertEquals("the real reason", HttpExceptionUtils.getResponseDetail( + connectionReturning("the real reason", "Forbidden", "text/plain"))); + } } From 048431b8ccb2eef19f6396c971a43f3ea43f7f99 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Tue, 1 Sep 2026 05:57:21 +0000 Subject: [PATCH 28/30] HADOOP-19972. Keep a webapp's error-page mappings when overriding the error handler. HttpServer2 overrides the context's error handler so that a PUT or a DELETE gets an error body rather than an empty response. It built a plain ee8.nested.ErrorHandler and replaced whatever the context had, which on ee8 is an ErrorPageErrorHandler installed by WebAppContext itself. That silently drops any and mapping a webapp declares in its web.xml, and turns the usual downstream cast of getErrorHandler() into a ClassCastException. Nothing about Jetty 12 required this: subclassing ErrorPageErrorHandler keeps the errorPageForMethod override and the capability both. Co-Authored-By: Claude --- .../src/main/java/org/apache/hadoop/http/HttpServer2.java | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java index b35472054f8daf..3ff58b1b06eb1a 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java @@ -90,8 +90,8 @@ import org.apache.hadoop.util.ReflectionUtils; import org.apache.hadoop.util.Shell; import org.apache.hadoop.util.StringUtils; -import org.eclipse.jetty.ee8.nested.ErrorHandler; import org.eclipse.jetty.ee8.nested.SessionHandler; +import org.eclipse.jetty.ee8.servlet.ErrorPageErrorHandler; import org.eclipse.jetty.ee8.servlet.FilterHolder; import org.eclipse.jetty.ee8.servlet.FilterMapping; import org.eclipse.jetty.ee8.servlet.ServletContextHandler; @@ -811,7 +811,10 @@ private void initializeWebServer(String name, String hostName, // because the detail also travelled in the reason phrase, which Jetty 12 // no longer puts on the wire, and losing both leaves a client with nothing // but the status code. - ErrorHandler errorHandler = new ErrorHandler() { + // Subclasses the handler a WebAppContext installs for itself, so that + // and mappings from a webapp's web.xml keep + // working; a plain ErrorHandler here would silently drop them. + ErrorPageErrorHandler errorHandler = new ErrorPageErrorHandler() { @Override public boolean errorPageForMethod(String method) { return true; From 8736bca88e92ce013d4e13c4d7056995e0b7ed00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Tue, 1 Sep 2026 05:57:33 +0000 Subject: [PATCH 29/30] HADOOP-19972. Read a refusal's reason from the body everywhere it is acted on. Earlier commits taught the readers that log or report a failure to look in the response body, because Jetty 12 never puts a reason phrase on the wire. Three readers that act on the reason were left behind. KMSClientProvider decides whether to reset its token and re-authenticate by matching the reason against "Invalid signature" and "Anonymous requests are disallowed". Against the canonical phrase that match can no longer succeed, so a 403 out of AuthenticationFilter - what an AuthenticationException from managementOperation produces - ended the call instead of retrying it. The new TestKMS case fails without this change and passes with it. HttpExceptionUtils.validateResponse fell back to the phrase whenever the body was not the JSON envelope, which is exactly the case where the body is the only place the reason is. Reading it back needs the error stream after the JSON parse has already failed on it, so the stream is buffered and marked first, and the reader is shielded from the close the JSON parser performs on its way out. An envelope too large to rewind still parses; only the text fallback is given up, which is what happened before. WebHdfsFileSystem reported the phrase in the two arms either side of the one fixed earlier. jsonParse now rejects an incompatible content type before it opens the stream, so a body it will not read is left intact for the caller to report rather than opened and closed unread. AuthenticatedURL's body reader kept a simpler HTML stripper than the one in HttpExceptionUtils, so an error page with inline script read differently through the two paths. The duplication is forced - hadoop-common sits above hadoop-auth - but the drift was not. Co-Authored-By: Claude --- .../client/AuthenticatedURL.java | 27 ++++++- .../crypto/key/kms/KMSClientProvider.java | 34 +++++++- .../hadoop/util/HttpExceptionUtils.java | 80 ++++++++++++++++++- .../hadoop/util/TestHttpExceptionUtils.java | 73 ++++++++++++++++- .../hadoop/crypto/key/kms/server/TestKMS.java | 76 ++++++++++++++++++ .../hadoop/hdfs/web/WebHdfsFileSystem.java | 25 +++--- 6 files changed, 292 insertions(+), 23 deletions(-) diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java index 1a5ca2a2a5745c..b6742c9e541a43 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java @@ -450,10 +450,8 @@ private static String responseDetail(HttpURLConnection conn) { && (n = es.read(body, read, body.length - read)) != -1) { read += n; } - // A container renders sendError as an HTML page; the reason is in - // there among the markup, which is no use in a one-line message. - String text = new String(body, 0, read, StandardCharsets.UTF_8) - .replaceAll("(?s)<[^>]*>", " ").replaceAll("\\s+", " ").trim(); + String text = toPlainText( + new String(body, 0, read, StandardCharsets.UTF_8)); if (!text.isEmpty()) { return text; } @@ -474,6 +472,27 @@ private static boolean isJson(String contentType) { && contentType.trim().toLowerCase().startsWith(APPLICATION_JSON_MIME); } + /** + * Reduces a response body to something readable in a one-line message. A + * container renders sendError as an HTML page, so the reason arrives buried + * in markup. + *

+ * Kept in step with {@code HttpExceptionUtils.toPlainText}, which does the + * same job for the same bodies one module up. The duplication is forced: + * hadoop-common depends on this module, not the other way round. + */ + private static String toPlainText(String body) { + String text = body; + if (text.indexOf('<') >= 0) { + text = text.replaceAll("(?s)<(script|style)\\b.*?", " ") + .replaceAll("(?s)<[^>]*>", " "); + } + text = text.replace("<", "<").replace(">", ">") + .replace(""", "\"").replace("'", "'") + .replace("&", "&"); + return text.replaceAll("\\s+", " ").trim(); + } + /** The reason phrase, or null if it cannot be read. */ private static String responsePhrase(HttpURLConnection conn) { try { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java index a3293620ab9e4b..04ec63ff3adf34 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java @@ -18,7 +18,6 @@ package org.apache.hadoop.crypto.key.kms; import org.apache.commons.codec.binary.Base64; -import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.crypto.key.KeyProvider; @@ -564,9 +563,7 @@ private T call(HttpURLConnection conn, Object jsonOutput, } if ((conn.getResponseCode() == HttpURLConnection.HTTP_FORBIDDEN - && (!StringUtils.isEmpty(conn.getResponseMessage()) - && (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED) - || conn.getResponseMessage().contains(INVALID_SIGNATURE)))) + && isAuthenticationFailure(conn)) || conn.getResponseCode() == HttpURLConnection.HTTP_UNAUTHORIZED) { // Ideally, this should happen only when there is an Authentication // failure. Unfortunately, the AuthenticationFilter returns 403 when it @@ -607,6 +604,35 @@ private T call(HttpURLConnection conn, Object jsonOutput, return ret; } + /** + * Whether a 403 is the AuthenticationFilter refusing to authenticate the + * caller, rather than the KMS refusing to authorize an authenticated one. + * Only the former is worth resetting the token and retrying for. + *

+ * The filter reports why through + * {@link javax.servlet.http.HttpServletResponse#sendError}. That detail used + * to arrive in the HTTP reason phrase; since Jetty 12 the phrase is always + * the canonical text for the status code and the detail is in the body, so + * read it from there. An authorization denial arrives instead as the JSON + * envelope {@link HttpExceptionUtils} writes, which + * {@link HttpExceptionUtils#getResponseDetail} leaves untouched - so a + * denial neither matches here nor has its body consumed before + * {@link HttpExceptionUtils#validateResponse} below rebuilds the exception + * from it. + *

+ * Matching {@link #ANONYMOUS_REQUESTS_DISALLOWED} on a substring rather than + * on the whole value is forced by the move into the body: the container + * wraps the reason in an error page. + * + * @param conn a connection whose response status has been read + * @return true when the response names an authentication failure + */ + private static boolean isAuthenticationFailure(HttpURLConnection conn) { + String detail = HttpExceptionUtils.getResponseDetail(conn); + return detail.contains(ANONYMOUS_REQUESTS_DISALLOWED) + || detail.contains(INVALID_SIGNATURE); + } + public static class KMSKeyVersion extends KeyVersion { public KMSKeyVersion(String keyName, String versionName, byte[] material) { super(keyName, versionName, material); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java index 9d378c8b69b0cb..b4e4fd6de30060 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java @@ -23,6 +23,8 @@ import javax.servlet.http.HttpServletResponse; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; +import java.io.BufferedInputStream; +import java.io.FilterInputStream; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; @@ -136,6 +138,11 @@ private static void throwException(Throwable ex) *

* NOTE: this method will throw the deserialized exception even if not * declared in the throws of the method signature. + *

+ * When the response does not carry the JSON envelope - a container error + * page, say - the detail is taken from the body via + * {@link #getResponseDetail}, because that is where a servlet's reason now + * is: Jetty 12 no longer puts one in the HTTP reason phrase. * * @param conn the HttpURLConnection. * @param expectedStatus the expected HTTP status code. @@ -149,8 +156,12 @@ public static void validateResponse(HttpURLConnection conn, Exception toThrow; InputStream es = null; try { - es = conn.getErrorStream(); - Map json = JsonSerialization.mapReader().readValue(es); + InputStream raw = conn.getErrorStream(); + if (raw != null) { + es = new BufferedInputStream(raw); + es.mark(ERROR_BODY_REWIND_LIMIT); + } + Map json = JsonSerialization.mapReader().readValue(shielded(es)); json = (Map) json.get(ERROR_JSON); String exClass = (String) json.get(ERROR_CLASSNAME_JSON); String exMsg = (String) json.get(ERROR_MESSAGE_JSON); @@ -177,7 +188,7 @@ public static void validateResponse(HttpURLConnection conn, } catch (Exception ex) { toThrow = new IOException(String.format( "HTTP status [%d], message [%s], URL [%s], exception [%s]", - conn.getResponseCode(), conn.getResponseMessage(), conn.getURL(), + conn.getResponseCode(), rewoundDetail(es, conn), conn.getURL(), ex.toString()), ex); } finally { if (es != null) { @@ -195,6 +206,17 @@ public static void validateResponse(HttpURLConnection conn, /** How much of a failed response body is worth quoting back. */ private static final int MAX_RESPONSE_DETAIL_CHARS = 4096; + /** + * How much of an error body {@link #validateResponse} keeps buffered so it + * can be rewound and read as text once the JSON parse has failed. Sized to + * hold {@link #MAX_RESPONSE_DETAIL_CHARS} characters of any UTF-8 body. A + * body longer than this still parses - the reader runs straight through it - + * it just cannot be rewound, which leaves the reason phrase as the fallback, + * as it was before. + */ + private static final int ERROR_BODY_REWIND_LIMIT = + 4 * MAX_RESPONSE_DETAIL_CHARS; + /** * Describes why a request failed, preferring the response body over the HTTP * reason phrase. @@ -234,6 +256,40 @@ public static String getResponseDetail(HttpURLConnection conn) { if (!body.isEmpty()) { return body; } + return responsePhrase(conn); + } + + /** + * Describes a failure whose body has already been read - and failed - as the + * JSON envelope. The body is the only place a servlet's reason can be now, + * so rewind and read it as text. The envelope guard {@link + * #getResponseDetail} applies does not belong here: nothing that parsed as + * the envelope reaches this point, so there is no envelope to protect. + * + * @param es the buffered error stream, marked at its start, or null + * @param conn the connection it came from + * @return a description of the failure, never null + */ + private static String rewoundDetail(InputStream es, HttpURLConnection conn) { + if (es != null) { + try { + es.reset(); + String body = toPlainText(readCapped(es)); + if (!body.isEmpty()) { + return body; + } + } catch (IOException ex) { + // read too far to rewind: fall through to the reason phrase + } + } + return responsePhrase(conn); + } + + /** + * The HTTP reason phrase, or "" when there is none. Since Jetty 12 this is + * always the canonical text for the status code. + */ + private static String responsePhrase(HttpURLConnection conn) { try { String phrase = conn.getResponseMessage(); return phrase == null ? "" : phrase; @@ -242,6 +298,24 @@ public static String getResponseDetail(HttpURLConnection conn) { } } + /** + * Hides {@link InputStream#close()} from a reader that would otherwise close + * the stream on its way out. The JSON reader closes its source even when the + * parse failed, and a closed stream can no longer be rewound and read as + * text. The caller keeps ownership and closes the real stream itself. + */ + private static InputStream shielded(InputStream in) { + if (in == null) { + return null; + } + return new FilterInputStream(in) { + @Override + public void close() { + // the caller owns the stream + } + }; + } + /** * Whether the content type names the JSON error envelope. The header can * carry parameters - "application/json; charset=utf-8" - so this matches a diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java index 981969fa30f699..d501866fc106a3 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHttpExceptionUtils.java @@ -111,12 +111,83 @@ public void testValidateResponseNonJsonErrorMessage() throws Exception { when(conn.getErrorStream()).thenReturn(is); when(conn.getResponseMessage()).thenReturn("msg"); when(conn.getResponseCode()).thenReturn(HttpURLConnection.HTTP_BAD_REQUEST); + // The body wins over the reason phrase: a servlet's reason travels in the + // body now, so "stream" is the detail and "msg" is only the canonical + // text for the status code. LambdaTestUtils.interceptAndValidateMessageContains(IOException.class, - Arrays.asList(Integer.toString(HttpURLConnection.HTTP_BAD_REQUEST), "msg", + Arrays.asList(Integer.toString(HttpURLConnection.HTTP_BAD_REQUEST), "stream", "com.fasterxml.jackson.core.JsonParseException"), () -> HttpExceptionUtils.validateResponse(conn, HttpURLConnection.HTTP_CREATED)); } + @Test + public void testValidateResponseHtmlErrorPageReportsTheReason() + throws Exception { + // What AuthenticationFilter's sendError looks like on the wire. + String page = "Error 403 Invalid signature" + + "" + + "

HTTP ERROR 403

Reason: Invalid signature

" + + ""; + HttpURLConnection conn = connectionReturning(page, "Forbidden", "text/html"); + when(conn.getResponseCode()).thenReturn(HttpURLConnection.HTTP_FORBIDDEN); + LambdaTestUtils.interceptAndValidateMessageContains(IOException.class, + Arrays.asList("Invalid signature"), + () -> HttpExceptionUtils.validateResponse(conn, HttpURLConnection.HTTP_OK)); + } + + @Test + public void testValidateResponseFallsBackToThePhraseWithNoBody() + throws Exception { + HttpURLConnection conn = connectionReturning(null, "Forbidden", "text/html"); + when(conn.getResponseCode()).thenReturn(HttpURLConnection.HTTP_FORBIDDEN); + LambdaTestUtils.interceptAndValidateMessageContains(IOException.class, + Arrays.asList("Forbidden"), + () -> HttpExceptionUtils.validateResponse(conn, HttpURLConnection.HTTP_OK)); + } + + @Test + public void testValidateResponseStillRebuildsTheEnvelopeException() + throws Exception { + // The rewind must not disturb the envelope path: a JSON body still + // reconstructs its exception rather than being quoted back as text. + Map json = new HashMap(); + json.put(HttpExceptionUtils.ERROR_EXCEPTION_JSON, + IllegalStateException.class.getSimpleName()); + json.put(HttpExceptionUtils.ERROR_CLASSNAME_JSON, + IllegalStateException.class.getName()); + json.put(HttpExceptionUtils.ERROR_MESSAGE_JSON, "EX"); + Map response = new HashMap(); + response.put(HttpExceptionUtils.ERROR_JSON, json); + String body = new ObjectMapper().writeValueAsString(response); + HttpURLConnection conn = + connectionReturning(body, "Forbidden", "application/json"); + when(conn.getResponseCode()).thenReturn(HttpURLConnection.HTTP_FORBIDDEN); + LambdaTestUtils.intercept(IllegalStateException.class, "EX", + () -> HttpExceptionUtils.validateResponse(conn, HttpURLConnection.HTTP_OK)); + } + + @Test + public void testValidateResponseParsesAnEnvelopeTooLargeToRewind() + throws Exception { + // Larger than the rewind buffer: the parser reads straight through, so the + // exception is still rebuilt - only the text fallback is given up. + Map json = new HashMap(); + json.put(HttpExceptionUtils.ERROR_EXCEPTION_JSON, + IllegalStateException.class.getSimpleName()); + json.put(HttpExceptionUtils.ERROR_CLASSNAME_JSON, + IllegalStateException.class.getName()); + json.put(HttpExceptionUtils.ERROR_MESSAGE_JSON, + "x".repeat(64 * 1024)); + Map response = new HashMap(); + response.put(HttpExceptionUtils.ERROR_JSON, json); + String body = new ObjectMapper().writeValueAsString(response); + HttpURLConnection conn = + connectionReturning(body, "Forbidden", "application/json"); + when(conn.getResponseCode()).thenReturn(HttpURLConnection.HTTP_FORBIDDEN); + LambdaTestUtils.intercept(IllegalStateException.class, + () -> HttpExceptionUtils.validateResponse(conn, HttpURLConnection.HTTP_OK)); + } + @Test public void testValidateResponseJsonErrorKnownException() throws Exception { Map json = new HashMap(); diff --git a/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java b/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java index 84bcbbde4fbed6..89da53b005d888 100644 --- a/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java +++ b/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java @@ -33,6 +33,10 @@ import org.apache.hadoop.crypto.key.kms.KMSDelegationToken; import org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider; import org.apache.hadoop.crypto.key.kms.ValueQueue; +import org.apache.hadoop.security.authentication.client.AuthenticationException; +import org.apache.hadoop.security.authentication.server.AuthenticationToken; +import org.apache.hadoop.security.authentication.util.SignerException; +import org.apache.hadoop.security.token.delegation.web.PseudoDelegationTokenAuthenticationHandler; import org.apache.hadoop.fs.CommonConfigurationKeysPublic; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.Text; @@ -61,6 +65,8 @@ import javax.net.ssl.HttpsURLConnection; import javax.security.auth.login.AppConfigurationEntry; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; import java.io.ByteArrayInputStream; import java.io.DataInputStream; @@ -91,6 +97,8 @@ import java.util.UUID; import java.util.concurrent.Callable; import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -1433,6 +1441,74 @@ public Void run() throws Exception { }); } + /** + * Refuses one request, once armed, with the reason an invalid signature + * produces, so that a test can drive the branch of KMSClientProvider's retry + * that reads it. + *

+ * The refusal is thrown from managementOperation rather than from + * authenticate because authenticate is reached only by a request that + * carries no valid token, and such a request is answered 401, not 403. An + * AuthenticationException out of managementOperation is what makes + * AuthenticationFilter answer 403 - with the reason in the body only. + */ + public static class FailWhenArmedAuthenticationHandler + extends PseudoDelegationTokenAuthenticationHandler { + + private static final AtomicBoolean ARMED = new AtomicBoolean(); + private static final AtomicInteger REFUSALS = new AtomicInteger(); + + @Override + public boolean managementOperation(AuthenticationToken token, + HttpServletRequest request, HttpServletResponse response) + throws IOException, AuthenticationException { + if (ARMED.compareAndSet(true, false)) { + REFUSALS.incrementAndGet(); + throw new AuthenticationException( + new SignerException("Invalid signature")); + } + return super.managementOperation(token, request, response); + } + } + + @Test + public void testKMSAuthFailureRetryOn403() throws Exception { + // The reason for a refusal reaches the client in the response body, not in + // the HTTP reason phrase: Jetty 12 never puts a phrase on the wire. A + // client that reads only the phrase sees "Forbidden", fails to recognise + // an authentication failure, and gives up without re-authenticating. + Configuration conf = new Configuration(); + File confDir = getTestDir(); + conf = createBaseKMSConf(confDir, conf); + conf.set("hadoop.kms.authentication.type", + FailWhenArmedAuthenticationHandler.class.getName()); + conf.set(KeyAuthorizationKeyProvider.KEY_ACL + "k1.ALL", "*"); + conf.set(KeyAuthorizationKeyProvider.KEY_ACL + "k2.ALL", "*"); + writeConf(confDir, conf); + FailWhenArmedAuthenticationHandler.ARMED.set(false); + FailWhenArmedAuthenticationHandler.REFUSALS.set(0); + + runServer(null, null, confDir, new KMSCallable() { + @Override + public Void call() throws Exception { + final Configuration conf = new Configuration(); + conf.setInt(KeyProvider.DEFAULT_BITLENGTH_NAME, 128); + final URI uri = createKMSUri(getKMSUrl()); + KeyProvider kp = createProvider(uri, conf); + // Establish a token first, so the refusal below is the only thing + // standing between the client and a successful call. + kp.createKey("k1", new KeyProvider.Options(conf)); + + FailWhenArmedAuthenticationHandler.ARMED.set(true); + // Succeeds only if the 403 was recognised and the call retried. + kp.createKey("k2", new KeyProvider.Options(conf)); + assertEquals(1, FailWhenArmedAuthenticationHandler.REFUSALS.get(), + "the armed request should have been refused exactly once"); + return null; + } + }); + } + @Test @SuppressWarnings("checkstyle:methodlength") public void testACLs() throws Exception { diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java index 371dbacb08c9de..e15a01145d49bb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java @@ -486,6 +486,18 @@ private Path makeAbsolute(Path f) { if (c.getContentLength() == 0) { return null; } + // Checked before the stream is opened: a body this method will not read + // is left intact for the caller to report, and getInputStream is not + // called for a response that has no input stream to give. + final String contentType = c.getContentType(); + if (contentType != null) { + final MediaType parsed = MediaType.valueOf(contentType); + if (!MediaType.APPLICATION_JSON_TYPE.isCompatible(parsed)) { + throw new IOException("Content-Type \"" + contentType + + "\" is incompatible with \"" + MediaType.APPLICATION_JSON + + "\" (parsed=\"" + parsed + "\")"); + } + } final InputStream in = useErrorStream ? c.getErrorStream() : c.getInputStream(); if (in == null) { @@ -493,15 +505,6 @@ private Path makeAbsolute(Path f) { " stream is null."); } try { - final String contentType = c.getContentType(); - if (contentType != null) { - final MediaType parsed = MediaType.valueOf(contentType); - if (!MediaType.APPLICATION_JSON_TYPE.isCompatible(parsed)) { - throw new IOException("Content-Type \"" + contentType - + "\" is incompatible with \"" + MediaType.APPLICATION_JSON - + "\" (parsed=\"" + parsed + "\")"); - } - } return JsonSerialization.mapReader().readValue(in); } finally { in.close(); @@ -529,13 +532,13 @@ private Path makeAbsolute(Path f) { } catch(Exception e) { throw new IOException("Unexpected HTTP response: code=" + code + " != " + op.getExpectedHttpResponseCode() + ", " + op.toQueryString() - + ", message=" + conn.getResponseMessage(), e); + + ", message=" + HttpExceptionUtils.getResponseDetail(conn), e); } if (m == null) { throw new IOException("Unexpected HTTP response: code=" + code + " != " + op.getExpectedHttpResponseCode() + ", " + op.toQueryString() - + ", message=" + conn.getResponseMessage()); + + ", message=" + HttpExceptionUtils.getResponseDetail(conn)); } else if (m.get(RemoteException.class.getSimpleName()) == null) { return m; } From f8e54b9b434397271731a3f37748ad6422f1914d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jose=20Luis=20L=C3=B3pez?= Date: Sat, 5 Sep 2026 01:00:26 +0200 Subject: [PATCH 30/30] HADOOP-19972. Keep jetty-util in the shaded client runtime. ITUseMiniCluster fails against the shaded client with NoClassDefFoundError: org/apache/hadoop/shaded/org/eclipse/jetty/util/Attributes from NameNode.startHttpServer: no client jar ships org.eclipse.jetty.util any more. hadoop-client-runtime is the jar that ships it, which is what hadoop-client-minicluster still says when it excludes jetty-util from hadoop-minicluster, hadoop-yarn-server-nodemanager and hadoop-yarn-server-web-proxy. YARN-11793 took jetty-util out of the runtime jar because the Jersey test framework had just moved onto Jetty and started pulling it into hadoop-client-minicluster on a path that module does not exclude, so the classes were in both jars and BanDuplicateClasses failed. HADOOP-19970 moved that test container back off Jetty, onto the JDK HTTP provider, so nothing pulls jetty-util into the minicluster jar any longer and the two exclusions together leave the package in neither. Stop excluding jetty-util from hadoop-client-runtime. Verified with the shadedclient invariants and integration tests: BanDuplicateClasses passes on both check modules, ITUseMiniCluster passes, and org/eclipse/jetty/util appears in hadoop-client-runtime (417 entries) and in neither hadoop-client-minicluster nor hadoop-client-api. Co-Authored-By: Claude Opus 5 --- hadoop-client-modules/hadoop-client-runtime/pom.xml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/hadoop-client-modules/hadoop-client-runtime/pom.xml b/hadoop-client-modules/hadoop-client-runtime/pom.xml index 3b9e6a7d133d47..4414d3c4d95184 100644 --- a/hadoop-client-modules/hadoop-client-runtime/pom.xml +++ b/hadoop-client-modules/hadoop-client-runtime/pom.xml @@ -174,8 +174,17 @@ which are all code that shipped inside those two jars on 9.4. jetty-util-ajax and jetty-continuation are gone from the tree, so they are no longer named here. + + jetty-util is not named here either. It is this jar's to + ship, as hadoop-client-minicluster still says when it + excludes it. YARN-11793 took it away when the Jersey test + framework moved onto Jetty and started pulling jetty-util + into hadoop-client-minicluster on a path that module does + not exclude; HADOOP-19970 moved that test container back + off Jetty, so nothing pulls it there any more and the two + exclusions would leave org.eclipse.jetty.util in neither + jar. --> - org.eclipse.jetty:jetty-util org.eclipse.jetty:jetty-server org.eclipse.jetty:jetty-session org.eclipse.jetty.ee8:jetty-ee8-servlet