HADOOP-15019. Honor HADOOP_USER_CLASSPATH_FIRST for deduped entries. - #8670
Open
deepujain wants to merge 2 commits into
Open
HADOOP-15019. Honor HADOOP_USER_CLASSPATH_FIRST for deduped entries.#8670deepujain wants to merge 2 commits into
deepujain wants to merge 2 commits into
Conversation
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
All checks are green on the current head. @stoty, this is a small |
deepujain
force-pushed
the
HADOOP-15019-user-classpath-first-dedupe
branch
from
September 1, 2026 16:04
fb10cf8 to
75bc926
Compare
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
Hi @slfan1989 @Hexiaoqiao @pan3793, this HADOOP-15019 classpath fix has been sitting open since August. I keep rebasing it on trunk and Yetus is +1 on the latest head. Any chance one of you could take a look or point me at the right reviewer? |
|
🎊 +1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Summary
When
HADOOP_USER_CLASSPATH_FIRST=true, a user-supplied classpath entry thatis already present in Hadoop's computed classpath can stay in its older
position instead of moving to the front. This happens because duplicate
classpath entries were treated as a no-op even when they were being re-added
with
before.This patch keeps the existing de-duplication behavior, but when a duplicate
entry is re-added with
before, it is moved to the front ofCLASSPATH.That preserves the expected override behavior for cases such as
HADOOP_CONF_DIRalso being present inHADOOP_CLASSPATH.Change
hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh:update
hadoop_add_classpath()so duplicate entries are reordered to thefront when added with
before.hadoop-common-project/hadoop-common/src/test/scripts/hadoop_add_classpath.bats:add a regression that proves a duplicate entry can be moved to the front.
hadoop-common-project/hadoop-common/src/test/scripts/hadoop_finalize_classpath.bats:add a higher-level regression that matches the JIRA scenario where
HADOOP_CONF_DIRis duplicated inHADOOP_CLASSPATHwhileHADOOP_USER_CLASSPATH_FIRST=true.Evidence it works
bash -n hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.shgit diff --check/bin/bash -lc 'set -eo pipefail; export HADOOP_SHELL_SCRIPT_DEBUG=true; export QATESTMODE=true; export HADOOP_LIBEXEC_DIR=$(cd -P hadoop-common-project/hadoop-common/src/main/bin >/dev/null && pwd -P); source hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh; tmpdir=$(mktemp -d); mkdir -p "$tmpdir/first" "$tmpdir/second"; CLASSPATH=""; hadoop_add_classpath "$tmpdir/first" after; hadoop_add_classpath "$tmpdir/second" after; hadoop_add_classpath "$tmpdir/second" before; test "$CLASSPATH" = "$tmpdir/second:$tmpdir/first"; rm -rf "$tmpdir"'/bin/bash -lc 'set -eo pipefail; export HADOOP_SHELL_SCRIPT_DEBUG=true; export QATESTMODE=true; export HADOOP_LIBEXEC_DIR=$(cd -P hadoop-common-project/hadoop-common/src/main/bin >/dev/null && pwd -P); source hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh; tmpdir=$(mktemp -d); mkdir -p "$tmpdir/new" "$tmpdir/old"; CLASSPATH=""; HADOOP_CONF_DIR="$tmpdir"; HADOOP_CLASSPATH="$tmpdir:$tmpdir/new:$tmpdir/old"; HADOOP_USER_CLASSPATH_FIRST=true; HADOOP_USE_CLIENT_CLASSLOADER=""; hadoop_translate_cygwin_path() { true; }; hadoop_finalize_classpath; test "$CLASSPATH" = "$tmpdir:$tmpdir/new:$tmpdir/old"; rm -rf "$tmpdir"'batsis not installed in this local environment, so the new.batsregressions were validated through the equivalent sourced-function shell
invocations above.
JIRA
Fixes HADOOP-15019
For code changes:
(e.g. 'HADOOP-17799. Your PR title ...')?
declared according to the connector-specific documentation? Not
applicable to this shell-script change.
in a way that is compatible for inclusion under
ASF 2.0?
No new dependencies are added.
LICENSE,LICENSE-binary,NOTICE-binaryfiles? No license or notice changes are required.AI Tooling
Contains content generated by Codex.
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html