Fix index misalignment in char_wb ngram padding

[adityaanikam]

Fixes #8416

Root cause

get_char_ngrams() pads each token with self.delimiter on both sides before generating character ngrams:

padding = Series(self.delimiter).repeat(len(tokens))

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d6b4779b-19d0-486d-ad39-04458c74c1c7

📥 Commits

Reviewing files that changed from the base of the PR and between ed13db7 and 1394360.

📒 Files selected for processing (2)

• python/cuml/cuml/feature_extraction/_vectorizers.py
• python/cuml/tests/test_text_feature_extraction.py

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Added get_feature_names_out() for retrieving vectorizer feature names in a NumPy array.
  • Updated text vectorizer behavior to align with the current feature-name API.

• Bug Fixes

  • Fixed character n-gram extraction to preserve correct positional alignment when constructing padded tokens.
  • Improved consistency of generated character features for inputs with non-sequential indexes.
  • Ensured character-boundary n-grams match expected feature names and results across supported n-gram lengths.

• Deprecations

  • Existing get_feature_names() calls now issue a future deprecation warning.

Walkthrough

CountVectorizer now supports get_feature_names_out() and preserves deprecated API behavior through a warning mixin. Character n-gram padding resets indexes before concatenation. Tests cover current feature-name access and char_wb TF-IDF output against scikit-learn.

Changes

Vectorizer API and character n-gram behavior

Layer / File(s)	Summary
Feature-name API compatibility python/cuml/cuml/feature_extraction/_vectorizers.py, python/cuml/tests/test_text_feature_extraction.py	CountVectorizer now provides get_feature_names_out() and inherits deprecated API support. Tests use NumPy array comparisons and verify FutureWarning behavior.
Character n-gram index alignment python/cuml/cuml/feature_extraction/_vectorizers.py	get_char_ngrams resets token and padding indexes before both concatenation operations.
Regression validation python/cuml/tests/test_text_feature_extraction.py	Tests compare char_wb TF-IDF matrices and feature names with scikit-learn for n-gram lengths 2 through 6.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 13943

This localized fix adjusts character n-gram padding alignment and includes focused test changes; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: jcrist

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the primary fix for char_wb n-gram index misalignment.
Description check	✅ Passed	The description explains the linked bug and identifies the character n-gram padding root cause.
Linked Issues check	✅ Passed	The code resets padding indexes before concatenation and updates tests, addressing issue #8416 requirements.
Out of Scope Changes check	✅ Passed	The API and test updates support the vectorizer fix and current feature-name API without unrelated changes.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

python/cuml/cuml/feature_extraction/_vectorizers.py (1)

216-224: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add an enabled char_wb Tfidf regression test.

In python/cuml/tests/test_text_feature_extraction.py, test TfidfVectorizer.fit_transform(analyzer="char_wb", ngram_range=(2, 6)). Compare the sparse output and feature names with scikit-learn using get_feature_names_out().

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/cuml/cuml/feature_extraction/_vectorizers.py` around lines 216 - 224,
Add an enabled regression test in test_text_feature_extraction.py covering
TfidfVectorizer with analyzer="char_wb" and ngram_range=(2, 6). Fit and
transform the same input with cuML and scikit-learn, then compare sparse outputs
and feature names using get_feature_names_out().

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@python/cuml/cuml/feature_extraction/_vectorizers.py`:
- Around line 216-224: Add an enabled regression test in
test_text_feature_extraction.py covering TfidfVectorizer with analyzer="char_wb"
and ngram_range=(2, 6). Fit and transform the same input with cuML and
scikit-learn, then compare sparse outputs and feature names using
get_feature_names_out().

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 80ca255e-7d3d-41f0-ab51-594df4f04c33

📥 Commits

Reviewing files that changed from the base of the PR and between 74f5ffa and 3edfa37.

📒 Files selected for processing (1)

• python/cuml/cuml/feature_extraction/_vectorizers.py

[csadorf]

@divyegala Can you review this, please?

[divyegala]

Thank you for the contribution! Could you please add a test?

[adityaanikam]

Added test_tfidf_vectorizer_char_wb_ngrams — compares cuML's TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 6)) against scikit-learn's on the existing DOCS fixture (13 documents, mixed lengths, includes empty docs), checking both the output matrix and feature names via get_feature_names_out(). That corpus mix is what actually exercises the bug, since the misalignment only showed up once documents had varying token counts. Also rebased onto latest main to pick up the get_feature_names_out() API change from #8480.
This also needs bug + non-breaking labels for the Label Checker I don't have permission to add them myself.

[divyegala]

Thank you, LGTM!

[divyegala]

/ok to test 36f9500

[divyegala]

/merge