WIP: Analytics: Support for Hierarchical Clustering for Variables (R package)

[hidekoji]

Description

R-side companion to exploratory-io/tam#38161 ("Analytics: Support for Hierarchical Clustering
for Variables") — adds exp_hclust_variables(), implementing hierarchical clustering of the
SELECTED VARIABLES (columns) of a data frame, based on correlation, distinct from Row Clustering
(tam#38157, which clusters observations/rows and is not yet implemented — see the paired
tam PR's design doc, docs/plans/design/38161_variable_clustering_design.md, for the full
reuse-plan discussion and why this ships as an independent function rather than reusing shared
Row Clustering infrastructure that doesn't exist yet).

New file: R/hclust_variables.R.

Pipeline: selected columns → variable × variable correlation matrix (reuses the existing
do_cor.cols()/do_cor_internal() type-aware Pearson/Spearman/Kendall/Polychoric/Mixed/Auto
correlation engine, so column-type gating stays consistent with do_cor()/Factor Analysis rather
than being re-decided here) → distance = 1 - correlation → fastcluster::hclust() (falls back
to stats::hclust() via requireNamespace() when the fastcluster package — newly added to
DESCRIPTION Imports: — isn't installed yet; a real deployment additionally needs a
requiredRPackagesReduced.json bump + package rebuild in the tam repo, an ops step out of scope
for this PR) → cutree() → a dependency-free right-angle dendrogram segment table built directly
from hc$merge/hc$height/hc$order (hclust_dendrogram_segments()), matching the branch
geometry shape described in tam#38157's requirements doc so a future shared Row/Variable
dendrogram renderer can consume it without a redesign.

Key decisions

• Distance is 1 - correlation, not 1 - |correlation| — a strong negative correlation is a
  real, distinct relationship from a strong positive one and should not be treated as "close" the
  same way.
• Ward's linkage is deliberately not offered/accepted — linkage_method only allows
  complete/average/single, and the function stop()s if ward.D2/ward is passed — Ward's
  method assumes an underlying Euclidean distance, and 1 - correlation is not Euclidean in
  general.
• Zero-variance / all-NA columns are detected and excluded with a clear message (a constant
  column's own correlation self-diagonal is NA, dropped by do_cor.cols()'s existing
  na.rm = TRUE), rather than letting an NA distance reach hclust() uncaught.
• Cluster IDs are renumbered by left-to-right dendrogram display order (not cutree()'s
  internal numbering), per tam#38157 requirements doc section 29.
• tidy.hclust_variables_exploratory() follows the existing tidy_rowwise(model, type=...) S3
  convention with 5 branches: summary/analysis_conditions, cluster_membership,
  correlation_matrix, dendrogram_segments, dendrogram_leaves.

NAMESPACE was hand-edited (one export() + one S3method() entry) since roxygen2 could not
be run in this environment — see the Checklist caveats below.

Checklist

• Add test cases for this fix/enhancement — tests/testthat/test_hclust_variables.R: basic
  structure, 1 - correlation distance semantics, zero-variance-column exclusion, the
  fewer-than-2-usable-variables error, the ward.D2 rejection, n_clusters bounds, all 3
  linkage methods, all 4+ correlation methods, pairwise-complete missing-value handling, the
  n = 2 edge case, max_nrow sampling, the canonical stress-test column name (spaces,
  multibyte characters, symbols — per this org's R-feature convention), and a standalone unit
  test of hclust_dendrogram_segments()'s geometry.
• Pass devtools::check() — not run, no R interpreter available in this sandbox (see below).
• Pass devtools::test() — not run, same reason. test_hclust_variables.R was written and
  carefully hand-traced against documented R semantics (cor() on a constant vector,
  hclust()'s merge/height/order contract, mat_to_df()'s na.rm behavior read
  directly from R/util.R) but never executed.
• Test installing from github — not attempted (no R environment).
• Tested with Exploratory — not attempted (no desktop/RServe environment in this sandbox).

Reason for the unchecked items: this PR was authored in a sandboxed environment with no R
or Rscript binary on PATH or found anywhere on the filesystem (which/command -v both came
back empty, plus a broad filesystem search). This is flagged explicitly rather than silently
skipped — please run devtools::check() / devtools::test(filter="hclust_variables") on a
machine with R + this package's dependencies before merging, and treat this PR as ready for
design/code review, not for merge, until that happens.

🤖 Generated with Claude Code

---

Generated by Claude Code