Skip to content

jit: skip expensive XLA graph passes when no clusters are compiled - #3

Draft
fenxcc with Copilot wants to merge 2 commits into
masterfrom
copilot/investigate-xla-performance-issues
Draft

jit: skip expensive XLA graph passes when no clusters are compiled#3
fenxcc with Copilot wants to merge 2 commits into
masterfrom
copilot/investigate-xla-performance-issues

Conversation

Copilot AI commented Mar 31, 2026

Copy link
Copy Markdown

Enabling XLA on CPU causes 2–5% throughput regression even when zero subgraphs are compiled (and >5% with small subgraphs), due to unconditional overhead in graph optimization passes that runs regardless of whether any ops are actually marked for XLA compilation.

Changes

EncapsulateSubgraphsPass: early exit when no XLA clusters exist

The pass previously performed expensive setup work unconditionally:

  • CPU Device creation via DeviceFactory
  • ProcessFunctionLibraryRuntime construction
  • Full graph copy inside EncapsulateSubgraphsInFunctions
  • O(n) ref-variable analysis and per-node attribute writes

Now the pass checks for kXlaClusterAttr during the existing TPU-skip scan and returns early if no nodes were marked for compilation:

bool has_xla_clusters = false;
for (Node* n : (*options.graph)->nodes()) {
  if (n->type_string() == "TPUExecute" || ...) return absl::OkStatus();
  if (n->attrs().Find(kXlaClusterAttr)) has_xla_clusters = true;
}
if (!has_xla_clusters) return absl::OkStatus();  // skip all expensive setup

MarkForCompilation: O(1) early-exit on repeat invocations

After MarkForCompilationPassImpl::Run() completes, the source node (ID 0, always first in graph->nodes() iteration) is marked with kXlaAlreadyClustered. Previously, for graphs where no clusters were created, the existing early-exit scan found nothing and forced a full O(n) re-analysis on every subsequent invocation (e.g. from PartitionedCall re-running the optimization pipeline). The source-node marker converts that to O(1).

Test updates (encapsulate_subgraphs_pass_test.cc)

  • RefVariablesMarked: updated to assert kXlaHasReferenceVarsAttr is not set when no clusters exist (attribute is only meaningful on XlaLaunch/_XlaCompile nodes)
  • NoRefVarsNoAttr: adds explicit kXlaClusterAttr so the full encapsulation + ref-var analysis path is exercised
  • Added NoXlaClustersEarlyExit: directly validates the early-exit optimization

…sary overhead when no XLA clusters exist

- EncapsulateSubgraphsPass: Skip expensive CPU device/FLR creation and graph
  copy when no nodes have kXlaClusterAttr (i.e., no XLA clusters were created by
  MarkForCompilationPass). This eliminates significant per-graph-construction
  overhead when XLA is enabled but no ops are actually compiled.

- MarkForCompilation: After running the analysis, mark the source node with
  kXlaAlreadyClustered so that subsequent calls on the same graph (e.g., from
  PartitionedCall re-running the optimization pipeline) return early in O(1)
  time instead of scanning all O(n) nodes without finding an early exit.

- Updated tests to reflect new behavior: kXlaHasReferenceVarsAttr is only set
  when XLA clusters exist (not on all nodes unconditionally). Added
  NoXlaClustersEarlyExit test to validate the optimization.

Agent-Logs-Url: https://github.com/fenxcc/tensorflow/sessions/24af514f-b99e-4428-a563-f4d9c64241ca

Co-authored-by: fenxcc <64964597+fenxcc@users.noreply.github.com>
Copilot AI changed the title [WIP] Investigate performance degradation with XLA enabled on armv9.2-a jit: skip expensive XLA graph passes when no clusters are compiled Mar 31, 2026
Copilot AI requested a review from fenxcc March 31, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants