Dipowell/node readiness timing

[Diamond (diamondpowell)]

Summary

Adds node_readiness_time as a separate metric in the open-source CRUD module to match internal repo behavior. The internal repo captures how long K8s nodes take to become Ready independently from when the ARM API completes. The open-source repo was missing this - it only had combined duration.

Azure API says "done" when the control plane finishes, but nodes might not be schedulable yet. Capturing both timestamps separately enables regression analysis:

• command_execution_time > node_readiness_time -> ARM layer is the bottleneck
• node_readiness_time > command_execution_time -> K8s layer is the bottleneck

Changes

utils/provisioning_instrumentation.py (new)

• Extracted instrument_nodepool_provisioning() - runs ARM and K8s readiness concurrently via ThreadPoolExecutor, captures separate timing metadata
• Extracted begin_create_or_update_with_retry() - ARM operation with retry on OperationNotAllowed/EtagMismatch, returns retry_occurred boolean

clients/aks_client.py

• Thin _instrument_nodepool_provisioning() wrapper that builds callables and delegates to the extracted module
• All three call sites (create, scale, progressive scale) now use retry consistently
• Removed use_retry parameter
• Net reduction: 1020 -> 958 lines (resolves pylint too-many-lines)

clients/kubernetes_client.py

• Updated wait_for_nodes_ready() docstring to reflect actual return type and parameter names

tests/utils/test_provisioning_instrumentation.py (new)

• 8 unit tests covering both extracted functions: timing metadata, retry_occurred flag, error propagation, timeout, retryable vs non-retryable errors

tests/clients/test_aks_client.py

• Added test_create_node_pool_retry_occurred_metadata for end-to-end retry verification
• Updated time mocks to patch utils.provisioning_instrumentation.time for timing assertions

Timing metadata stored via op.add_metadata():

• node_readiness_time: seconds from start until K8s nodes were Ready
• command_execution_time: seconds from start until ARM operation completed
• retry_occurred: boolean, True if ARM operation hit a transient error and retried

Implementation notes

• Uses ThreadPoolExecutor(max_workers=2) to run ARM polling and K8s readiness checks concurrently
• Both tasks run to completion even if one fails, enabling partial diagnostics
• retry_occurred metadata enables filtering skewed command_execution_time values from regression analysis
• Extraction keeps aks_client.py under 1,000 lines without disabling pylint

[Copilot]

Pull request overview

This PR adds independent node-readiness timing to the Python AKS CRUD flow by extending the Kubernetes wait helper to optionally return a readiness timestamp, then running the ARM poller and the K8s readiness wait concurrently so both timings can be recorded for regression analysis.

Changes:

• Extend KubernetesClient.wait_for_nodes_ready() with return_timestamp to optionally return (ready_nodes, ready_timestamp).
• Add concurrent ARM + K8s readiness execution in AKSClient and record node_readiness_time / command_execution_time metadata.
• Update AKS and Kubernetes client unit tests to cover the new return shape and timing metadata recording.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
modules/python/clients/kubernetes_client.py	Adds return_timestamp option so callers can capture when nodes became Ready.
modules/python/clients/aks_client.py	Runs ARM and readiness concurrently and stores separate timing metadata.
modules/python/tests/clients/test_kubernetes_client.py	Adds a unit test validating the timestamp-returning behavior.
modules/python/tests/clients/test_aks_client.py	Updates tests to expect timestamp return and to assert timing metadata is recorded.

[Diamond (diamondpowell)]

Pylint too-many-lines (C0302) is failing because aks_client.py was at 999 lines on main after #1233. This PR adds 60 net lines for the instrumentation method. Both eks_client.py and kubernetes_client.py already disable this rule for the same reason. Something to think about for future additions to this file.

[github-actions]

For reviewers only: reply /run-tf-integration to trigger the terraform integration pipeline before approving the PR.