Skip to content

feat(provider/lambdai): add Kubernetes workload identity auth - #440

Open
alaski-lambda wants to merge 3 commits into
NVIDIA:mainfrom
alaski-lambda:feat/lambdai-workload-identity-upstream
Open

feat(provider/lambdai): add Kubernetes workload identity auth#440
alaski-lambda wants to merge 3 commits into
NVIDIA:mainfrom
alaski-lambda:feat/lambdai-workload-identity-upstream

Conversation

@alaski-lambda

@alaski-lambda alaski-lambda commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Authenticate the lambdai provider with Kubernetes workload identity instead of a long-lived API token Secret. When lambda-pod-identity-webhook injects LAMBDA_ROLE_LRN and LAMBDA_WORKLOAD_IDENTITY_TOKEN_FILE into the pod, topograph exchanges the projected ServiceAccount token at POST /api/v1/oidc/token for a short-lived Lambda API key, caches it process-wide, and refreshes it before expiry (single-flight, jittered, and tolerant of a transient exchange failure while the cached key is valid).

Static-token mode is unchanged when LAMBDA_ROLE_LRN is absent.

Description

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.
  • All commits are signed off per DCO (git commit -s).

@copy-pr-bot

copy-pr-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@greptile-apps

greptile-apps Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds Kubernetes workload-identity authentication to the Lambda AI provider.

  • Exchanges projected ServiceAccount tokens for short-lived Lambda API keys with process-wide caching and refresh.
  • Preserves explicit static-token precedence and rejects malformed or ambiguous credentials.
  • Retries workload-identity requests once after invalidating the specific key rejected with 401.
  • Adds Helm configuration guidance, provider documentation, and regression coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported credential-selection, alias-handling, and stale-401 invalidation issues are addressed by the current implementation and targeted regression tests.

Important Files Changed

Filename Overview
pkg/providers/lambdai/provider.go Selects explicit credentials ahead of ambient identity, validates malformed and ambiguous credential maps, and retries rejected workload-identity keys safely.
pkg/providers/lambdai/token.go Implements bounded process-wide credential caching, single-flight token exchange, refresh backoff, expiry handling, and compare-and-invalidate behavior.
pkg/providers/lambdai/provider_test.go Adds regression and integration coverage for credential precedence, malformed aliases, workload-identity exchange, 401 retry, and stale-response concurrency.
pkg/providers/lambdai/token_test.go Covers token caching, refresh, exchange failures, expiry parsing, invalidation, and cache bounds.
docs/providers/lambdai.md Documents workload-identity setup, credential precedence, runtime behavior, and operational caveats.
charts/topograph/values.k8s.lambdai-workload-identity-example.yaml Provides an example Helm configuration using an annotated ServiceAccount without a credentials Secret.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Loader as Lambda Provider Loader
    participant Cache as Workload Identity Cache
    participant OIDC as Lambda OIDC Endpoint
    participant API as Lambda Topology API
    Client->>Loader: Generate topology request
    alt Explicit token supplied
        Loader->>API: Topology request with static token
    else Pod workload identity
        Loader->>Cache: Request API key
        alt No usable cached key
            Cache->>OIDC: Exchange projected ServiceAccount token
            OIDC-->>Cache: Short-lived API key
        end
        Cache-->>Loader: Cached or refreshed API key
        Loader->>API: Topology request with API key
        alt API returns 401
            Loader->>Cache: Invalidate rejected key if current
            Cache->>OIDC: Exchange token
            OIDC-->>Cache: Replacement API key
            Loader->>API: Retry once with replacement key
        end
    end
    API-->>Client: Topology response
Loading

Reviews (10): Last reviewed commit: "fix(provider/lambdai): harden workload-i..." | Re-trigger Greptile

Comment thread pkg/providers/lambdai/provider.go Outdated
Comment thread pkg/providers/lambdai/provider.go Outdated
@alaski-lambda
alaski-lambda force-pushed the feat/lambdai-workload-identity-upstream branch 2 times, most recently from 19048f2 to 2c6a9e0 Compare July 31, 2026 17:09
Comment thread pkg/providers/lambdai/provider.go Outdated
@alaski-lambda
alaski-lambda force-pushed the feat/lambdai-workload-identity-upstream branch 2 times, most recently from b70716a to 0923802 Compare July 31, 2026 17:58
Comment thread pkg/providers/lambdai/provider.go Outdated
@alaski-lambda
alaski-lambda force-pushed the feat/lambdai-workload-identity-upstream branch from 0923802 to 51d93fb Compare July 31, 2026 18:15
Authenticate the lambdai provider with Kubernetes workload identity instead
of a long-lived API token Secret. When lambda-pod-identity-webhook injects
LAMBDA_ROLE_LRN and LAMBDA_WORKLOAD_IDENTITY_TOKEN_FILE into the pod,
topograph exchanges the projected ServiceAccount token at
POST /api/v1/oidc/token for a short-lived Lambda API key, caches it
process-wide, and refreshes it before expiry (single-flight, jittered, and
tolerant of a transient exchange failure while the cached key is valid).

Static-token mode is unchanged when LAMBDA_ROLE_LRN is absent.

Signed-off-by: Andrew Laski <alaski@lambdal.com>
@alaski-lambda
alaski-lambda force-pushed the feat/lambdai-workload-identity-upstream branch from 51d93fb to d0cbe1c Compare August 3, 2026 19:43
@dmitsh

dmitsh commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

/ok-to-test 36c6a9a

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Comment thread pkg/providers/lambdai/provider.go
Comment thread pkg/providers/lambdai/token.go
Comment thread pkg/providers/lambdai/token.go
Comment thread pkg/providers/lambdai/token.go Outdated
Addresses review feedback on the workload-identity implementation.

Reject duplicate case-insensitive spellings of a credential key.
mapstructure matches keys case-insensitively, so "token" and "Token" both
feed the same field and Go's randomized map iteration chose the winner --
the same request could authenticate as a different principal, or against a
different workspace, from one run to the next. The ambiguity is reported
rather than resolved arbitrarily.

Share a failed token refresh with queued callers. The graceful-degradation
path returned the still-valid cached key but left refreshAt untouched, so
every caller queued behind refreshMu observed the same refresh-due state and
ran its own retry cycle against a failing endpoint. The next attempt is now
deferred, never past the hard expiry, and republished as a copy so values
already handed out stay immutable.

Bound the process-level credential cache. baseURL is a per-request provider
parameter, so the cache key space is caller-controlled; entries are evicted
in insertion order, which costs at most one extra exchange.

Move the test-only resetCredentialCache helper into token_test.go.

Signed-off-by: Andrew Laski <alaski@lambdal.com>
@alaski-lambda
alaski-lambda force-pushed the feat/lambdai-workload-identity-upstream branch from b315a66 to 490717e Compare August 4, 2026 13:38
@greptile-apps

greptile-apps Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Want your agent to iterate on Greptile's feedback? Try greploops.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants