Skip to content

feat(ui-test): Playwright UI E2E suite — helpers + feature flows (stage 2)#2264

Open
cjlg-soloio wants to merge 31 commits into
kagent-dev:mainfrom
cjlg-soloio:cjlg-soloio/playwrightTestsStage2
Open

feat(ui-test): Playwright UI E2E suite — helpers + feature flows (stage 2)#2264
cjlg-soloio wants to merge 31 commits into
kagent-dev:mainfrom
cjlg-soloio:cjlg-soloio/playwrightTestsStage2

Conversation

@cjlg-soloio

@cjlg-soloio cjlg-soloio commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Description

Browser-level end-to-end tests for the kagent UI, built on the merged Playwright foundation (#2262). This suite covers the one gap nothing else does: multi-step user flows across components, driven against a mocked backend.

Because the UI fetches data server-side (Next.js server actions), browser page.route can't intercept /api/**. Instead the suite runs a standalone stub backend (playwright/mocks/server.mjs) and points Next at it via BACKEND_INTERNAL_URL; both servers are booted automatically by Playwright's webServer.

Test infrastructure

  • Scenario-override engine in the stub backend — POST /__mock/scenario sets a per-endpoint { status, body } override; POST /__mock/reset clears them. Default stays the happy path.
  • Typed control seam (mocks/control.ts) surfaced as a mock fixture (fixtures/test.ts): mock.noAgents(), mock.agentsError(), etc. The fixture resets the stub before every test, so scenarios never leak between tests.
  • Driver helperspage.ts (loadPage, expectNoErrors, expectToast), nav.ts (dropdown-aware gotoView / gotoCreate), select.ts (combobox driver), and a2a.ts (chat/stream SSE fixtures).
  • Typed spec-side builders (mocks/data.ts) mirror the stub's payload shapes against the app's own @/types, so drift from the real API surface fails at compile time.

Feature flows

One spec per core UI area, each split into a success journey and a failure journey (except the app shell — one consolidated smoke + navigation test). Each journey asserts both UI state and the payload POSTed/PUT to the backend, walking happy, empty, error, and validation states.

Spec Flow
tests/app-shell.spec.ts Agents list states (populated / empty / error) + header navigation
tests/agents/agents.spec.ts Agent create (declarative + harness) → delete; validation / error toasts / delete-cancel
tests/chat/chat-session.spec.ts Chat send / streamed reply / tool call / feedback / reload history; stream error + session-not-found
tests/models/models.spec.ts Model config create / edit / delete; create validation + delete error
tests/mcp/mcp-server.spec.ts Remote + stdio MCP server create / filter / delete; URL validation + create error
tests/prompts/prompt-libraries.spec.ts Prompt library create / view / edit / delete; empty-name validation
tests/onboarding/onboarding.spec.ts Onboarding wizard completion (existing model + create-model path) + skip

A slowMo launch delay (250ms, overridable via E2E_SLOW_MO_MS) keeps the retained-on-failure videos followable in real time.

Product code changes are limited to additive data-testid hooks across 8 components (chat input/send, tool-call card, combobox/dialog anchors, loading overlay) where role/text selectors were ambiguous — no runtime behavior changed.

Demo videos

Each feature area recorded as a success and a failure journey (with slowMo so the flows are followable). Grouped below by area.

Playwright flow videos

App shell — agents list states (populated / empty / error) + header navigation

app-shell-list-states-and-nav.webm

Agents — success — create (declarative + harness) → delete

agents-success-create-declarative-harness-delete.webm

Agents — failure — validation / error toasts / delete-cancel

agents-failure-validation-toasts-delete-cancel.webm

Chat — success — session send / streamed reply / feedback / reload history

chat-success-session-feedback-reload-history.webm

Chat — failure — stream error + session-not-found

chat-failure-error-and-session-not-found.webm

Models — success — create / edit / delete a config

models-success-create-edit-delete.webm

Models — failure — validation + delete error

models-failure-validation-and-delete-error.webm

MCP — success — remote + stdio server create / filter / delete

mcp-success-remote-stdio-filter-delete.webm

MCP — failure — URL validation + create error

mcp-failure-url-validation-create-error.webm

Prompts — success — create / view / edit / delete

prompts-success-create-view-edit-delete.webm

Prompts — failure — blocks create when name is empty

prompts-failure-blocks-empty-name.webm

Onboarding — wizard create path

onboarding-wizard-create-path.webm

Onboarding — end with an existing model

onboarding-end-with-existing-model.webm

Onboarding — skips the wizard

onboarding-skips-wizard.webm

Testing

cd ui
npm install
npx playwright install chromium   # first time only
npm run test:pw

Playwright boots the stub backend and dev server automatically and drives each flow through happy, empty, error, and validation states. Runs serially (workers: 1) against the shared stub. Full suite: 14 tests passing locally.

Change type

test — adds the Playwright browser E2E suite. Product changes limited to additive data-testid hooks; no runtime behavior changed.

Changelog

NONE

cjlg-soloio and others added 25 commits July 13, 2026 15:12
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: cjlg-soloio <carlos.logrono@solo.io>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
- Force single Playwright worker unconditionally (shared stub + Next server)
- Default req.method/req.url before parsing in the stub server
- Fail fast on stub reset failure in CI; stay non-fatal locally
- Drop unused pull-requests:write permission from the workflow

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Pairs fullyParallel with workers: 1 so both stay off until per-test data
isolation lands, preventing tests from silently racing against the shared
stub backend if the worker count is raised.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
- Pin STUB_PORT on the stub webServer so a shell-set STUB_PORT can't misbind it
- Stop reusing the dev server so BACKEND_INTERNAL_URL always applies (no silent stub bypass)
- Type ToolServerListEntry as the real RemoteMCPServerResponse for compile-time drift detection
- Drop no-op **/*.mjs from the playwright tsconfig include

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
…ght video

Resolve Charlesthebird's review on PR kagent-dev#2180:
- Playwright workflow now reads node-version-file: ui/.nvmrc instead of a
  hardcoded major version.
- Remove Cypress entirely: delete cypress.config.ts and cypress/e2e/smoke.cy.ts,
  drop the cypress and now-unused start-server-and-test devDeps plus the
  test:e2e/test:e2e:cypress scripts, and remove the dead CYPRESS_INSTALL_BINARY
  env from the ui-playwright, ci, and ui-chromatic workflows and the Dockerfile.
- Enable Playwright video recording (video: "on"); artifacts already upload via
  the existing test-results step.
- De-Cypress the Playwright README (Jest, Vitest, Storybook untouched).

data.ts is intentionally kept — it is consumed by the stacked stage-2 PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Restore a conventional test:e2e entrypoint pointing at the Playwright
suite, per review feedback on kagent-dev#2180.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
- rename fixture callback param to `provide` (drops the rules-of-hooks eslint-disable)
- record video only on failure to match screenshot/trace
- pin @playwright/test and playwright to aligned 1.61.1

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Builds on the Stage 0 foundation to unlock non-happy-path testing and give
specs a small, proven driver surface:

- Scenario-override engine in the stub backend: POST /__mock/scenario sets a
  per-endpoint { status, body } override; /__mock/reset clears them. Default
  stays the happy path.
- Typed control seam (mocks/control.ts) surfaced as a `mock` fixture
  (mock.noAgents(), mock.agentsError(), ...); the fixture resets the stub
  before every test so scenarios never leak between tests.
- page.ts (loadPage, expectNoErrors, expectToast) and nav.ts (dropdown-aware
  gotoView/gotoCreate) driver helpers.
- home spec (happy / empty / error) proving the engine, and a nav spec proving
  dropdown routing.

Form/select/dialog helpers are deferred to Stage 2 (built demand-driven with the
create-agent flow).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
First Stage 2 feature spec — the declarative create-agent flow:
- happy path: fill name/description, pick the model, submit -> asserts the
  redirect to /agents and the captured POST /api/agents payload
- validation: empty required fields block submit (no request sent)
- failure: forced create error surfaces an error toast, no redirect

Groundwork introduced here (demand-driven, reused by later sub-stages):
- stub captures mutation (POST/PUT/DELETE) bodies and exposes them via
  GET /__mock/requests; supports per-mutation error overrides; reset clears both
- control.ts: setMutation, agentsCreateError, capturedRequests, lastRequest
- helpers/select.ts: Radix <Select> driver

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Second Stage 2 feature (commit 2/8) — the chat streaming flow:
- happy path: send a message, mock the A2A SSE reply -> asserts the user
  message, the streamed agent text, and a rendered tool-call block
- empty state: a new chat shows "Start a conversation"
- failure: an aborted stream surfaces an error toast, no agent reply

Groundwork introduced here:
- helpers/a2a.ts: mockAgentReply / mockAgentStreamError — page.route mock of the
  browser-side A2A SSE call (text/event-stream JSON-RPC frames)
- stub: dynamic GET routes (agent detail, sessions-for-agent) + POST /api/sessions
  returning a session with an id (needed for the streamed contextId)
- data-testid hooks: chat-input, chat-send (ChatInterface), tool-call (ToolDisplay)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Third Stage 2 feature (commit 3/8) — model config lifecycle:
- create: pick provider (OpenAI) + model + API key -> asserts redirect to
  /models, success toast, and the captured POST /api/modelconfigs payload
- edit-save: open the seeded config via the edit-model hook -> Save Changes ->
  asserts redirect + captured PUT /api/modelconfigs/<ref>

Groundwork:
- stub: /api/models keyed "OpenAI" (was lowercase); stock/configured provider
  endpoints (/api/modelproviderconfigs/*); GET /api/modelconfigs/<ns>/<name> detail
- data-testid hooks: model-provider-select, model-select, model-api-key-input

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Fourth Stage 2 feature (commit 4/8) — create a RemoteMCPServer tool server:
- happy: fill server name + URL, submit -> asserts redirect to /mcp and the
  captured POST /api/toolservers payload (type, url, name)
- failure: forced 500 shows the inline "Couldn't create server" alert, no redirect

Groundwork: stub GET /api/toolservertypes (blocking gate on /mcp/new).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Fifth Stage 2 feature (commit 5/8) — the harness create variant at
/agents/new-harness:
- happy: name + model, submit -> asserts redirect to /agents and the captured
  POST /api/agentharnesses (kind AgentHarness, backend openclaw)
- validation: empty required fields block submit (no request)

Groundwork: stub POST /api/agentharnesses returns data.agent (required by
createAgentHarnessFromForm).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Sixth Stage 2 feature (commit 6/8) — delete an agent from the list:
- happy: Agent options menu -> Delete -> confirm -> asserts the dialog closes
  and DELETE /api/agents/<ns>/<name> was captured
- cancel: dismissing the dialog issues no DELETE

No stub change (the generic mutation capture covers DELETE).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Seventh Stage 2 feature (commit 7/8) — prompt libraries:
- empty state: /prompts (namespace kagent) with no libraries
- create: name + a fragment key/content, submit -> asserts redirect to the
  detail page, success toast, and the captured POST /api/prompttemplates payload

Groundwork: stub GET /api/prompttemplates (list) + GET /api/prompttemplates/<ns>/<name> (detail).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Eighth Stage 2 feature (commit 8/8) — complete the first-run onboarding wizard:
walks Welcome -> Model (select existing) -> Agent Setup -> Tools -> Review ->
Finish, asserting the POST /api/agents at finalize, that the wizard dismisses to
the Agents list, and that localStorage['kagent-onboarding'] flips to "true".

No new stub routes (all endpoints already seeded).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
cjlg-soloio and others added 3 commits July 16, 2026 12:19
Finish the Stage 2 consolidation: each feature area is now one success
journey and one failure journey (~14 videos total) instead of a journey
plus scattered edge tests.

- Merge every negative/edge case per area into a single failure journey
  with a test.step() per case; each step calls mock.reset() first so
  captured requests and scenario overrides don't leak across steps.
- Fold empty states into the success journeys (chat, prompt libraries).
- Force Playwright's dev server onto webpack (--webpack). Turbopack's dev
  server corrupts the RSC client manifest when a route recompiles after a
  full navigation, breaking the second cold load of "/" (onboarding
  variants, app-shell error state). Local `npm run dev` stays on Turbopack.
- Update the README conventions + roadmap to the success/failure model.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
…tTestsStage2

# Conflicts:
#	ui/package-lock.json
cjlg-soloio and others added 2 commits July 17, 2026 11:57
…tTestsStage2

# Conflicts:
#	ui/playwright.config.ts
#	ui/playwright/README.md
#	ui/playwright/fixtures/test.ts
#	ui/playwright/mocks/data.ts
#	ui/playwright/mocks/server.mjs
Recorded videos play at real time; a 250ms per-action delay (overridable
via E2E_SLOW_MO_MS) makes the failure-run videos followable. Mirrors the
enterprise Playwright config.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jul 17, 2026
@cjlg-soloio
cjlg-soloio marked this pull request as ready for review July 17, 2026 16:36
@cjlg-soloio
cjlg-soloio requested a review from peterj as a code owner July 17, 2026 16:36
Copilot AI review requested due to automatic review settings July 17, 2026 16:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the Stage 2 Playwright UI E2E suite for kagent’s Next.js UI, extending the existing Playwright foundation by introducing reusable helpers, a scenario-controlled stub backend, and multi-step “success + failure” feature-flow specs that validate both UI state and backend payloads.

Changes:

  • Adds Playwright feature-flow specs (agents, chat session streaming, models, MCP servers, prompt libraries, onboarding) plus consolidated app-shell journey.
  • Extends the stub backend with scenario overrides (/__mock/scenario), reset, and mutation request capture for payload assertions.
  • Adds targeted data-testid hooks in UI components to stabilize selectors where role/text is ambiguous.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
ui/src/components/ToolDisplay.tsx Adds a stable data-testid hook for tool-call blocks used in chat flow assertions.
ui/src/components/ProviderCombobox.tsx Adds a data-testid to stabilize provider selection in model-related E2E flows.
ui/src/components/models/new/AuthSection.tsx Adds a data-testid to stabilize API key input selection in model E2E flows.
ui/src/components/ModelCombobox.tsx Adds a data-testid to stabilize model selection in model E2E flows.
ui/src/components/LoadingState.tsx Adds a data-testid for the loading overlay to support “app ready” gating in tests.
ui/src/components/create/SelectToolsDialog.tsx Adds per-tool-row data-testid hooks to drive tool selection reliably in agent creation flows.
ui/src/components/create/ContextSection.tsx Adds a data-testid for context compaction switch used in agent creation flow.
ui/src/components/chat/ChatInterface.tsx Adds data-testid hooks for chat input and send button used in chat flow tests.
ui/playwright/tests/smoke.spec.ts Removes the stage-0 smoke test after folding coverage into the app-shell journey.
ui/playwright/tests/prompts/prompt-libraries.spec.ts Adds prompt library success/failure journeys with payload capture assertions.
ui/playwright/tests/onboarding/onboarding.spec.ts Adds onboarding wizard completion, create-path, and skip-path flows.
ui/playwright/tests/models/models.spec.ts Adds model config lifecycle + validation/error coverage with payload assertions.
ui/playwright/tests/mcp/mcp-server.spec.ts Adds MCP server lifecycle flows (remote + stdio) plus validation/error coverage.
ui/playwright/tests/chat/chat-session.spec.ts Adds chat session streaming flow (SSE via page.route) and failure paths.
ui/playwright/tests/app-shell.spec.ts Adds consolidated agents list state coverage + header navigation journey.
ui/playwright/tests/agents/agents.spec.ts Adds agent declarative + harness creation flows plus delete and failure coverage.
ui/playwright/README.md Updates documentation to reflect Stage 2 suite structure, conventions, and roadmap status.
ui/playwright/mocks/server.mjs Adds scenario override engine, request capture API, and richer stub coverage for Stage 2 flows.
ui/playwright/mocks/data.ts Adds typed ok() envelope and updates typed builders for stub parity.
ui/playwright/mocks/control.ts Adds a typed semantic control seam (mock.*) over the stub’s scenario/reset/capture endpoints.
ui/playwright/helpers/select.ts Adds a Radix Select driver helper for consistent option selection.
ui/playwright/helpers/page.ts Adds page helpers for navigation, loading-overlay gating, error absence, and toast assertions.
ui/playwright/helpers/nav.ts Adds dropdown-menu navigation helpers for the persistent header.
ui/playwright/helpers/a2a.ts Adds SSE stream mocking helpers for chat streaming flows.
ui/playwright/fixtures/test.ts Extends the shared fixture with mock and ensures stub reset + onboarding bypass per test.
ui/playwright.config.ts Adds slowMo configuration and forces webpack dev server for test stability.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread ui/playwright/mocks/control.ts Outdated
Comment thread ui/playwright/helpers/nav.ts
Comment thread ui/playwright.config.ts
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jul 17, 2026
- Complete the EndpointSlug union in control.ts to cover every GET slug the
  stub serves (providers/configuredProviders/toolservertypes/prompttemplates),
  so the control seam needs no type casts.
- Wait for the LoadingState overlay to clear after navigation in nav.ts,
  preventing flaky follow-up menu clicks landing on the overlay.
- Coerce + validate E2E_SLOW_MO_MS, falling back to the default on a
  non-finite/negative value instead of passing NaN to Playwright's slowMo.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>

@Charlesthebird Charlesthebird left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are looking good! I have a couple notes - testing this out locally, I think it might be more useful to have all but the chat using real data. We can sync next week about a plan for that.

@Charlesthebird

Copy link
Copy Markdown
Collaborator

Updated with a real backend now in cjlg-soloio#5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants