feat(ui-test): Playwright UI E2E suite — helpers + feature flows (stage 2)#2264
feat(ui-test): Playwright UI E2E suite — helpers + feature flows (stage 2)#2264cjlg-soloio wants to merge 31 commits into
Conversation
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: cjlg-soloio <carlos.logrono@solo.io> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
- Force single Playwright worker unconditionally (shared stub + Next server) - Default req.method/req.url before parsing in the stub server - Fail fast on stub reset failure in CI; stay non-fatal locally - Drop unused pull-requests:write permission from the workflow Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Pairs fullyParallel with workers: 1 so both stay off until per-test data isolation lands, preventing tests from silently racing against the shared stub backend if the worker count is raised. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
- Pin STUB_PORT on the stub webServer so a shell-set STUB_PORT can't misbind it - Stop reusing the dev server so BACKEND_INTERNAL_URL always applies (no silent stub bypass) - Type ToolServerListEntry as the real RemoteMCPServerResponse for compile-time drift detection - Drop no-op **/*.mjs from the playwright tsconfig include Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
…ght video Resolve Charlesthebird's review on PR kagent-dev#2180: - Playwright workflow now reads node-version-file: ui/.nvmrc instead of a hardcoded major version. - Remove Cypress entirely: delete cypress.config.ts and cypress/e2e/smoke.cy.ts, drop the cypress and now-unused start-server-and-test devDeps plus the test:e2e/test:e2e:cypress scripts, and remove the dead CYPRESS_INSTALL_BINARY env from the ui-playwright, ci, and ui-chromatic workflows and the Dockerfile. - Enable Playwright video recording (video: "on"); artifacts already upload via the existing test-results step. - De-Cypress the Playwright README (Jest, Vitest, Storybook untouched). data.ts is intentionally kept — it is consumed by the stacked stage-2 PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Restore a conventional test:e2e entrypoint pointing at the Playwright suite, per review feedback on kagent-dev#2180. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
- rename fixture callback param to `provide` (drops the rules-of-hooks eslint-disable) - record video only on failure to match screenshot/trace - pin @playwright/test and playwright to aligned 1.61.1 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Builds on the Stage 0 foundation to unlock non-happy-path testing and give
specs a small, proven driver surface:
- Scenario-override engine in the stub backend: POST /__mock/scenario sets a
per-endpoint { status, body } override; /__mock/reset clears them. Default
stays the happy path.
- Typed control seam (mocks/control.ts) surfaced as a `mock` fixture
(mock.noAgents(), mock.agentsError(), ...); the fixture resets the stub
before every test so scenarios never leak between tests.
- page.ts (loadPage, expectNoErrors, expectToast) and nav.ts (dropdown-aware
gotoView/gotoCreate) driver helpers.
- home spec (happy / empty / error) proving the engine, and a nav spec proving
dropdown routing.
Form/select/dialog helpers are deferred to Stage 2 (built demand-driven with the
create-agent flow).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
First Stage 2 feature spec — the declarative create-agent flow: - happy path: fill name/description, pick the model, submit -> asserts the redirect to /agents and the captured POST /api/agents payload - validation: empty required fields block submit (no request sent) - failure: forced create error surfaces an error toast, no redirect Groundwork introduced here (demand-driven, reused by later sub-stages): - stub captures mutation (POST/PUT/DELETE) bodies and exposes them via GET /__mock/requests; supports per-mutation error overrides; reset clears both - control.ts: setMutation, agentsCreateError, capturedRequests, lastRequest - helpers/select.ts: Radix <Select> driver Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Second Stage 2 feature (commit 2/8) — the chat streaming flow: - happy path: send a message, mock the A2A SSE reply -> asserts the user message, the streamed agent text, and a rendered tool-call block - empty state: a new chat shows "Start a conversation" - failure: an aborted stream surfaces an error toast, no agent reply Groundwork introduced here: - helpers/a2a.ts: mockAgentReply / mockAgentStreamError — page.route mock of the browser-side A2A SSE call (text/event-stream JSON-RPC frames) - stub: dynamic GET routes (agent detail, sessions-for-agent) + POST /api/sessions returning a session with an id (needed for the streamed contextId) - data-testid hooks: chat-input, chat-send (ChatInterface), tool-call (ToolDisplay) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Third Stage 2 feature (commit 3/8) — model config lifecycle: - create: pick provider (OpenAI) + model + API key -> asserts redirect to /models, success toast, and the captured POST /api/modelconfigs payload - edit-save: open the seeded config via the edit-model hook -> Save Changes -> asserts redirect + captured PUT /api/modelconfigs/<ref> Groundwork: - stub: /api/models keyed "OpenAI" (was lowercase); stock/configured provider endpoints (/api/modelproviderconfigs/*); GET /api/modelconfigs/<ns>/<name> detail - data-testid hooks: model-provider-select, model-select, model-api-key-input Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Fourth Stage 2 feature (commit 4/8) — create a RemoteMCPServer tool server: - happy: fill server name + URL, submit -> asserts redirect to /mcp and the captured POST /api/toolservers payload (type, url, name) - failure: forced 500 shows the inline "Couldn't create server" alert, no redirect Groundwork: stub GET /api/toolservertypes (blocking gate on /mcp/new). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Fifth Stage 2 feature (commit 5/8) — the harness create variant at /agents/new-harness: - happy: name + model, submit -> asserts redirect to /agents and the captured POST /api/agentharnesses (kind AgentHarness, backend openclaw) - validation: empty required fields block submit (no request) Groundwork: stub POST /api/agentharnesses returns data.agent (required by createAgentHarnessFromForm). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Sixth Stage 2 feature (commit 6/8) — delete an agent from the list: - happy: Agent options menu -> Delete -> confirm -> asserts the dialog closes and DELETE /api/agents/<ns>/<name> was captured - cancel: dismissing the dialog issues no DELETE No stub change (the generic mutation capture covers DELETE). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Seventh Stage 2 feature (commit 7/8) — prompt libraries: - empty state: /prompts (namespace kagent) with no libraries - create: name + a fragment key/content, submit -> asserts redirect to the detail page, success toast, and the captured POST /api/prompttemplates payload Groundwork: stub GET /api/prompttemplates (list) + GET /api/prompttemplates/<ns>/<name> (detail). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Eighth Stage 2 feature (commit 8/8) — complete the first-run onboarding wizard: walks Welcome -> Model (select existing) -> Agent Setup -> Tools -> Review -> Finish, asserting the POST /api/agents at finalize, that the wizard dismisses to the Agents list, and that localStorage['kagent-onboarding'] flips to "true". No new stub routes (all endpoints already seeded). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Finish the Stage 2 consolidation: each feature area is now one success journey and one failure journey (~14 videos total) instead of a journey plus scattered edge tests. - Merge every negative/edge case per area into a single failure journey with a test.step() per case; each step calls mock.reset() first so captured requests and scenario overrides don't leak across steps. - Fold empty states into the success journeys (chat, prompt libraries). - Force Playwright's dev server onto webpack (--webpack). Turbopack's dev server corrupts the RSC client manifest when a route recompiles after a full navigation, breaking the second cold load of "/" (onboarding variants, app-shell error state). Local `npm run dev` stays on Turbopack. - Update the README conventions + roadmap to the success/failure model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
…tTestsStage2 # Conflicts: # ui/package-lock.json
…tTestsStage2 # Conflicts: # ui/playwright.config.ts # ui/playwright/README.md # ui/playwright/fixtures/test.ts # ui/playwright/mocks/data.ts # ui/playwright/mocks/server.mjs
Recorded videos play at real time; a 250ms per-action delay (overridable via E2E_SLOW_MO_MS) makes the failure-run videos followable. Mirrors the enterprise Playwright config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
There was a problem hiding this comment.
Pull request overview
Adds the Stage 2 Playwright UI E2E suite for kagent’s Next.js UI, extending the existing Playwright foundation by introducing reusable helpers, a scenario-controlled stub backend, and multi-step “success + failure” feature-flow specs that validate both UI state and backend payloads.
Changes:
- Adds Playwright feature-flow specs (agents, chat session streaming, models, MCP servers, prompt libraries, onboarding) plus consolidated app-shell journey.
- Extends the stub backend with scenario overrides (
/__mock/scenario), reset, and mutation request capture for payload assertions. - Adds targeted
data-testidhooks in UI components to stabilize selectors where role/text is ambiguous.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| ui/src/components/ToolDisplay.tsx | Adds a stable data-testid hook for tool-call blocks used in chat flow assertions. |
| ui/src/components/ProviderCombobox.tsx | Adds a data-testid to stabilize provider selection in model-related E2E flows. |
| ui/src/components/models/new/AuthSection.tsx | Adds a data-testid to stabilize API key input selection in model E2E flows. |
| ui/src/components/ModelCombobox.tsx | Adds a data-testid to stabilize model selection in model E2E flows. |
| ui/src/components/LoadingState.tsx | Adds a data-testid for the loading overlay to support “app ready” gating in tests. |
| ui/src/components/create/SelectToolsDialog.tsx | Adds per-tool-row data-testid hooks to drive tool selection reliably in agent creation flows. |
| ui/src/components/create/ContextSection.tsx | Adds a data-testid for context compaction switch used in agent creation flow. |
| ui/src/components/chat/ChatInterface.tsx | Adds data-testid hooks for chat input and send button used in chat flow tests. |
| ui/playwright/tests/smoke.spec.ts | Removes the stage-0 smoke test after folding coverage into the app-shell journey. |
| ui/playwright/tests/prompts/prompt-libraries.spec.ts | Adds prompt library success/failure journeys with payload capture assertions. |
| ui/playwright/tests/onboarding/onboarding.spec.ts | Adds onboarding wizard completion, create-path, and skip-path flows. |
| ui/playwright/tests/models/models.spec.ts | Adds model config lifecycle + validation/error coverage with payload assertions. |
| ui/playwright/tests/mcp/mcp-server.spec.ts | Adds MCP server lifecycle flows (remote + stdio) plus validation/error coverage. |
| ui/playwright/tests/chat/chat-session.spec.ts | Adds chat session streaming flow (SSE via page.route) and failure paths. |
| ui/playwright/tests/app-shell.spec.ts | Adds consolidated agents list state coverage + header navigation journey. |
| ui/playwright/tests/agents/agents.spec.ts | Adds agent declarative + harness creation flows plus delete and failure coverage. |
| ui/playwright/README.md | Updates documentation to reflect Stage 2 suite structure, conventions, and roadmap status. |
| ui/playwright/mocks/server.mjs | Adds scenario override engine, request capture API, and richer stub coverage for Stage 2 flows. |
| ui/playwright/mocks/data.ts | Adds typed ok() envelope and updates typed builders for stub parity. |
| ui/playwright/mocks/control.ts | Adds a typed semantic control seam (mock.*) over the stub’s scenario/reset/capture endpoints. |
| ui/playwright/helpers/select.ts | Adds a Radix Select driver helper for consistent option selection. |
| ui/playwright/helpers/page.ts | Adds page helpers for navigation, loading-overlay gating, error absence, and toast assertions. |
| ui/playwright/helpers/nav.ts | Adds dropdown-menu navigation helpers for the persistent header. |
| ui/playwright/helpers/a2a.ts | Adds SSE stream mocking helpers for chat streaming flows. |
| ui/playwright/fixtures/test.ts | Extends the shared fixture with mock and ensures stub reset + onboarding bypass per test. |
| ui/playwright.config.ts | Adds slowMo configuration and forces webpack dev server for test stability. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Complete the EndpointSlug union in control.ts to cover every GET slug the stub serves (providers/configuredProviders/toolservertypes/prompttemplates), so the control seam needs no type casts. - Wait for the LoadingState overlay to clear after navigation in nav.ts, preventing flaky follow-up menu clicks landing on the overlay. - Coerce + validate E2E_SLOW_MO_MS, falling back to the default on a non-finite/negative value instead of passing NaN to Playwright's slowMo. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Carlos Logroño Guerrero <carlos.logrono@solo.io>
Charlesthebird
left a comment
There was a problem hiding this comment.
These tests are looking good! I have a couple notes - testing this out locally, I think it might be more useful to have all but the chat using real data. We can sync next week about a plan for that.
|
Updated with a real backend now in cjlg-soloio#5 |
Description
Browser-level end-to-end tests for the kagent UI, built on the merged Playwright foundation (#2262). This suite covers the one gap nothing else does: multi-step user flows across components, driven against a mocked backend.
Because the UI fetches data server-side (Next.js server actions), browser
page.routecan't intercept/api/**. Instead the suite runs a standalone stub backend (playwright/mocks/server.mjs) and points Next at it viaBACKEND_INTERNAL_URL; both servers are booted automatically by Playwright'swebServer.Test infrastructure
POST /__mock/scenariosets a per-endpoint{ status, body }override;POST /__mock/resetclears them. Default stays the happy path.mocks/control.ts) surfaced as amockfixture (fixtures/test.ts):mock.noAgents(),mock.agentsError(), etc. The fixture resets the stub before every test, so scenarios never leak between tests.page.ts(loadPage,expectNoErrors,expectToast),nav.ts(dropdown-awaregotoView/gotoCreate),select.ts(combobox driver), anda2a.ts(chat/stream SSE fixtures).mocks/data.ts) mirror the stub's payload shapes against the app's own@/types, so drift from the real API surface fails at compile time.Feature flows
One spec per core UI area, each split into a success journey and a failure journey (except the app shell — one consolidated smoke + navigation test). Each journey asserts both UI state and the payload POSTed/PUT to the backend, walking happy, empty, error, and validation states.
tests/app-shell.spec.tstests/agents/agents.spec.tstests/chat/chat-session.spec.tstests/models/models.spec.tstests/mcp/mcp-server.spec.tstests/prompts/prompt-libraries.spec.tstests/onboarding/onboarding.spec.tsA
slowMolaunch delay (250ms, overridable viaE2E_SLOW_MO_MS) keeps the retained-on-failure videos followable in real time.Product code changes are limited to additive
data-testidhooks across 8 components (chat input/send, tool-call card, combobox/dialog anchors, loading overlay) where role/text selectors were ambiguous — no runtime behavior changed.Demo videos
Each feature area recorded as a success and a failure journey (with
slowMoso the flows are followable). Grouped below by area.Playwright flow videos
App shell — agents list states (populated / empty / error) + header navigation
app-shell-list-states-and-nav.webm
Agents — success — create (declarative + harness) → delete
agents-success-create-declarative-harness-delete.webm
Agents — failure — validation / error toasts / delete-cancel
agents-failure-validation-toasts-delete-cancel.webm
Chat — success — session send / streamed reply / feedback / reload history
chat-success-session-feedback-reload-history.webm
Chat — failure — stream error + session-not-found
chat-failure-error-and-session-not-found.webm
Models — success — create / edit / delete a config
models-success-create-edit-delete.webm
Models — failure — validation + delete error
models-failure-validation-and-delete-error.webm
MCP — success — remote + stdio server create / filter / delete
mcp-success-remote-stdio-filter-delete.webm
MCP — failure — URL validation + create error
mcp-failure-url-validation-create-error.webm
Prompts — success — create / view / edit / delete
prompts-success-create-view-edit-delete.webm
Prompts — failure — blocks create when name is empty
prompts-failure-blocks-empty-name.webm
Onboarding — wizard create path
onboarding-wizard-create-path.webm
Onboarding — end with an existing model
onboarding-end-with-existing-model.webm
Onboarding — skips the wizard
onboarding-skips-wizard.webm
Testing
Playwright boots the stub backend and dev server automatically and drives each flow through happy, empty, error, and validation states. Runs serially (
workers: 1) against the shared stub. Full suite: 14 tests passing locally.Change type
test — adds the Playwright browser E2E suite. Product changes limited to additive
data-testidhooks; no runtime behavior changed.Changelog
NONE