feat: add headless CLI and MCP server modes - #16
Merged
Conversation
Add Frasy::CliArgs struct with static parse() and get() methods. Supports the following flags: --headless, --product, --operator, --serial (repeatable), --config, --output-format, --output-dir, --skip-verification, --popup-timeout, --help Validates required flags in headless mode (product, operator, serial) and exits with code 2 on invalid arguments. Non-headless mode ignores headless-specific flags gracefully. Includes 25 unit tests covering all parsing paths, validation errors, and death tests for std::exit behavior.
…rface - Add Frasy::Headless::ProductProvider abstract class with: - validateSerialNumber() for product-specific serial validation - setup() to configure orchestrator and CANopen for a product - onTestComplete() optional post-test hook - Add setProductProvider()/getProductProvider() on Frasy::Interpreter - Modify Brigerad EntryPoint.h to: - Parse CLI args before application creation - Branch on --headless after CreateApplication - Check for registered ProductProvider (exits with code 2 if missing) - Use exitCode variable to avoid returning inside BR_GUARDED_SCOPE - Add demo_mode/CMakeLists.txt for cmake-based building - Fix demo_mode to use new getExpectationsMutex/getExpectationsVector API Tested: headless mode exits with code 2 (no provider), GUI mode runs normally.
- Add 'visible' field to WindowProps (default: true) - Add GLFW_VISIBLE=FALSE hint in WindowsWindow::Init() when not visible - Add 'visible' parameter to Brigerad::Application constructor - Frasy::Interpreter passes !headless as visible to Application base In headless mode the OpenGL context still initializes (required by framework internals) but no window appears on screen. Normal GUI mode is unaffected — visible defaults to true. Tested: headless runs without any visible window, GUI mode shows window as before.
Add Frasy::Headless::HeadlessRunner that drives the full headless test lifecycle: 1. Discovers products by scanning lua/user/ 2. Validates requested product exists (lists available on error) 3. Validates serial numbers via ProductProvider 4. Calls ProductProvider::setup() to configure orchestrator 5. Validates serial count matches UUT count from environment 6. Runs the solution asynchronously and polls for completion 7. Reports results and returns exit code (0=pass, 1=fail, 2=error) Wire HeadlessRunner into EntryPoint.h replacing the stub. Add DemoProductProvider in demo_mode that accepts any non-empty serial and configures orchestrator/CANopen from the IB map. Add CLI-tests product (simple math/range/string tests with no hardware) for headless mode validation. Tested: - CLI-tests product runs all 5 tests, all pass, exits 0 - Invalid product name lists available products, exits 2 - GUI mode unaffected
Implement text-based popup interaction via stdin/stdout: - Present popups with text, numbered inputs, and button labels - Accept N=value to set inputs, button label to consume - Call button action callbacks with input values (matching GUI behavior) - Support human and JSON output formats - Timeout support via --popup-timeout flag (auto-cancel on expiry) - EOF handling: auto-cancel gracefully if stdin closes - Trim trailing whitespace from stdin (Windows pipe compat) Multi-UUT popup serialization: - Hold ioMutex for entire popup interaction (present + read + consume) - Ensures one popup completes before the next presents - Prevents interleaving from concurrent UUT threads Redirect logs to stderr in headless mode: - Add useStderr parameter to Brigerad::Log::Init() - In headless mode, stdout is exclusively for structured output - Logs go to stderr, eliminating interleaving with popup display Add setPopupImport() to Orchestrator: - Allows HeadlessRunner to install alternative popup handling - Falls back to default GUI importPopup if not set Add test products: - CLI-tests/popup_test.lua: confirm popup, single input, multi-input - CLI-tests-multi: 2-UUT product validating popup serialization Tested: single-UUT popups with input validation, multi-UUT serialization, clean stdout/stderr separation.
Add plan for --mcp-server mode where Frasy acts as an MCP stdio tool server (JSON-RPC). Enables AI agents to launch tests, interact with popups, and retrieve results via Kiro's native MCP integration. Tools: list_products, run_tests, get_status, get_pending_popup, respond_to_popup, get_results, abort. Shifted Progress Reporter to Task 7, Report Summary to Task 8, Demo Refactor to Task 9, Documentation to Task 10.
Implement --mcp-server flag that runs Frasy as a Model Context Protocol (MCP) stdio server. AI agents (via Kiro) can launch tests, interact with popups, and retrieve results using native MCP tool calls. MCP protocol implementation (mcp_server.h/.cpp): - JSON-RPC over stdin/stdout (newline-delimited, protocol v2024-11-05) - Handles initialize handshake, tools/list, tools/call, ping - Tool registration with JSON Schema input definitions MCP tools (mcp_runner.h/.cpp): - list_products: discovers available products in lua/user/ - run_tests: validates serials, sets up orchestrator, launches async - get_status: returns running/passed/failed/error + per-UUT states - get_results: reads report JSONs, returns test counts and duration - get_pending_popup: returns next popup awaiting interaction - respond_to_popup: sends inputs and button press to unblock popup MCP popup handler (mcp_popup_handler.h/.cpp): - Installs into Lua state as alternative popup import - Queues popups with condition variable for blocking - Agent polls get_pending_popup, responds via respond_to_popup - Button action callbacks invoked with input values Additional changes: - --mcp-server and --headless are mutually exclusive - Window hidden and logs to stderr in MCP mode - Graceful shutdown: waits for running orchestrator before exit - Add CLI-tests-nopopup product for no-popup MCP testing Tested live via Kiro MCP connection: - list_products returns all products - run_tests + get_status + get_results lifecycle works - Popup interaction (confirm, single input, multi-input) all pass - 8/8 tests passed on CLI-tests product with agent-driven popups
Replace polling approach with event-driven progress reporting: Orchestrator changes: - Add setProgressCallback() method - Install __progress.report() Lua binding in initLua for execution stage - No-op binding for generation/validation stages Lua framework changes (orchestrator.lua): - Call __progress.report at sequence start/end - Call __progress.report at test start/end - Call __progress.report at each expectation evaluation - Events carry type, UUT, name, parent context, and pass/fail ProgressReporter (headless): - Receives events via callback (no polling thread) - Human format: indented tree with >> sequence, > test, [PASS]/[FAIL] - JSON format: structured JSON-lines per event HeadlessRunner integration: - Installs progress callback before runSolution - Maps string type to ProgressEvent enum - Forwards to ProgressReporter::onEvent() Every sequence, test, and expectation is now reported in real-time regardless of execution speed.
In headless and MCP server modes, console logs are now suppressed by default. Only structured progress output appears on stdout. Logs still go to the rotating file sink (logs/frasy/log.json). Add --verbose flag to re-enable stderr logging when needed for debugging. GUI mode is unaffected (always logs to stdout). Changes: - Add 'silent' parameter to Brigerad::Log::Init() - When silent, no console sink is added (file sink only) - Add --verbose to CliArgs and help text - EntryPoint passes silent=true when headless/MCP without --verbose Tested: without --verbose stderr is empty, with --verbose stderr shows all logs.
Use ANSI color codes in human-readable progress format: - White (reset): SequenceStart, TestStart entries - Green: entries with pass=true (SequenceEnd, TestEnd, Expectation) - Red: entries with pass=false JSON format is unaffected (no color codes in structured output).
Replace the simple log-based result reporting with a full summary:
Human format:
- Colored bordered table with per-UUT results
- Test counts (passed/total), duration, report file paths
- Green [PASS] / Red [FAIL] with overall result
JSON format:
- Structured run_end object with overall_pass, product, and
per-UUT array (serial, state, tests_passed, tests_total,
duration, report path)
Exit codes:
- 0: all enabled UUTs passed
- 1: one or more UUTs failed (expectation failures)
- 2: error (orchestrator crash, missing reports, setup failure)
Reads logs/last/{uut}.json for detailed test counts and timing.
Tested: all-pass returns 0, popup-timeout failure returns 1.
Refactor MyMainApplicationLayer::makeOrchestrator to delegate orchestrator and CANopen setup to ProductProvider::setup(), establishing the shared-logic pattern between GUI and headless paths. Before: makeOrchestrator duplicated loadUserFiles, CANopen node configuration, and start logic inline. After: makeOrchestrator calls provider->setup() for all orchestrator/ CANopen work, then handles GUI-specific post-setup only (serial fields resize, active product assignment). This demonstrates the pattern for other applications: implement ProductProvider::setup() once, call it from both GUI makeOrchestrator and the headless/MCP runner. Tested: GUI mode launches normally, headless mode passes all tests.
Add getting-started/headless-mode.md: - Full CLI flag reference (--headless, --mcp-server, --product, etc.) - Exit code meanings (0=pass, 1=fail, 2=error) - Human and JSON output format examples - Popup interaction protocol (stdin commands + JSON response) - MCP server mode: tool descriptions, workflow, configuration Add developer-guide/headless-mode.md: - ProductProvider interface and method responsibilities - Registration in Interpreter constructor - makeOrchestrator refactoring pattern (before/after) - Product routing examples - Testing instructions for all modes Update architecture/overview.md: - Add headless/MCP to Key Design Decisions with links Update mkdocs.yml: - Add both pages to Getting Started and Developer Guide nav Verified: mkdocs build --strict passes.
Add FrasyTest_Headless module with 32 tests covering: MCP server (6 tests): - Tool registration - JSON-RPC message format validation (initialize, tools/list, tools/call result, error responses) Console popup parsing (14 tests): - parseHumanLine: button-only, input assignment (N=value), values with equals, non-numeric equals, empty values - parseJsonLine: button-only, button+inputs, inputs-only, invalid JSON, empty object, multiple inputs Progress reporter (9 tests): - Human format: sequence start/end, test start, expectation - JSON format: sequence_start, test_end, expectation with all fields, timestamp presence Product discovery (3 tests): - Directory structure with environment.lua detection - Directories without environment.lua excluded - Empty user directory returns no products Refactored parseHumanLine/parseJsonLine from anonymous namespace to Frasy::Headless (declared in header) for testability. Total test count: 57 (25 cli_args + 32 headless), all passing.
smartel99
marked this pull request as ready for review
August 5, 2026 20:47
The orchestrator.lua now calls __progress.report() during execution, but the test fixture never defined this global, causing all ExpectationClass tests to crash with 'attempt to index a nil value'. Adds a no-op __progress.report mock matching the stub that orchestrator.cpp provides for non-callback scenarios.
CTest runs each test as a separate process in parallel. All three ProductDiscovery tests were using the same fixed temp path, so one test's remove_all would delete directories another test just created. Use a randomly-suffixed unique directory per test invocation to eliminate the race condition.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add two headless execution modes to Frasy at the framework level:
--headless) — run tests from the command line with colored progress output, stdin-based popup interaction, and JSON/human report formats--mcp-server) — run as a Model Context Protocol tool server over stdio, enabling AI agents (via Kiro) to launch tests, interact with popups, and retrieve results using native tool callsKey Features
setup()method shared between GUI and headless paths (no code duplication)list_products,run_tests,get_status,get_pending_popup,respond_to_popup,get_results--verboseis passed; stdout is exclusively structured outputTesting
Documentation