solidify the metrics to mlflow - #1
Closed
ashtarkb wants to merge 393 commits into
Closed
Conversation
…launching a command
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
…erdomo2/forge into feat/benchmark-matrix-isolation
The secret name was hardcoded to "storage-config". Now it's driven by the benchmarks.guidellm.hf_token_secret config key and conditionally injected—if the key is empty or omitted, no HF_TOKEN env var is added to the K8s job manifest. Co-authored-by: Cursor <cursoragent@cursor.com>
[llm-d] Minor improvements
…build the llmisvc src
[llm-d] Add dry-run capability
…mark-matrix-isolation feat: Isolate benchmark workloads with per-deployment lifecycle
Adds a clusterless nightly pipeline that automatically detects new image versions and triggers test runs when a new version is found. Architecture: - Core nightly package (projects/core/nightly/) defines abstract interfaces (ImageReceiver, NightlyVerifier) with built-in retry logic. Projects implement these to define how versions are fetched and how run history is checked. - Phase handlers (receive_image.py, confirm.py) use polymorphism to call the project's implementations without knowing the concrete types. - Auto-discovery in the project's resolvers/ package — adding a new source is just dropping a new file with a NAME attribute. mcp_gateway implementation: - GHCRReceiver: finds the latest commit with a published ghcr.io image - MLflowVerifier: queries MLflow experiment runs to find the last tested version - CronJob triggers a clusterless FournosJob nightly Pipeline flow: 1. CronJob creates a clusterless FournosJob with pipeline: "nightly" 2. receive-image task: calls project's get_receiver().get_latest_version() 3. confirm-nightly task: calls project's get_verifier().get_last_tested_version(), compares versions, creates a full FournosJob if new version detected Also generalizes task-forge-step env var export to loop over all spec.env keys instead of hardcoding specific variable names. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Address PR review comments: - Merge two pipeline tasks (receive-image, confirm-nightly) into single "nightly" task for simplicity; error visibility via notification files - Remove retry from base ABCs, keep it in concrete implementations - Reuse fournos_launcher submit_and_wait with wait=False + EarlyReturn - Rename resolvers/ to nightly_resolvers/ - Use env module instead of raw os.environ for artifact paths - Make should_init_vaults configurable via vaults.skip_phases in config - Use quay.io/openshift/origin-cli image, fix namespace to psap-automation - Fix list_vaults() to skip the skip_phases key Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The DSL runtime skips underscore-prefixed context attributes when propagating between tasks, causing wait_service_mesh_ready to never see the readiness spec set by install_service_mesh_instance. Co-authored-by: Cursor <cursoragent@cursor.com>
Sends a webhook notification with job metadata (model, TP/DP, version, cluster, workloads, error) so failures are visible without checking the Fournos dashboard. Co-authored-by: Cursor <cursoragent@cursor.com>
Each FournosJob now gets a unique vLLM deployment name derived from the FJOB_NAME suffix, allowing multiple jobs to run in parallel on the same namespace without colliding on the same InferenceService. Co-authored-by: Cursor <cursoragent@cursor.com>
…s channel - Use topsail bot token instead of webhook for sending Slack notifications - Route RHAIIS notifications to dedicated psap-rhaiis-alerts channel - Add pipeline-level failure detection in post_cleanup for infra errors (e.g. image pull timeout) - Fix UnboundLocalError for 'os' caused by redundant import in _run_test Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…tom catalog is enabled
…r prefill/decode pods
Replace the stub run_kpi_analysis with a working regression engine that: - Matches current KPIs to historical baselines via configurable comparison_keys/ignored_keys on record labels - Runs per-KPI regression tests (relative threshold, higher/lower-is-better) - Produces a structured YAML report with processed/tested/results/overall - Supports plugin-provided AnalysisConfig for per-project tuning Exit codes: 0=pass, 1=error, 2=no-baseline, 3=regression-detected. Includes end-to-end test suite (15 tests). Co-authored-by: Cursor <cursoragent@cursor.com>
[llm-d] Start testing P/D on CKS
Move metrics.json generation from project-specific parsers into caliper as a generic mechanism. After kpis.json is produced, caliper now automatically writes per-run metrics.json and parameters.json files using KPI ids as metric keys. The MLflow export backend picks these up via _log_metrics_and_params_from_tree without any project-specific code. - Add caliper/engine/kpi/metrics_from_kpis.py with generate_metrics_from_kpis() - Wire into postprocess pipeline after kpis.json generation step - Remove mcp_gateway-specific metrics.json/parameters.json writing - Simplify mcp_gateway notifications MLflow metric key lookup - Update tests to reflect new ownership Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
Made with Cursor