Skip to content

WIP [llm-d] feat: Add new EPP profiles - #185

Open
albertoperdomo2 wants to merge 12 commits into
openshift-psap:mainfrom
albertoperdomo2:feat/new-epp-profiles
Open

WIP [llm-d] feat: Add new EPP profiles#185
albertoperdomo2 wants to merge 12 commits into
openshift-psap:mainfrom
albertoperdomo2:feat/new-epp-profiles

Conversation

@albertoperdomo2

@albertoperdomo2 albertoperdomo2 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 25, 2026
@coderabbitai

coderabbitai Bot commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f16c8fc3-b4ba-4cc8-ba9b-b4bb43e36ce9

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 25, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign albertoperdomo2 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 25, 2026
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 25, 2026
@psap-forge-bot

psap-forge-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

🟢 Execution of llm_d upstream-optimized-baseline llama-33-70b 🟢

Execution Engine Configuration

forge:
  args:
  - cpt-release-testing-llama-33-70b
  configOverrides:
    model_cache.pvc.access_mode: ReadWriteMany
    model_cache.pvc.storage_class_name: nfs-rwx
    platform.cluster.skip_gpu_readiness: true
    platform.operators.rhods-operator.channel: stable-3.5
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e47d6cf935c27d664d9c3ce287aeb3dcae2f3ffe9133ac062baa0a0c4f58eed7
    runtime.deployment_profile: release-upstream-optimized-baseline
    workloads.pvc_storage_class: nfs-rwx
  project: llm_d

Artifact Links

Test Logs

✅ 00 Preflight 2 seconds

✅ 01 Test 2 hours, 30 minutes, 49 seconds

Test Description

This test validates the llm_d project performance for RHOAI 3.5 EA2 release by benchmarking the Llama-3.3-70B-Instruct-FP8-dynamic model on H200 GPUs across three deployment profiles: release-distributed-default, release-precise-prefix-cache, and release-approximate-prefix-cache. It executes concurrent, heavy-heterogeneous, and multi-turn workloads to evaluate system scalability and prefix caching effectiveness under varying load conditions.

🔄 02 Export-Artifacts

Post-processing Status ✅

@psap-forge-bot

psap-forge-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

🟢 Execution of llm_d upstream-optimized-baseline gpt-oss-120b 🟢

Execution Engine Configuration

forge:
  args:
  - cpt-release-testing-gpt-oss-120b
  configOverrides:
    model_cache.pvc.access_mode: ReadWriteMany
    model_cache.pvc.storage_class_name: nfs-rwx
    platform.cluster.skip_gpu_readiness: true
    platform.operators.rhods-operator.channel: stable-3.5
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e47d6cf935c27d664d9c3ce287aeb3dcae2f3ffe9133ac062baa0a0c4f58eed7
    runtime.deployment_profile: release-upstream-optimized-baseline
    workloads.pvc_storage_class: nfs-rwx
  project: llm_d

Artifact Links

Test Logs

✅ 00 Preflight 3 seconds

✅ 01 Test 2 hours, 11 minutes, 29 seconds

Test Description

This test validates the llm_d project for RHOAI 3.5 EA2 release on H200 GPUs using the gpt-oss-120b model, evaluating performance across three deployment profiles (release-distributed-default, release-precise-prefix-cache, release-approximate-prefix-cache) and three benchmark workloads (concurrent-1k-1k, heavy-heterogeneous, multi-turn). It specifically measures text completion throughput and latency under these varying deployment and load configurations.

🔄 02 Export-Artifacts

Post-processing Status ✅

kpouget
kpouget reviewed Aug 26, 2026
View reviewed changes
Comment thread projects/llm_d/orchestration/config.d/deployments.yaml
# The serving image expands VLLM_ADDITIONAL_ARGS through `eval`; literal
# single quotes keep the JSON intact for vLLM after that shell parse.
kv_events_config: "'{\"enable_kv_cache_events\": true, \"publisher\": \"zmq\", \"endpoint\": \"tcp://__INFERENCE_SERVICE_NAME__-epp-service:5557\", \"topic\": \"kv@$(POD_IP)@$(MODEL_NAME)\"}'"
kv_events_config: "'{\"enable_kv_cache_events\": true, \"publisher\": \"zmq\", \"endpoint\": \"tcp://*:5556\", \"topic\": \"kv@$(POD_IP)@$(MODEL_NAME)\"}'"

@kpouget kpouget Aug 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I started reworking this in my PR, to make this more flexible
I hope it will work for you 🤞🏻

@albertoperdomo2 albertoperdomo2 Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, no problem! I'll rebase and adapt it when you are done 👌

@psap-forge-bot

psap-forge-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown

🟢 Execution of llm_d cpt-gpt-oss-120b precise-v2 multi-turn 🟢

Execution Engine Configuration

forge:
  args:
  - cpt-release-testing-gpt-oss-120b
  configOverrides:
    model_cache.pvc.access_mode: ReadWriteMany
    model_cache.pvc.storage_class_name: nfs-rwx
    platform.cluster.skip_gpu_readiness: true
    platform.operators.rhods-operator.channel: stable-3.5
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e47d6cf935c27d664d9c3ce287aeb3dcae2f3ffe9133ac062baa0a0c4f58eed7
    runtime.benchmark_key: multi-turn
    runtime.deployment_profile: release-precise-prefix-cache
    workloads.pvc_storage_class: nfs-rwx
  project: llm_d

Artifact Links

Test Logs

✅ 00 Preflight 3 seconds

✅ 01 Test 12 minutes, 7 seconds

Test Description

This test validates the llm_d project's performance for the GPT-OSS-120B model on H200 GPUs as part of the RHOAI 3.5 EA2 release testing suite. It evaluates three deployment profiles—distributed default, precise prefix cache, and approximate prefix cache—across concurrent, heavy heterogeneous, and multi-turn benchmark workloads.

🔄 02 Export-Artifacts

Post-processing Status ✅

@psap-forge-bot

psap-forge-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

🔴 Execution of llm_d cpt-gpt-oss-120b precise-v2 multi-turn 🔴

Execution Engine Configuration

forge:
  args:
  - cpt-release-testing-gpt-oss-120b
  configOverrides:
    model_cache.pvc.access_mode: ReadWriteMany
    model_cache.pvc.storage_class_name: nfs-rwx
    platform.cluster.skip_gpu_readiness: true
    platform.operators.rhods-operator.channel: stable-3.5
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e47d6cf935c27d664d9c3ce287aeb3dcae2f3ffe9133ac062baa0a0c4f58eed7
    runtime.benchmark_key: multi-turn
    runtime.deployment_profile: release-precise-prefix-cache
    workloads.pvc_storage_class: nfs-rwx
  project: llm_d

Artifact Links

Test Logs

✅ 00 Preflight 3 seconds

❌ 01 Test 13 minutes, 1 second

Test Description

This FORGE test runs the llm_d RHOAI release CPT for openai/gpt-oss-120b on the athena-fire cluster, selecting the release-precise-prefix-cache deployment (“precise-v2”) and the multi-turn benchmark. It validates the precise prefix-caching configuration and KPI export under the cpt-release-testing-gpt-oss-120b preset.

🔄 02 Export-Artifacts

Post-processing Status ❌

  • ✅ parse: success
  • ❌ visualize: failure
  • ❌ artifacts_to_kpis: failed
  • ✅ kpis_to_csv: success
  • ⏭️ artifacts_to_ai_data: disabled
    • kpi.artifacts_to_ai_data disabled
  • ✅ s3_import: success
  • ❌ analyse_kpis: failed
  • ✅ s3_export: success
    • 📤 Exported to: s3://psap-dashboard-data/forge-cpt/llm-d/release/26-08-27_095536_466/
    • ✅ Uploaded files: 1

@psap-forge-bot

psap-forge-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

🔴 Execution of llm_d cpt-gpt-oss-120b precise-v2 multi-turn 🔴

Execution Engine Configuration

forge:
  args:
  - cpt-release-testing-gpt-oss-120b
  configOverrides:
    model_cache.pvc.access_mode: ReadWriteMany
    model_cache.pvc.storage_class_name: nfs-rwx
    platform.cluster.skip_gpu_readiness: true
    platform.operators.rhods-operator.channel: stable-3.5
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e47d6cf935c27d664d9c3ce287aeb3dcae2f3ffe9133ac062baa0a0c4f58eed7
    runtime.benchmark_key: multi-turn
    runtime.deployment_profile: release-precise-prefix-cache
    workloads.pvc_storage_class: nfs-rwx
  project: llm_d

Artifact Links

Test Logs

✅ 00 Preflight 3 seconds

❌ 01 Test 12 minutes, 54 seconds

Test Description

This FORGE test runs the llm_d project’s RHOAI release CPT for openai/gpt-oss-120b on the athena-fire OCP cluster. It focuses on the release-precise-prefix-cache deployment profile and a multi-turn text-completions benchmark.

🔄 02 Export-Artifacts

Post-processing Status ❌

  • ✅ parse: success
  • ❌ visualize: failure
  • ❌ artifacts_to_kpis: failed
  • ✅ kpis_to_csv: success
  • ⏭️ artifacts_to_ai_data: disabled
    • kpi.artifacts_to_ai_data disabled
  • ✅ s3_import: success
  • ❌ analyse_kpis: failed
  • ✅ s3_export: success
    • 📤 Exported to: s3://psap-dashboard-data/forge-cpt/llm-d/release/26-08-27_102632_985/
    • ✅ Uploaded files: 1

@albertoperdomo2
fix: API versions
cd1f370 
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
@psap-forge-bot

psap-forge-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

🔴 Execution of llm_d cpt-gpt-oss-120b precise-v2 multi-turn 🔴

Execution Engine Configuration

forge:
  args:
  - cpt-release-testing-gpt-oss-120b
  configOverrides:
    model_cache.pvc.access_mode: ReadWriteMany
    model_cache.pvc.storage_class_name: nfs-rwx
    platform.cluster.skip_gpu_readiness: true
    platform.operators.rhods-operator.channel: stable-3.5
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e47d6cf935c27d664d9c3ce287aeb3dcae2f3ffe9133ac062baa0a0c4f58eed7
    runtime.benchmark_key: multi-turn
    runtime.deployment_profile: release-precise-prefix-cache
    workloads.pvc_storage_class: nfs-rwx
  project: llm_d

Artifact Links

Test Logs

✅ 00 Preflight 3 seconds

❌ 01 Test 11 minutes, 46 seconds

Test Description

This FORGE test runs the llm_d project under the RHOAI release CPT preset for openai/gpt-oss-120b, specifically testing the release-precise-prefix-cache deployment profile with the multi-turn benchmark on the athena-fire OCP cluster.

🔄 02 Export-Artifacts

Post-processing Status ❌

  • ✅ parse: success
  • ❌ visualize: failure
  • ❌ artifacts_to_kpis: failed
  • ✅ kpis_to_csv: success
  • ⏭️ artifacts_to_ai_data: disabled
    • kpi.artifacts_to_ai_data disabled
  • ✅ s3_import: success
  • ❌ analyse_kpis: failed
  • ✅ s3_export: success
    • 📤 Exported to: s3://psap-dashboard-data/forge-cpt/llm-d/release/26-08-27_132004_765/
    • ✅ Uploaded files: 1

@albertoperdomo2
fix: Temporary switch to external
8c9955b 
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
@psap-forge-bot

psap-forge-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

🟢 Execution of llm_d cpt-gpt-oss-120b precise-v2 multi-turn 🟢

Execution Engine Configuration

forge:
  args:
  - cpt-release-testing-gpt-oss-120b
  configOverrides:
    model_cache.pvc.access_mode: ReadWriteMany
    model_cache.pvc.storage_class_name: nfs-rwx
    platform.cluster.skip_gpu_readiness: true
    platform.operators.rhods-operator.channel: stable-3.5
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e47d6cf935c27d664d9c3ce287aeb3dcae2f3ffe9133ac062baa0a0c4f58eed7
    runtime.benchmark_key: multi-turn
    runtime.deployment_profile: release-precise-prefix-cache
    workloads.pvc_storage_class: nfs-rwx
  project: llm_d

Artifact Links

Test Logs

✅ 00 Preflight 3 seconds

✅ 01 Test 16 minutes, 44 seconds

Test Description

This FORGE run tests the llm_d project’s RHOAI-release CPT for openai/gpt-oss-120b on the athena-fire OCP cluster. It focuses on the release-precise-prefix-cache deployment profile using the multi-turn text-completions benchmark.

🔄 02 Export-Artifacts

Post-processing Status ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kpouget kpouget kpouget left review comments

Assignees

No one assigned

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@albertoperdomo2 @kpouget