End to End IOT Platform benchmark that can handle upto 2 million rows/sec by hyperscaledesignhub · Pull Request #8 · apache/fluss-benchmarks

hyperscaledesignhub · 2026-03-02T17:28:10Z

Purpose

Linked issue: NA

This pull request is for creating end to end fluss benchmark for IOT case where complete realtime IOT system that is inserting 2 million rows/sec into apache fluss and flink job is consuming and aggregating the data from fluss in realtime

Brief change log

complete end to end system. Which can be deployed in AWS and create the platform and stress the platform with 2 million rows/sec

Tests

NA

API and Format

NA

Documentation

NA

…g data rates upto 2 million rows/sec

wuchong · 2026-03-19T07:52:18Z

+        securityContext:
+          runAsUser: 0  # Run as root to allow writing to /etc/hosts
+        command:
+        - /app/entrypoint.sh


This image copies the wrapper to /opt/flink/bin/entrypoint.sh and the jar to /opt/flink/usrlib/fluss-flink-realtime-demo.jar, but this Job still executes /app/entrypoint.sh with /app/fluss-flink-realtime-demo.jar. In the deploy_k8s_jobs.sh flow the producer pod may fails immediately with no such file or directory before Java starts; k8s-flink-aggregator-job.yaml has the same mismatch.

Changed the entry point to /opt/flink/bin/entrypoint.sh

wuchong · 2026-03-19T07:52:18Z

+# limitations under the License.
+#
+
+#!/bin/bash


Because Dockerfile uses exec-form ENTRYPOINT ["/opt/flink/bin/entrypoint.sh"], this file needs #! in its first two bytes. With the ASF header before the shebang, running the image without overriding command returns exec format error instead of launching Java.

Changed the #! to first line

wuchong · 2026-03-19T07:52:18Z

+        long currentTime = System.currentTimeMillis();
+        long total = totalRecords.sum();
+        long elapsedSeconds = (currentTime - startTime.get()) / 1000;
+        long windowRecords = lastStatsRecords.get();


updateStats() stores the cumulative record count, but /metrics divides that total by only the time since the last update. After the first stats window, fluss_producer_records_per_second_window keeps drifting upward even if the actual short-term rate is flat or falling. This needs to use a delta count for the current window.

Changed code to use delta count by subtracting total records at this point from last updated records.

wuchong · 2026-03-19T07:52:18Z

+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+# Navigate to project root (3 levels up from this script)
+WORKDIR=$(cd "${SCRIPT_DIR}/../../.." && pwd)


From e2e-iot/fluss_flink_realtime, ../../.. resolves to the parent of fluss-benchmarks, so this script looks for demos/demo/fluss_flink_realtime_demo outside the repository. In the current layout that path does not exist, so run_kind_demo.sh cannot build or find the jar before it starts the cluster; test-local.sh and e2e-iot/high-infra/k8s/jobs/check-table-buckets.sh still use the same stale path.

Fixed this issue. Currently made SCRIPT_DIR <e2e-iot/fluss_flink_realtime > which is also made to point to DEMO_DIR. From there accessing the jar file

wuchong · 2026-03-19T07:59:29Z

+### Step 1: Build the JAR
+
+```bash
+cd /Users/vijayabhaskarv/IOT/FLUSS


There are many places still using the speicifc directory /Users/vijayabhaskarv/, could you change that to a more general directory like ~/

Replaced hardcoded /Users/vijayabhaskarv/ paths in documentation with generic paths (~/, $FLUSS_HOME, and e2e-iot/... relative to the fluss-benchmarks repo root).

wuchong · 2026-03-19T07:59:59Z

+## Prerequisites
+
+1. **Maven** - For building the JAR
+2. **Fluss 0.8.0** - Extracted to `demos/demo/deploy_local_kind_fluss/fluss-0.8.0-incubating/`


Fluss have official released 0.9, could you upgrade it to 0.9.0?

Default Fluss version is 0.9.0-incubating (in pom.xml and docs). It can be overridden via FLUSS_VERSION by running the first benchmarking step:

source ./default.env.sh --fluss-version
That version is then used for the Maven JAR build, Fluss Docker image push, and deploy.

wuchong · 2026-03-19T08:02:30Z

Thanks for the giant contribution and sorry for the late reviewing. I left some comments below.

wuchong · 2026-03-19T10:19:15Z

+        System.out.println("Starting JDBC Flink IoT Consumer with AVRO Support and 1-Minute Aggregation...");
+        System.out.println("Pulsar URL: " + pulsarUrl);
+        System.out.println("Pulsar Admin URL: " + pulsarAdminUrl);
+        System.out.println("Consuming Topic: " + baseTopicName + " (all partitions)");
+        System.out.println("ClickHouse URL: " + clickhouseUrl);


Is this class actually being used? I haven't seen any references to it, and our benchmarks don't seem to include any joint tests with Pulsar or ClickHouse.

Good catch — JDBCFlinkConsumer.java was unused (no references to it in this benchmark, and no Pulsar/ClickHouse integration). I’ve removed the file.

wuchong · 2026-03-19T11:41:31Z

+#
+
+apiVersion: v2
+appVersion: 0.8.0-incubating


It seems this helm charts is copied from apache/fluss project. Can we just to reuse the helm charts to avoid duplicate work?

We can’t use the stock Apache Helm chart as-is for this benchmark. The published chart doesn’t properly support persistence.enabled for tablet servers (still uses emptyDir), but this setup needs local NVMe PVCs on EC2 for high-throughput tablet storage.

We keep a vendored chart under e2e-iot/high-infra/helm-charts/fluss/ with volumeClaimTemplates and local-storage persistence. The Fluss image version is not fixed in the chart — it comes from FLUSS_VERSION via source ./default.env.sh --fluss-version …, then push-images-to-ecr.sh and deploy.sh (--set image.tag).

wuchong · 2026-03-30T11:57:38Z

Hi @hyperscaledesignhub , do you have time to update this? Otherwise, I can help it.

hyperscaledesignhub · 2026-05-24T08:44:46Z

Hi @hyperscaledesignhub , do you have time to update this? Otherwise, I can help it.

Hi @wuchong sorry for delay, i'll update

hyperscaledesignhub · 2026-06-10T11:00:21Z

I am doing testing again after making changes. It may take few more days

Support --fluss-version on push-images-to-ecr.sh and download-helm-chart.sh, defaulting to 0.9.0-incubating, and wire FLUSS_VERSION through default.env.sh and deploy scripts so the selected image tag is used end-to-end.

Explain --fluss-version, default.env.sh, and matching ECR tags across deployment guides, scripts docs, and local testing instructions.

default.env.sh sets FLUSS_VERSION from a required --fluss-version argument. push-images-to-ecr.sh and download-helm-chart.sh read FLUSS_VERSION from the environment instead of CLI flags. Update deploy scripts and docs.

Fix K8s job paths and entrypoint shebang, reduce producer log noise with interval-based stats, correct Prometheus window metrics, align bucket checks with 128-bucket deployments, upgrade to Fluss 0.9, remove unused JDBC consumer, and improve AWS deploy scripts, monitoring, and Helm chart configuration.

Update markdown guides to reference e2e-iot/fluss_flink_realtime and e2e-iot/high-infra from the fluss-benchmarks repository root instead of the old benchmark/e2e-platform-aws paths.

Use -Dfluss.version from FLUSS_VERSION in Kind demo scripts and Flink Docker build paths so local builds match the version set in default.env.sh.

Fix K8s job paths and entrypoint shebang, reduce producer log noise with interval-based stats, correct Prometheus window metrics, align bucket checks with 128-bucket deployments, upgrade to Fluss 0.9, remove unused JDBC consumer, and improve AWS deploy scripts, monitoring, and Helm chart configuration.

hyperscaledesignhub · 2026-06-19T11:26:10Z

I have finished all my testing after these changes. Everything is good

hyperscaledesignhub · 2026-07-17T11:28:40Z

@wuchong any update? Its been long time pending.

hyperscaledesignhub added 4 commits February 17, 2026 07:47

Add AWS E2E platform benchmarking folder

72934e9

Adding gitignore

5dbb363

deleting gitignore to copy into benchmark specific folder

f561a79

Adding primary benchmarks of complete end to end iot platform handlin…

a595d25

…g data rates upto 2 million rows/sec

wuchong reviewed Mar 19, 2026

View reviewed changes

hyperscaledesignhub added 3 commits June 12, 2026 12:37

Add configurable Fluss version for ECR push and deploy.

b478093

Support --fluss-version on push-images-to-ecr.sh and download-helm-chart.sh, defaulting to 0.9.0-incubating, and wire FLUSS_VERSION through default.env.sh and deploy scripts so the selected image tag is used end-to-end.

Document FLUSS_VERSION configuration for push and deploy.

877e661

Explain --fluss-version, default.env.sh, and matching ECR tags across deployment guides, scripts docs, and local testing instructions.

Require --fluss-version in default.env.sh for push and deploy.

ffd948a

default.env.sh sets FLUSS_VERSION from a required --fluss-version argument. push-images-to-ecr.sh and download-helm-chart.sh read FLUSS_VERSION from the environment instead of CLI flags. Update deploy scripts and docs.

hyperscaledesignhub force-pushed the e2e-benchmark-aws branch from bdf0fd2 to 64e884f Compare June 15, 2026 09:50

hyperscaledesignhub added 2 commits June 15, 2026 15:47

Fix documentation paths to use e2e-iot layout.

7c18e2a

Update markdown guides to reference e2e-iot/fluss_flink_realtime and e2e-iot/high-infra from the fluss-benchmarks repository root instead of the old benchmark/e2e-platform-aws paths.

Pass FLUSS_VERSION to Maven in remaining JAR build paths.

b1878f9

Use -Dfluss.version from FLUSS_VERSION in Kind demo scripts and Flink Docker build paths so local builds match the version set in default.env.sh.

hyperscaledesignhub force-pushed the e2e-benchmark-aws branch 2 times, most recently from 6f6acc5 to b1878f9 Compare June 15, 2026 14:15

Uh oh!

Conversation

hyperscaledesignhub commented Mar 2, 2026

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wuchong commented Mar 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wuchong commented Mar 30, 2026

Uh oh!

hyperscaledesignhub commented May 24, 2026

Uh oh!

hyperscaledesignhub commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hyperscaledesignhub commented Jun 19, 2026

Uh oh!

hyperscaledesignhub commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hyperscaledesignhub commented Jun 10, 2026 •

edited

Loading