Skip to content

Add a network-contention benchmark to the demo harness - #30

Merged
Fiona2016 merged 3 commits into
mainfrom
feat/netbench-harness
Aug 18, 2026
Merged

Add a network-contention benchmark to the demo harness#30
Fiona2016 merged 3 commits into
mainfrom
feat/netbench-harness

Conversation

@Fiona2016

Copy link
Copy Markdown
Collaborator

Why

0.5.0 shipped three pacing settings and advice on when to reach for them, but
nothing measured whether upload traffic actually delays an app's own requests, or
by how much. The advice rested on the mechanism being plausible.

It holds up. Against a silent-SDK baseline at 2.7 Mbps, the default pacing costs
+59% at p95, and RARE + LARGE + LOW costs +7%.

Group (2.7 Mbps, median of 3 runs) mean p95 vs baseline
SDK silent 58 ms 59 ms
default (AVERAGE/MEDIUM/MEDIUM) 66 ms 94 ms mean +13.8% · p95 +59.3%
low (RARE/LARGE/LOW) 57 ms 63 ms mean −1.7% · p95 +6.8%

Cross-checked at 0.5 Mbps: +53.1% against +20.2% at p95. Same direction, and the
effect sits in the tail rather than the mean — the shape reported for the Android
investigation.

What

A netbench scenario on the existing e2e harness rather than a separate project:
the mock intake already serves both the SDK's uploads and a stand-in origin for
the app's requests, which is exactly the shared link the experiment needs. Runs
are driven by aa start --ps and collected from HiLog, so a full run is
scriptable over hdc — emulator or real device, no tapping.

UPLINK_BPS=2700000 node scripts/e2e/mock-intake.mjs 19533 /tmp/capture.ndjson
hdc shell aa start -a EntryAbility -b com.flashcat.sdk.harmony \
  --ps e2e_endpoint http://10.0.2.2:19533 --ps e2e_scenario netbench \
  --ps netbench_pacing low --ps netbench_rounds 40
hdc shell "hilog -x" | grep NETBENCH

Throttling is opt-in via UPLINK_BPS, so existing smoke runs are untouched.

Notes for review

The server models a link, not a quota. A token bucket sized to a second of
bytes was the first attempt and it measured nothing: the bucket starts full, so a
request smaller than the bucket draws its bytes instantly and never queues. Real
links have no burst credit — they serialize at the line rate, and a request
arriving mid-transmission waits. That queueing is the effect under test.

Groups are whole runs, not alternating arms. Alternating uploads on/off
inside one run reads as the stronger design, but consent PENDING stops new drains
rather than an upload already in flight. Once an upload outlives its round it
spills into the supposedly quiet arm and both arms carry traffic. At 0.5 Mbps
that erased the effect entirely — not because contention had gone, but because
the baseline was contaminated. Drift is handled by repetition and medians.

No flush(). Forcing an upload bypasses the batch window, the cycle interval
and the per-cycle bound — every setting the run exists to compare. Load is
organic and the SDK's own scheduler decides when to upload.

Verification

  • ./scripts/ci-check.sh — ALL CHECKS PASSED (148 tests, HAR build, demo HAP
    build, ArkTS unit-test type check).
  • Results above reproduced after the final refactor: off 56/61, default 66/100,
    low 58/66.
  • Measured on the DevEco emulator. The emulator reproduces contention and the
    relative ordering faithfully, but its queue is cleaner than a real radio, so
    the absolute figures should not be quoted externally without a device run.

The pacing settings shipped in 0.5.0 were only ever exercised against fakes.
Nothing measured whether upload traffic actually delays the app's own requests,
or by how much, so the tuning advice rested on the mechanism being plausible
rather than on a number.

Adds a `netbench` scenario to the existing e2e harness rather than a separate
project: the mock intake already serves both the SDK's uploads and a stand-in
origin for the app's requests, which is exactly the shared link the experiment
needs. Runs are driven by `aa start --ps` and collected from HiLog, so a whole
run is scriptable over hdc with no tapping, on an emulator or a real device.

The link model matters and the obvious one is wrong. A token bucket sized to a
second of bytes measures nothing: it starts full, so a request smaller than the
bucket draws its bytes instantly and never queues. Real links have no burst
credit — they serialize at the line rate, and a request arriving mid-transmission
waits. That queueing is the effect under test, so the server models a link, not
a quota. Throttling stays opt-in via UPLINK_BPS, leaving existing smoke runs
untouched.

Arms alternate round by round rather than running as two blocks, so drift over
the session cancels instead of landing entirely on whichever arm ran second.

Known limitation, recorded here because the numbers look usable and are not:
the harness force-flushes to guarantee the upload overlaps the measured request,
and a force-flush bypasses the very pacing it is meant to compare. Results are
therefore valid for "does upload traffic cause contention" (it does — p95 rose
32-44% at 2.7 Mbps) and invalid for comparing BatchProcessingLevel settings. A
second iteration needs organic load and no flush.
The first harness alternated uploads on/off round by round inside a single run
and force-flushed each round to guarantee the upload overlapped the measured
request. Both choices defeated the experiment.

The flush was the direct problem: forcing an upload bypasses the batch window,
the cycle interval and the per-cycle bound, so every setting the run existed to
compare was inert. Load is now organic — events are generated continuously and
the SDK's own scheduler decides when to upload.

The alternation was the subtler one. Consent PENDING stops new drains, not an
upload already in flight, so once an upload outlives its round it spills into
the supposedly quiet arm and both arms carry upload traffic. At 0.5 Mbps that
erased the effect completely: p95 delta fell from +32..44% to roughly zero, not
because contention had gone but because the baseline had become contaminated.
Groups are now whole runs, with drift handled by repetition and medians.

With that, the settings separate cleanly and reproducibly. At 2.7 Mbps against a
silent-SDK baseline, the default pacing costs +59% at p95 while RARE + LARGE +
LOW costs +7%; at 0.5 Mbps, +53% against +20%. Same direction at both, which is
what the shipped tuning advice claims.
The launch-parameter plumbing defaulted every knob, and so did the code that
reads them, which left two copies to drift apart. They had: the forwarder
supplied 200 events and 'demo' pacing, the reader assumed 10 and 'off'. Since
the forwarder always wrote, its copy always won and the reader's was dead — so
omitting a parameter quietly ran a different experiment than the one the reader
described. Under the between-group semantics 200 meant 200 events every 20 ms.

Forward a parameter only when it was actually supplied, and keep the defaults
where they are read. A measurement tool that silently changes what it measures
is worse than one that fails.
@Fiona2016
Fiona2016 merged commit 9cdaba1 into main Aug 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant