Add a network-contention benchmark to the demo harness - #30
Merged
Conversation
The pacing settings shipped in 0.5.0 were only ever exercised against fakes. Nothing measured whether upload traffic actually delays the app's own requests, or by how much, so the tuning advice rested on the mechanism being plausible rather than on a number. Adds a `netbench` scenario to the existing e2e harness rather than a separate project: the mock intake already serves both the SDK's uploads and a stand-in origin for the app's requests, which is exactly the shared link the experiment needs. Runs are driven by `aa start --ps` and collected from HiLog, so a whole run is scriptable over hdc with no tapping, on an emulator or a real device. The link model matters and the obvious one is wrong. A token bucket sized to a second of bytes measures nothing: it starts full, so a request smaller than the bucket draws its bytes instantly and never queues. Real links have no burst credit — they serialize at the line rate, and a request arriving mid-transmission waits. That queueing is the effect under test, so the server models a link, not a quota. Throttling stays opt-in via UPLINK_BPS, leaving existing smoke runs untouched. Arms alternate round by round rather than running as two blocks, so drift over the session cancels instead of landing entirely on whichever arm ran second. Known limitation, recorded here because the numbers look usable and are not: the harness force-flushes to guarantee the upload overlaps the measured request, and a force-flush bypasses the very pacing it is meant to compare. Results are therefore valid for "does upload traffic cause contention" (it does — p95 rose 32-44% at 2.7 Mbps) and invalid for comparing BatchProcessingLevel settings. A second iteration needs organic load and no flush.
The first harness alternated uploads on/off round by round inside a single run and force-flushed each round to guarantee the upload overlapped the measured request. Both choices defeated the experiment. The flush was the direct problem: forcing an upload bypasses the batch window, the cycle interval and the per-cycle bound, so every setting the run existed to compare was inert. Load is now organic — events are generated continuously and the SDK's own scheduler decides when to upload. The alternation was the subtler one. Consent PENDING stops new drains, not an upload already in flight, so once an upload outlives its round it spills into the supposedly quiet arm and both arms carry upload traffic. At 0.5 Mbps that erased the effect completely: p95 delta fell from +32..44% to roughly zero, not because contention had gone but because the baseline had become contaminated. Groups are now whole runs, with drift handled by repetition and medians. With that, the settings separate cleanly and reproducibly. At 2.7 Mbps against a silent-SDK baseline, the default pacing costs +59% at p95 while RARE + LARGE + LOW costs +7%; at 0.5 Mbps, +53% against +20%. Same direction at both, which is what the shipped tuning advice claims.
The launch-parameter plumbing defaulted every knob, and so did the code that reads them, which left two copies to drift apart. They had: the forwarder supplied 200 events and 'demo' pacing, the reader assumed 10 and 'off'. Since the forwarder always wrote, its copy always won and the reader's was dead — so omitting a parameter quietly ran a different experiment than the one the reader described. Under the between-group semantics 200 meant 200 events every 20 ms. Forward a parameter only when it was actually supplied, and keep the defaults where they are read. A measurement tool that silently changes what it measures is worse than one that fails.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
0.5.0 shipped three pacing settings and advice on when to reach for them, but
nothing measured whether upload traffic actually delays an app's own requests, or
by how much. The advice rested on the mechanism being plausible.
It holds up. Against a silent-SDK baseline at 2.7 Mbps, the default pacing costs
+59% at p95, and
RARE+LARGE+LOWcosts +7%.Cross-checked at 0.5 Mbps: +53.1% against +20.2% at p95. Same direction, and the
effect sits in the tail rather than the mean — the shape reported for the Android
investigation.
What
A
netbenchscenario on the existing e2e harness rather than a separate project:the mock intake already serves both the SDK's uploads and a stand-in origin for
the app's requests, which is exactly the shared link the experiment needs. Runs
are driven by
aa start --psand collected from HiLog, so a full run isscriptable over hdc — emulator or real device, no tapping.
Throttling is opt-in via
UPLINK_BPS, so existing smoke runs are untouched.Notes for review
The server models a link, not a quota. A token bucket sized to a second of
bytes was the first attempt and it measured nothing: the bucket starts full, so a
request smaller than the bucket draws its bytes instantly and never queues. Real
links have no burst credit — they serialize at the line rate, and a request
arriving mid-transmission waits. That queueing is the effect under test.
Groups are whole runs, not alternating arms. Alternating uploads on/off
inside one run reads as the stronger design, but consent PENDING stops new drains
rather than an upload already in flight. Once an upload outlives its round it
spills into the supposedly quiet arm and both arms carry traffic. At 0.5 Mbps
that erased the effect entirely — not because contention had gone, but because
the baseline was contaminated. Drift is handled by repetition and medians.
No
flush(). Forcing an upload bypasses the batch window, the cycle intervaland the per-cycle bound — every setting the run exists to compare. Load is
organic and the SDK's own scheduler decides when to upload.
Verification
./scripts/ci-check.sh— ALL CHECKS PASSED (148 tests, HAR build, demo HAPbuild, ArkTS unit-test type check).
low 58/66.
relative ordering faithfully, but its queue is cleaner than a real radio, so
the absolute figures should not be quoted externally without a device run.