Experiment with using construct for packet parsing

[flip-dots]

This PR aims to implement dynamic negotiation, this means dynamically generating a public/private key, timestamps, and time-zones instead of using statically defined ones, while this could be implemented without overhauling how packets are encoded and decoded, I think its about time to switch to a more concrete approach.

As it stands this PR currently contains my experiments with using construct to replace the previous packet decoding system and from this it seems that construct is definitely the right way to go, future work will be done in this PR to switch the rest of the code over and then implement the dynamic negotiation features.

That being said I think I will leave device._data alone for now, since changing that would require updating all of the devices and I have created a method to convert the new structures to the legacy ones.

[kb1ibt]

Since this PR is rebuilding the packet layer, here are three findings from our capture work that bear directly on the design while it's still cheap to change. All of it comes from four PacketLogger captures (A91B2 / A2345 / C2000 G2) cross-checked against the iOS app's own debug logs, which conveniently log both the session key and the pre-encryption plaintext of every frame.

1. The device tells you which negotiation path it speaks — before you connect

The manufacturer data in the advert (company id 0xffff) is:

fmt(1) │ MAC(6) │ 00 │ ?? │ model?(1) │ SKU (3-4 ASCII) │ capability(1)

device	fmt	SKU	capability	negotiation path
A91B2 station	01	JTB	00	cleartext 0xxx
A2345 charger	01	QJB	04	encrypted 4xxx
C2000 G2 (A1783)	02	DKKE	04	encrypted 4xxx

The capability byte is an 8-bit mask (0x04 = bit 2), and it is exactly what the app uses to choose the path, before sending any frame:

IotAnkerDevice:onMessageDispatch {bleMac: …, data: {type: 1, capability: 4}}
capability === capability: 4 ---- abilityList:[0,0,1,0,0,0,0,0] --- type：1
设备的鉴权方式 === true            ("device auth method")
⭐️ 开始握手协商                    (start handshake negotiation)
更新aeskey ------- b8ff7422955d4eb6d554a2c470280559

That last line is byte-identical to prime_device.NEGOTIATION_KEY, so bit 2 simply means "this device speaks the static-key encrypted handshake."

The device then repeats the same fact on the wire in its stage-2 response, so a client never strictly needs the advert:

field	app's name	values
a1	stage	02
a2	mtu	253 (fd00) on A1783/A91B2/A2345 · 297 (2901) on a 160W
a3	encryptMethod	0x44 = ECDH · 0x04 = legacy static-key
a4	—	always 01
a5	authMethod	00 = cleartext path · 02 = encrypted path · absent on older firmware

The payload is otherwise byte-identical across families — a5 is the only varying byte:

A91B2      00 a10102 a202fd00 a30144 a40101 a50100
A2345      00 a10102 a202fd00 a30144 a40101 a50102
C2000 G2   00 a10102 a202fd00 a30144 a40101 a50102
160W       00 a10102 a202 2901 a30144 a40101 a50102

The practical upshot: the SolixBLEDevice vs PrimeDevice split is a runtime-detectable property rather than a per-model taxonomy decision. You've asked a few times on the stacked PRs whether a given device could just use the base class — for the A91B2 the answer is yes, and the device says so itself with capability 00. discover_devices() already receives this in its callback's advertising_data and currently discards everything but the service UUID.

One caveat worth knowing: the app reports capability per type, not once per device. The same MAC gives capability: 4 at type 1/2 and capability: 0 at type 3/5/6/7; only type: 1 drives the auth decision. Also note 0x4 on an opcode means "encrypted", not "AES-GCM" — the A91B2 sends 4022/4023 encrypted with CBC, so cipher can't be inferred from the opcode.

2. _MAX_PACKET_SIZE = 253 — the device declares this too

The gate in _process_notification is the right idea, and it's the same conclusion we reached in #48. But the constant is the problem, in both directions:

• Hardcoded 253 breaks the F2000 at MTU 247 that you flagged on feat(ble): reassemble multi-fragment frames for every session command (#42) #48 — the cap there is 244, so len(data) == 253 never fires and fragments get treated as complete payloads.
• Our own mtu_size - 3 in feat(ble): reassemble multi-fragment frames for every session command (#42) #48 is no better. bleak's BlueZ backend returns a hardcoded 23 unless _acquire_mtu() has been called, with UserWarning: Using default MTU value. We observed this live on a Pi Zero 2W. Reassembly still works because the comparison is >=, but the guard silently degrades to a no-op on Linux — which is what Home Assistant runs on.

The stage-2 a2 field above is the fix: the device declares its own MTU, it agrees with ATT_MTU - 3 in every observation we have, and it needs no _acquire_mtu() call on any platform. Something like "prefer the stage-2 a2, fall back to mtu_size - 3" is correct everywhere.

Worth noting for the fragmentation logic generally: not every device fragments. Across four captures the A91B2's largest frame is a single unfragmented 250-byte 4a00 whose first byte is ciphertext (0xd8, which would decode as the impossible "fragment 13 of 8"), and its 4303 stream is ~90 B. So the threshold has to be compared against the cap, not against "the largest frame this device has sent."

3. The type byte in Parameter isn't length-derivable

"type" / If(lambda p: ... p.length > 1, Int8ul),

Tags are ordinal per opcode rather than a global vocabulary — the same tag means unrelated things in different messages (a3 is encryptMethod in 0803, module firmware in 0829, and the model string in 0830; the module firmware itself moves from a3 to a1 between those two). Whether a type byte is present follows the same rule: it's a property of (msgtype, field index).

Concretely, the 40-character owner id appears both ways. a2 29 04 <40 bytes> is length 0x29 with a type byte 04, and parses correctly under the current rule. But a2 28 <40 bytes> — the form in the 0023/0027 binds — has no type byte, so it gets split as type=0x61 (the ASCII a) plus a 39-byte value. to_legacy() re-prepends the type byte so nothing breaks downstream today, but anyone reading the new-style .value gets a truncated id.

Same family of issue, on the encode side: the confer currently sends hardcoded lengths and a frozen a5 timezone string. POSIX TZ strings vary in length (GMT0BST,M3.5.0/1,M10.5.0 is 24, CET-1CEST,M3.5.0,M10.5.0/3 is 26, JST-9 is 5), and serials are 16 or 17 bytes across our own three devices, so a3 10 for the serial is wrong for some units. Every length wants deriving from its value.

On dynamic negotiation

Since that's the stated goal here — we have that working and hardware-validated on a C2000 G2 and a hardened A2345, currently sitting in #49: live timestamps in each negotiation frame, and the stage-5 confer carrying the host's real POSIX TZ string read from the system zoneinfo (with the a5 length derived). Newer firmware rejects a stale timestamp and drops the link mid-handshake, which is the disconnect in #22.

Two things we found on that path that are easy to miss:

• 4022.a3 is a UTC offset in seconds, 4-byte LE — 40380000 = 14400 = 4 h, matching EST5EDT in July. The app recomputes it from the OS every time, so it has to track DST rather than being frozen. We have no capture from an east-of-UTC device, so whether it's signed is unverified.
• The native client's 0005 echoes back the device's declared values (a4 = the mtu from 0803's a2, a5 = encryptMethod, a6 = authMethod), confirming the negotiated capability. The current frozen constant replays the 0003 proposal values instead (a4=00f0, a5=0x40, no a6), which every device tested tolerates — but it means a device declaring a non-default mtu, like the 160W's 297, would be ignored.

Happy to rework any of this to fit whatever shape you land on here, or to just hand over the findings if you'd rather implement it yourself — you're the one who has to maintain it. If the reassembly work in #48 is useful as-is it's rebased on current main and green, but I'm equally happy to close it if this PR supersedes it.

[flip-dots]

@kb1ibt I appreciate your input on this, I have not tested your solution to the fragmentation problem yet but its something I plan on looking into. With the type thing I have not decided what to do about it yet, my current plan is to make it so that parsed packets do not use it (it all goes to value) but you can build a packet using it and its automatically added. The information about the supported negotiation paths is interesting, but I don't think its worth implementing yet.

In terms of #48 I plan on merging this instead, my prior implementation of packet parsing and building left a lot to be desired as it was created without much understanding of the protocol, so rather than build upon that mess I made this to replace it and then implement more dynamic features like key generation, timestamps, and timezones on top of it, which should also make it easier to add in support for devices using more exotic negotiations (like ones which need the cloud-id).

[kb1ibt]

Since this replaces the packet layer wholesale, one structural finding is worth having before the schemas harden — it's the thing most likely to force a rework later.

TLV field tags are ordinal per opcode, not a global vocabulary. A tag is "the Nth field of this message", and its meaning is only defined relative to the opcode. The same tag means unrelated things in different messages:

field	0003 stage-2	0029 stage-3	0030 version info
a1	02 — stage no.	03 — stage no.	v0.2.9.7 — module fw
a2	fd00 — mtu	ESP32 — chip	v2.1.1.6 — device fw
a3	44 — encryptMethod	0.0.0.3 — module fw	A2345 — model
a4	01	serial (ASCII)	A2345_mcu
a5	00/02 — authMethod	MAC (6 B)	A2345_esp32

Note the module firmware moves from a3 in 0029 to a1 in 0030. So a construct schema keyed on tag → meaning globally will decode some messages into confidently wrong fields; the key has to be (msgtype, field index).

Two corollaries that bite in practice:

• There is no "a228 tag." That notation is really field 2, length 0x28 (=40). Likewise a224 = field 2 / len 36 (a dashed UUID), a516 = field 5 / len 22 (a POSIX TZ string), a140 = field 1 / len 64 (an uncompressed P-256 point minus its 04 prefix). Any constant of that shape in the current code is a length that must be derived from the value, not a tag. a5 16 breaks the moment someone is in a zone whose POSIX string isn't 22 bytes (CET-1CEST,M3.5.0,M10.5.0/3 is 26, JST-9 is 5), and a4 10/a3 10 for the serial breaks on the 17-byte serials some units carry.
• Strip the GCM tag before parsing. The payloads in Anker Prime 250W "Unexpected end of packet", sensors unavailable #17 parse cleanly as TLVs and then hit exactly 16 trailing bytes — the AES-GCM auth tag, decrypted along with the body. The walker reads the tag's first byte as a tag/length pair and pops an empty bytearray, which is the Unexpected end of packet / IndexError: pop from empty bytearray in that issue. Worth encoding structurally in the new parser so it can't recur.

All of the above is from wire captures (PacketLogger + the app's own debug logs, cross-checked against live devices) — happy to paste the specific frames for any of it if that's useful while you're shaping the schemas.

[kb1ibt]

Two more things for the packet layer, both from live A2345 captures this week.

There is a third frame pattern: 030101. The library currently knows 030001 (negotiation) and 03010f (session/telemetry), and logs anything else as

Unexpected packet type b'\x03\x01\x01' sent by device!

…then discards it. On an A2345 that discarded frame is the grant that arms telemetry: after a 4027 registration the device answers 09 with field 1 = 1e00 (30) meaning awaiting physical confirmation, and when the operator presses the unit's side button it pushes an unsolicited 4827 00 on 030101 — not a reply to anything, and the only signal that the client is now authorized. Telemetry starts immediately after. So 030101 needs to be a first-class pattern in the new parser, and the frame it carries needs to reach the device layer rather than being dropped at the packet layer. Details and the full sequence are in the #50 thread.

Timezone/timestamp specifics, since dynamic generation is this PR's goal:

• 4022's field 3 is a UTC offset in seconds, 4 B LE — 40380000 = 14400 = 4 h, matching EST5EDT in July. The app recomputes it from the OS on every connection, so it must be DST-aware rather than a constant. Open question I could not resolve: the observed value is positive for a west-of-UTC zone (POSIX "seconds west"), and I have no capture from an east-of-UTC device, so whether the field is signed two's complement is unverified — worth a guard if anyone in CET/JST tests it.
• Field 5 of the same message is the POSIX TZ string, whose length varies by zone (see the previous comment) — derive it.
• Every session body carries the live session timestamp; a stale one is rejected as a replay and the device then acks but never streams. That failure is silent, which makes it easy to misattribute to the telemetry path.

On fragmentation, in case the rewrite touches the cap: the device declares its own frame cap as field 2 of the stage-2 response (253 on A1783/A91B2/A2345, 297 on a 160 W), and that value agrees with ATT_MTU − 3 in all of the packet captures I have. Reading the declared value is the more self-contained of the two, and avoids client.mtu_size, which on BlueZ returns a hardcoded 23 with UserWarning: Using default MTU value unless _acquire_mtu() has been called — we hit that live on a Pi Zero 2 W.

[kb1ibt]

One more structure worth pulling into construct while the parsing layer is being rebuilt: the advertisement manufacturer record (company id 0xffff). It's fixed-shape with one variable-length field, and it's fully decoded now:

fmt(1) │ MAC(6) │ 00 │ [8](1) │ [9](1) │ SKU (3–4 ASCII) │ capability(1)

device	advert name	fmt	MAC	[8]	[9]	SKU	capability
A91B2 station	A91B2_19B2	01	f49d8a2519b2	b4	01	JTB	00
A2345 charger	A2345_05F0	01	f49d8a2f05f0	b4	02	QJB	04
A2345 (sealed)	A2345_CDAA	01	007f1d2ecdaa	b4	02	QJB	04
A2345 (sealed)	A2345_E79E	01	007f1d44e79e	b4	02	QJB	04
C2000 G2 (A1783)	SOLIX C2000 Gen 2	02	7ce91346c50c	b1	1a	DKKE	04

fmt selects the record layout — 01 → 14 B total with a 3-char SKU, 02 → 15 B with a 4-char SKU — and capability is always the last byte, so parse it length-relative rather than at a fixed offset. Invalidated by #63

What fmt and [8] actually denote is unresolved, and my samples can't settle it. Every 01 I have is a Prime-line device and a charger (A91B2 station, A2345 ×4); my only 02 is a C-series device and a power station (C2000 G2). So "brand line" and "device class" fit the data identically — 01=Prime/02=C-series and 01=charger/02=PPS are indistinguishable here, and the same confound applies to [8] (b4 vs b1). Anything that breaks the pattern would settle it in one scan: a Solix-branded charger, a Prime-line power bank, or any of the C300/C800/F-series/Solarbank units. If you or anyone on the issues has one in range, the raw 0xffff bytes would be enough. Until then I'd branch the parser on the layout (which is certain) and leave the semantics unnamed.

Three things fall out that the library could use regardless:

• The real MAC is available pre-connection. On macOS CoreBluetooth only exposes a rotating per-host UUID, so this is the only way to identify a specific unit before connecting — no negotiation needed.

• capability declares what the device supports, readable before a single frame is sent — and 04 means the fully encrypted 4xxx negotiation is available on that unit. This is the one I'd flag hardest: the C2000 G2 advertises 04, and SolixBLE currently negotiates it in plaintext 0xxx. Both work, which is presumably why it has gone unnoticed — the app honours the capability and goes 4xxx against the same hardware. So the byte isn't "which path is in use", it's "which paths this unit will accept", and today the library is taking the weaker one on devices that would accept the encrypted one. Worth deciding deliberately rather than by default, and it's free to read at scan time. It is also intrinsic rather than provisioning state: a never-bound unit advertises the same 04 as a bound one.

  The 00 case answers two of your review questions on feat(prime): A2345 charger + A91B2 station live BLE telemetry #51 —
  "Does using a base SolixBLEDevice class work?"
  and "this looks suspiciously similar to the negotiation for the regular Solix devices".
  The A91B2 advertises 00, i.e. cleartext-only, which is the base-class path — so it
  isn't a Prime device with a Solix-looking handshake, it's a cleartext device. That suggests
  the class split wants to key on auth path rather than product line, since the advert (and
  then stage-2's authMethod field) states it outright. Full reasoning and the caveat about
  telemetry still differing: feat(prime): A2345 charger + A91B2 station live BLE telemetry #51 (comment)

• [9] is a stable model discriminator, confirmed across four A2345s (two bound, two sealed) spanning two MAC OUIs — all b4/02/QJB. Whatever [8] denotes, the ([8], [9], SKU) triple is a reliable pre-connect model identity.

That last one matters more than it looks, because device-class resolution currently keys on the advertised name, and a unit within ~5 minutes of its first factory boot has no name at all. I scanned a sealed A2345 immediately after first power-on: name=None while broadcasting a complete, well-formed 0xffff record; a later scan (nothing had connected to it in between) showed A2345_E79E. So the name is populated by the device during first boot, not set in the factory image. A client that resolves by name silently skips a brand-new device during that window — no error, no candidate — while the manufacturer record is well-formed from the very first advertisement.

Also worth noting for any MAC-prefix matching: current-production hardware is on 00:7f:1d now, so there are multiple OUIs that need to be worried about.

All five adverts above are raw passive-scan captures — I can paste the byte strings if you want them as test vectors for the parser.