permessage-deflate inflation does not enforce max_message_size

[jhugard]

Summary

When the permessage-deflate extension is enabled, websocketpp inflates compressed payloads without enforcing the configured max_message_size during decompression. As a result, a compressed frame can inflate beyond the per-message size limit before the library detects an error.

This behavior differs from the handling of uncompressed messages, where max_message_size is treated as a strict upper bound.

---

Details

websocketpp already exposes max_message_size as the primary per-message size guardrail. However, when handling compressed messages, the limit is only applied after inflation has completed.

For highly compressible inputs, this means both the decompression work and memory growth can exceed the configured budget before control returns to the caller or the processor layer rejects the message.

---

Reproduction (repo-local tests)

This can be reproduced using small extensions to the existing test suite:

1. Extension-level repro (test/extension/permessage_deflate.cpp)

• Configure permessage-deflate with max_message_size = 16 * 1024 * 1024
• Decompress a payload that inflates to exactly the limit → succeeds
• Decompress a payload that inflates to limit + 1

Current behavior:

• decompress() leaves v.ec unset
• Output buffer grows to 16,777,217 bytes instead of stopping at the configured cap

2. Processor-level repro (test/processors/hybi13.cpp)

• Configure a processor with a small max_message_size
• Feed a compressed frame whose decompressed payload exceeds that limit

Current behavior:

• The processor does not surface processor::error::message_too_big
• Message processing continues instead of rejecting the oversized message

---

Expected behavior

• Inflation should stop as soon as the configured max_message_size would be exceeded
• Oversized decompressed messages should be rejected consistently with existing message_too_big handling
• Compressed and uncompressed messages should observe the same per-message size contract

---

Actual behavior

• Decompression continues past the configured limit
• Oversized decompressed messages can be fully materialized before an error is reported
• Processor-level handling does not translate the condition into a size violation