GH-404: Clarify dictionary fallback encoding - #609
Conversation
The dictionary encoding docs currently say that writers fall back to PLAIN encoding when a dictionary gets too large. That is not always true. A writer can stop using dictionary encoding and write later data pages with another valid encoding. The actual encoding is already stored in each data page header, so readers should use that field instead of assuming the fallback is PLAIN. Update the wording to describe that behavior, and clarify that the dictionary data page layout only applies to dictionary-encoded pages.
| or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is | ||
| written first, before the data pages of the column chunk. | ||
| or number of distinct values, the writer may stop using dictionary encoding and fall back to another | ||
| valid encoding for subsequent data pages. The fallback encoding is chosen by the writer and recorded | ||
| in the data page header's `encoding` field; readers must use that field to determine whether each | ||
| data page is dictionary encoded or uses another encoding. The dictionary page is written first, | ||
| before the data pages of the column chunk. |
There was a problem hiding this comment.
The new wording implies that the first N pages of a column chunk can be dictionary-encoded and all other pages after need to be non-dictionary (specifically the "fall back to another
valid encoding for subsequent data pages"). However, IIUC it is valid to interleave non-dict- and dict-encoded pages within a column chunk as long as only one dictionary is used. If you agree, maybe simpler wording could be something like:
Within a column chunk, the dictionary page is written first, before the data pages of the column chunk. After the dictionary page, data pages can use dictionary encodings or another valid encoding for the column's data type. Dictionary and non-dictionary encoded data pages may be interleaved. Readers must use each page header's
encodingfield to determine how the page should be decoded.
2 participants
Rationale for this change
The dictionary encoding docs currently say that writers fall back to PLAIN encoding when a dictionary gets too large.
That is too narrow. Writers can stop using dictionary encoding and write later data pages with another valid encoding. The actual encoding is already stored in each data page header, so readers should use that field instead of assuming that fallback pages are PLAIN.
What changes are included in this PR?
This PR updates
Encodings.mdto clarify that dictionary fallback encoding is chosen by the writer and recorded in the data page header.It also clarifies that the documented dictionary data page layout applies only to dictionary-encoded data pages.
Do these changes have PoC implementations?
I feel no PoC implementation is needed, but please let me know if you think otherwise. This is a documentation clarification for behavior that the format already supports.
Closes #404