feat: Add a pluggable text codec for string patterns - #246
Open
scottanderson wants to merge 1 commit into
Open
Conversation
scottanderson
force-pushed
the
encoding
branch
3 times, most recently
from
September 1, 2026 06:57
636bffa to
992c20e
Compare
WerWolv
requested changes
Sep 1, 2026
| auto size = std::min<size_t>(this->getSize(), 0x7F); | ||
| auto *evaluator = this->getEvaluator(); | ||
| const auto fullSize = this->getSize(); | ||
| auto size = std::min<size_t>(fullSize, 0x7F); |
Owner
There was a problem hiding this comment.
Is this truncating still okay to do now? We should probably try to not cut codepoints in half anymore
Author
There was a problem hiding this comment.
I attempted to improve this in the next revision
Add StringEncodeDecode: an interface with decode(), encode(), and encodeLossy(). The host application sets one on the Evaluator. With none set, a string pattern keeps its raw-byte behavior. decode() and encode() return std::optional. A nullopt result marks a byte sequence, or a character, the named encoding cannot represent. encodeLossy() never fails. It substitutes a replacement character for anything the encoding cannot represent. PatternString routes reads and writes through the codec. getValue() and getBytesOf() throw core::err::E0004 on a nullopt result. A script can catch this error with try/catch. setValueLossy() writes through encodeLossy(), and clears the cached bytes and the cached display string; getBytesOf() otherwise reflects a write only after the next pattern run. setValue() clears the cached bytes the same way, on every pattern type, not just PatternString. formatDisplayValue() decodes through decode(), throwing the same core::err::E0004 on a nullopt result. It reads a little past its display budget, and backs decode() off a few bytes at a time on failure, so a multi-byte codepoint straddling the read cutoff doesn't report the whole value as invalid. It then trims the decoded text itself, not the raw bytes, to the display budget, so a codepoint at that cutoff stays whole instead of splitting mid-sequence. getBytesOf() caps the encoded result to the pattern's own size. getEncodingName() reads the string's own encoding attribute. With none, it reads the evaluator's default encoding, set through Evaluator::setDefaultEncoding(). Add a libstd #pragma encoding. It sets the evaluator's default encoding to its value, unconditionally; the codec, not the pragma, knows which encoding names are valid. PatternWideString keeps its fixed UTF-16 behavior. The codec does not apply to it. Add PatternLanguage::clearFormatCaches(). It clears every placed pattern's cached display value, across every section.
scottanderson
force-pushed
the
encoding
branch
from
September 1, 2026 19:49
992c20e to
f502caf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
StringEncodeDecode: an interface withdecode(),encode(), andencodeLossy(). The host application sets one on the Evaluator. With none set, a string pattern keeps its raw-byte behavior.decode()andencode()returnstd::optional. Anulloptresult marks a byte sequence, or a character, the named encoding cannot represent. encodeLossy() never fails. It substitutes a replacement character for anything the encoding cannot represent.PatternStringroutes reads and writes through the codec.getValue()andgetBytesOf()throwcore::err::E0004on anulloptresult. A script can catch this error with try/catch.setValueLossy()writes throughencodeLossy(), and clears the cached bytes and the cached display string;getBytesOf()otherwise reflects a write only after the next pattern run.setValue()clears the cached bytes the same way, on every pattern type, not justPatternString.formatDisplayValue()decodes throughdecode(). It returns the raw decoded text. The host application escapes the text for display.getBytesOf()caps the encoded result to the pattern's own size.getEncodingName()reads the string's ownencodingattribute.PatternWideStringkeeps its fixed UTF-16 behavior. The codec does not apply to it.Add
PatternLanguage::clearFormatCaches(). It clears every placed pattern's cached display value, across every section.