BIP85: Add Codex32 as application 93' - #1958
Conversation
jonatack
added
Proposed BIP modification
BenWestgate
changed the title
BenWestgate
changed the title
|
Documenting recent discussions: |
It seems you'd like to consolidate some of the paths. There's a few ways to do this, if you have a favorite or one that immediately stands out as obvious let me know. I'm thinking the identifier could be the bech32 encoding of the bip85 index, as the purpose of incrementing the index is to get new seeds, and BIP93 says "...the identifier SHOULD be distinct for all master seeds the user may need to disambiguate." index = 0 -> identifier = qqqq, index = 1 -> identifier qqqp, and so on. A particular identifier could be selected by converting it to an integer {index} once index reaches 32^4, it can fall back to the default BIP-0032 fingerprint or roll over. On byte extraction: I agree we should draw If we output share indices still can use the current read one byte at a time method. |
jonatack
left a comment
jonatack
left a comment
There was a problem hiding this comment.
Pinging @scgbckbone (who has been active on BIP85 review) for feedback.
|
Seems to me this is well over the BIP-85 application scope. As I understand it, BIP85 generates "a thing" from "a thing". Your application is generating "multiple things" from "a thing". Why are you generating multiple initial shares via BIP85 ? What I imagined BIP85 application should looks like after reading BIP93:
** maybe even threshold should be part of the BIP32 derivation path, BUT I think not as it has no effect to the actual secret generated (it only affects checksum) Assuming I'm not wrong in my "specualtion", why not just use m/83696968'/128169'/{num_bytes}'/{index}' to generate deterministic bytes from BIP-32 root seed for any share ? |
jonatack
added
the
PR Author action required
This comment was marked as outdated.
This comment was marked as outdated.
jonatack
removed
the
PR Author action required
agreed, rest my case here...
this is bad comparison, as 12/24 words represent encoding of 16/32 bytes of entropy. While your approach creates multiple shares. Same as if I would create multiple 12 words seeds from 16 bytes of entropy. this is imo ok (using code from your snippets whithout ever running it or reviewing it). My understanding is that each line in below snippet, generates just one share? with what I have issue is this, where you just generating multiple shares (somehow):
only Don't get me wrong, I'm not intending to block this BIP update. Updated version is much better. I'm only trying to figure out why is this needed & whether there is any advantage in what you're doing vs. what I'm doing. Here is my pseudo-code, to try to prove the point that nothing else than simple "one share generation" is needed here & rest can be left to BIP-93 interpolation: Above pseudo-code always generate the same shares. |
This comment was marked as outdated.
This comment was marked as outdated.
|
@akarve thoughts here? (thanks!) |
|
Yeah. Not to slow down the innovation here (and in @3rdIteration 's PRs) but my thinking is for me take on implementing both applications in the current reference implementation as Python protocols. My belief is that if we can come up with a standard duck typing interface for all BIP-85 applications that this—as yet non-existent—abstraction will stand the test of time. Of course anyone else in this thread is free to propose the shape of the protocol and even implement it without me. My experience is that we will need full unit tests and such for the protocol to be hard enough to stand the test of time. I'm volunteering to complete said protocol + implementation this quarter but if anyone wants to go first down this path they're welcome. Protocol designs can go in this thread. The ultimate goal is we have a standard protocol/interface for BIP-85 applications and a standard "graph" that they all go through and boom out come the entropy products. In this way all applications use the same protocol and same core logic. This would benefit current and future applications as well as benefit PR-clearing speed (after the initial investment) once the protocol is in place. Said protocol will also resolve the "one-product/two-product" style debates happening in this PR because the quacking of the duck types will resolve such points as how many things turn into how many other things. I hope that makes sense. |
This comment was marked as off-topic.
This comment was marked as off-topic.
After Heavy thinking, I concur the best solution is to generate single codex32 strings in a manner very similar to what @scgbckbone proposed. I have rewritten the BIP text to match this design, it is a fraction of the original's length so that's another win. I look forward to updated feedback and will be re-writing the reference implementation in the coming weeks. The use-case that sold me on single share outputs was:
By limiting the shares to the first threshold indicies and including the identifier in share entropy derivation, it avoids the issue of there being multiple seeds recoverable at a given threshold and identifier, at least from a single bip85 root key. |
There was a problem hiding this comment.
much better!
few more questions/comments:
what are CANONICAL_INDICES ? it seems that you limit maximum number of shares to 10 by using them ?
The identifier SHOULD default to the 4 left-most characters of the Bech32-encoded BIP-0032 fingerprint derived from the master seed.
why? imo this should be left out from this spec and left to specific implementations to decide
Users should enter a unique identifier instead of incrementing {index} as different share sets SHOULD have unique identifiers.
I agree here & thinking that maybe we should remove index completely...
What do you think about below derivation path calculation:
CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l"
VALID_HRP = {
"ms": 0,
"cl": 1,
}
def pack_header(threshold: int, identifier: str, share_idx: str) -> int:
id_a = CHARSET.index(identifier[0])
id_b = CHARSET.index(identifier[1])
id_c = CHARSET.index(identifier[2])
id_d = CHARSET.index(identifier[3])
idx = CHARSET.index(share_idx)
return (threshold << 25) | (id_a << 20) | (id_b << 15) | (id_c << 10) | (id_d << 5) | idx
def unpack_header(n: int):
threshold = (n >> 25) & 31
id_a = (n >> 20) & 31
id_b = (n >> 15) & 31
id_c = (n >> 10) & 31
id_d = (n >> 5) & 31
idx = n & 31
identifier = ""
for i in [id_a, id_b, id_c, id_d]:
identifier += CHARSET[i]
return threshold, identifier, CHARSET[idx]
def make_path(hrp, byte_len, threshold, identifier, share_idx, ):
header = pack_header(threshold, identifier, share_idx)
pth = f"m/83696968h/93h/{VALID_HRP[hrp]}h/{byte_len}h/{header}h"
return pth
b85_path = make_path("ms", 16, 9, "llll", "l")
print(b85_path)
# m/83696968h/93h/0h/16h/335544319h
print(unpack_header(int(b85_path.split("/")[-1][:-1])))
# (9, 'llll', 'l')- removed idx (even tho I have second thoughts, as it is now very different than other BIP-85 apps, but maybe no care as it is already very different as with most other app you only choose index and nothing else. Here, on the other hand, there is plenty to specify)
- added HRP into the path directly - useful imo - users can see immediately what is it for
|
Quoting BIP93: For a fresh master seed to justify my answers:
I renamed that constant to: It is the secret index "s" followed by the 31 share indices (excluding 's') alphabetically sorted. Prepending "s" makes
BIP93 prescribes using the first k alphabetical share indices for "random initial shares" (our bip85 entropy application). BIP85 deriving share indices beyond this initial k makes recovery ambiguous where knowing the common share header of a backup ( Example: Breaking BIP85's {index} customary rule:
And BIP93's identifier rule:
If The additional 31-k shares beyond the first k may be derived using BIP93 interpolation, which is outside spec as it needs no fresh randomness. Outputting interpolated shares would be like our XPRV application producing an extended private key encoding of HMAC-SHA512("Bitcoin seed", S) where S is the payload of our WIF application. We don't do that, we always generate fresh randomness or nothing at all. And I've proven the problem with generating fresh randomness for shares beyond the first k alphabetical indices.
My first thought was "OH, that's good!", I like this derivation path, provided we add the However, Are you sure you want to eliminate the bip85 {index}? All deployed BIP85 applications use this derivation path feature and the reference implementation expects one:
It's recommended headers be distinct for unique seeds, but not required as with no {child index} derivation.
Identifier needs an assignment to encode the string so it may as well be the most useful deterministic data. If the user can specify and we keep a child {index}, we break the "SHOULD be distinct for every seed" rule. I resorted to the fingerprint default based on an assumption bip85 applications MUST have a child index for "millions of unique" "fresh secrets" as per quotes above. I dislike the least significant bits of {index}' being Retaining child index and using a default fingerprint identifier for unshared secrets makes them most like other bip85 apps. It's not required though, we could derive unshared secrets like shares by serializing the identifier into the child index, subject to the restriction k must be 0 if This version always requires the specifying the identifier even for secrets: IDX_ORDER = "sacdefghjk"
VALID_HRP = {
"ms": 0,
"cl": 1
}
def bip93_parameters_to_path(hrp, threshold=0, identifier="", share_idx, byte_len, index=0):
if threshold and share_idx not in IDX_ORDER[1:threshold+1]:
raise InvalidShareIndex()
if not identifier:
raise MissingIdentifier()
if index > 31:
raise ValueError("maximum index 31 allowed for codex32 strings")
index = (
+ threshold * 2 // 2 + IDX_ORDER.index(share_idx) << 25
+ index << 20
+ int.from_bytes(bech32_decode(identifier), 'big')
)
return f"m/83696968'/93'/{VALID_HRP[hrp]}'/{byte_len}'/{index}'"This allows 32 child index values which combined with a million+ identifiers meets the existing "millions" spec. Incrementing this level changes the identifier as expected by BIP93 for a unique secret. I still prefer combining def bip93_parameters_to_path(hrp, k=0, identifier="", share_idx, byte_len, index=0):
if share_idx not in IDX_ORDER[min(1, k) : k + 1]:
raise InvalidShareIndex()
header_path_int = VALID_HRP[hrp] * 100 + k * 10 + IDX_ORDER.index(share_idx)
if share_idx != "s":
if not identifier:
raise MissingIdentifier()
if index > 2**11 - 1:
raise ValueError("maximum index 2047 allowed for codex32 shares")
index = index << 20 + from_bytes(bech32_decode(identifier), 'big')
return f"m/83696968'/93'/{header_path_int}'/{byte_len}'/{index}'"What do you think? |
completely missed that one, thanks for the reminder + examples
no, I'm not. Completely unsure about it tbh. I've got second thought while writing it & even more after I published. Now I think we should just have a proper index ((2**31) -1) at the end.
I like the idea of using master pubkey fingerprint, but is it even possible for bech32 charset where we do not have number
I your encoding two-way? can you get back to header used from int ? |
We have index 0 through 2**32-1 for secrets (k= 0, In codex32 and bip85, For identifier collision resistance, Since only
We can bech32-encode the 20-MSB of master pubkey fingerprint into the 4-character identifier instead of 16-bits if we mapped hex characters, which we can't since
Agreed, this simplifies implementations assumptions about path lengths.
There are 45
Here is an example path encoding that uses the 20-LSB of the identifier parameter as the 20-LSB of the child index derivation level, and left-shifts any optional "index" parameter 20-bits to not mangle the identifier part.
Yes. CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l"
IDX_ORDER = sorted(CHARSET, key=lambda c: (c != 's', c.isdigit(), c)) # "s" then alphabetical order
VALID_HRP = ["ms", "cl"]
def bip93_parameters_to_path(hrp, k, share_idx, byte_len, ident="", index=0):
if k == 1 or not (0 <= k <= 9):
raise ValueError("Invalid threshold parameter")
if share_idx not in IDX_ORDER[min(1, k) : k + 1]:
raise ValueError("Invalid share index")
if len(ident) != 4:
raise ValueError("Missing unique 4-character bech32 identifier")
if index > 2**11 - 1:
raise ValueError("maximum index 2047 allowed for codex32 shares")
header_path_int = VALID_HRP.index(hrp) * 100 + k * 10 + IDX_ORDER.index(share_idx)
for char in ident:
index = index << 5 | CHARSET.find(char)
return f"m/83696968'/93'/{header_path_int}'/{byte_len}'/{index}'"
def bip93_path_to_parameters(path=''):
header_int, byte_len, index = [int(segment[:-1]) for segment in path.split("/")[3:]]
hrp = VALID_HRP[header_int // 100]
k = header_int // 10 % 10
share_idx = IDX_ORDER[header_int % 10]
ident = "".join([CHARSET[(index >> s) & 31] for s in (15, 10, 5, 0)])
return hrp, k, share_idx, byte_len, ident, index >> 20
I left out the custom rule for Here are some test vectors: assert bip93_path_to_parameters("m/83696968'/93'/0'/16'/0'") == ('ms', 0, 's', 16, 'qqqq', 0)
assert bip93_path_to_parameters("m/83696968'/93'/199'/32'/999999999'") == ('cl', 9, 'k', 32, '4j0l', 953)
assert bip93_path_to_parameters("m/83696968'/93'/199'/64'/2147483647'") == ('cl', 9, 'k', 64, 'llll', 2047)
assert bip93_parameters_to_path("ms", 0, "s", 16, 'test', 0) == "m/83696968'/93'/0'/16'/386571'"
assert bip93_parameters_to_path("ms", 3, "a", 16, 'cash', 0) == "m/83696968'/93'/31'/16'/816663'"
assert bip93_parameters_to_path("ms", 3, "c", 16, 'cash', 0) == "m/83696968'/93'/32'/16'/816663'"
assert bip93_parameters_to_path("ms", 2, "a", 16, 'qqqq', 0) == "m/83696968'/93'/21'/16'/0'"
assert bip93_parameters_to_path("ms", 2, "c", 32, 'qqqp', 0) == "m/83696968'/93'/22'/32'/1'"
assert bip93_parameters_to_path("ms", 3, "a", 16, 'llll', 0) == "m/83696968'/93'/31'/16'/1048575'"
assert bip93_parameters_to_path("ms", 3, "c", 64, 'qqqq', 1) == "m/83696968'/93'/32'/64'/1048576'"
assert bip93_parameters_to_path("ms", 3, "d", 16, 'qqqp', 1) == "m/83696968'/93'/33'/16'/1048577'"This If we cram identifier into
They're not the same thing but they're close enough we should handle them together for derivation purposes. |
I'm planning to extende the BIP(s) with new HRP, that will encode chaincode+privkey (64bytes) for compatibility with BIP-39. I will use master fingerprint 20 MSB for it in my application as default. Users will have ability to change to custom ID. So up to you. Here is my xfp to codex32 id converter: c = "qpzry9x8gf2tvdw0s3jn54khce6mua7l"
def xfp_to_codex32_id(xfp):
x = (int(xfp, 16) >> 12) & 0xFFFFF # Extract exactly 20 MSB
return c[(x >> 15) & 31] + c[(x >> 10) & 31] + c[(x >> 5) & 31] + c[x & 31]ok, I'm sold. You definitely put much more thought into this than I did. Thanks for lengthy & helpful explanations! Concept ACK |
This comment was marked as off-topic.
This comment was marked as off-topic.
I doubt anyone will start using bech32 encoded extended keys, at least I do not plan to, even tho better readability than base58, that standard is set in stone at this point. If you consider it useful, I do not mind if you optimize this BIP-85 app for it. |
This comment has been minimized.
This comment has been minimized.
you're probably right that it make sense to attempt it all the way to proper extended key encoding, but no I wasn't treating it as such, I was just storing secret that consist from chaincode + privkey from which naked/root (without meta) extended private key can be re-assembled
do you have link ? |
This comment has been minimized.
This comment has been minimized.
|
@BenWestgate howdy and happy holidays. thanks for all of your input so far. i did manage to get an app protocol together so that you and others can hopefully add bip-85 apps to the reference implementation easily: i'm sure it needs some improvements as i rushed it but again should be a big step up. lmk if it works for your app and if you desire any protocol changes. https://github.com/akarve/bipsea/blob/protocol/src/bipsea/app_protocol.py |
|
@akarve, @BenWestgate: What’s the status on this one? It sounds to me like you have made progress, but it’s not clear to me whether this is ready to be merged. |
This comment was marked as resolved.
This comment was marked as resolved.
1 similar comment
This comment was marked as duplicate.
This comment was marked as duplicate.
Kinda looks to be moving now though?
Can you link to the above comment so I can read it in context?
Apologies you've probably already done this but can you show how the respective BIP-85 derivations would differ and what the tradeoff is?
Same request as above. Linking to existing comments is fine. From my side I did the "app refactor" but didn't push it, I aim to complete that this month so you can submit a codex app to the ref. implementation. |
|
@BenWestgate app refactor is in the reference implementation is live. Lmk if a friendly ping to |
|
I'm happy to rereview this but @BenWestgate can you clean up the git history? It's hard for me to follow the series of merges and understand what's actually changing. (I also think that the merge policies of this repo want you to have a linear history.) With #2040 it looks like there is still some back-and-forth between you and Russell so I will let that settle before re-reviewing. It may make sense to just squash everything into one commit and rebase on master. |
jonatack
added
the
PR Author action required
|
I think this is in your court at this time, @BenWestgate. @apoelstra, I think the summary of changes across all commits is fairly readable (https://github.com/bitcoin/bips/pull/1958/changes), but I agree that this might be more accessible for another round of review if it were squashed and rebased. |
|
This PR was based on #2040 which was closed after six months without activity. I assume that this PR will therefore also not continue to make progress. I will close this on 2026-07-30 or later, unless I hear otherwise. |
|
ACK |
This comment has been minimized.
This comment has been minimized.
Define a deterministic BIP85 derivation for canonical initial codex32 strings as specified by BIP93. Serialize the six-character codex32 header into the final hardened path element so each valid path identifies one canonical output for a fixed root key. Reserve profiles for future secret formats and define profile 0 for BIP32 master seeds using the ms HRP. Keep byte length and zero-padding rules specific to master seed construction while making random share payload construction common to assigned profiles. Include reproducible vectors for an unshared master seed and a threshold-2 initial basis. Organize the specification around master seeds and shares so the table of contents exposes the two output concepts while each example follows the existing BIP85 INPUT/OUTPUT style.
|
@apoelstra @murchandamus I squashed and rebased and updated PR description. This no longer depends on #2040 as I only defined profile @akarve I updated the bipsea reference implementation to match the new BIP text. The questions I asked you are resolved. @scgbckbone Share payload characters must be sampled uniformly from the BIP85 DRNG. I replaced CRC-padding with zero-padding on master seed bytes on Requesting re-review at 120c923. |
jonatack
removed
the
PR Author action required
6 participants
This allows wallets to derive codex32 secrets and codex32 shares from BIP-0032 root keys.
Summary of changes
93'2.2.0Rationale
Specification
0uses BIP93'smshuman-readable part and Master seed format0derives secrets according to profile specific constructionTests
Reference tests included in the reference bipsea implementation:
https://github.com/akarve/bipsea/pull/68/changes
Mailing List
Discussion: https://groups.google.com/g/bitcoindev/c/--lHTAtq0Qc
Status
Ready for renewed conceptual and approach review. This change is additive and does not modify existing BIP-85 behavior.