proof-of-concept: mm support - #254
Draft
briaguya0 wants to merge 24 commits into
Draft
Conversation
GetFileOffsetFromNodeStr gated on IsValidHex, which requires an 0x prefix and at least three characters. A bare "0" fails that, falls through to the filelist lookup, and throws for a name that was never meant to be one. IsValidOffset exists for exactly this case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven MM files under archives/ are not plain data at their DMA offset. They are containers: a table of big-endian u32 offsets followed by one independent Yaz0 stream per asset, loaded in-game by CmpDma_LoadFile. The format is CmpDma_GetFileInfo in mm/src/code/sys_cmpdma.c. Offsets in the table are relative to dataStart, and sub-file 0 is implicit. The container itself is stored uncompressed, so it carries no magic and GetCompressionType cannot see it -- torch read the raw table bytes and every texture in those files decoded to garbage. Decoding concatenates the sub-files, which is what CmpDma_LoadAllFiles builds and what the asset offsets already address, so AutoDecode indexes it exactly like any other whole-file decode. Opt-in only, via `compression: CMPDMA` in a file's :config:. A container starts with its dataStart word (0000003C for icon_item_24_static_yar), which is plausible leading data for an unrelated file, so sniffing it would be unsafe. Being explicit also keeps this additive: no existing yaml sets the key, so OoT extraction is untouched. Takes MM texture extraction from 9495/10014 to 10014/10014 byte-identical against the OTRExporter reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Majora's Mask runs on the same engine as Ocarina of Time and declares the same asset formats, so MM: type names resolve to the OoT factories. Which ones actually produce byte-identical output is now measurable per type; that is the input to deciding what moves to a shared zelda64 namespace and what needs a real MM implementation. OoTDListHelpers hardcoded OOT:ARRAY and OOT:MTX when resolving references out of a display list. That code is shared, so it now accepts either prefix. ExportMtx could not simply try each candidate: GetSafeStringByAddr throws when a node exists with a different type rather than returning nullopt, so the first probe aborted the run. It resolves the node once and checks its type against the accepted set instead. First measurements against the reference: MM:ARRAY 85/85 byte-identical MM:MTX 11/11 byte-identical GFX 610/5666 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The MM reference declares 688 paths and none of them come from XML -- Majora's Mask has no Path element, so every one is recovered from the reference archive. The OoT path factory serves them once the harness can supply num_paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MM's pathway list carries two fields OoT treats as padding. The rom struct is 8 bytes -- numPoints (u8), unk1 (s8), unk2 (s16), pointsAddr (u32) -- and OTRExporter's PathExporter.cpp writes unk1 and unk2 under a MM_RETAIL check. Torch read them as three padding bytes and wrote neither, which is why every failing path was short by an exact multiple of 3: three bytes per pathway. Both readers now carry them. The pair<uint8_t,uint32_t> the pathway list was built from could not hold them, so it becomes a Pathway struct. Two changes were needed, not one. The second only became visible after the first: OoT's alternate headers export just the first pathway of a shared list, and MM exports all of them. Instrumenting showed exactly 19 truncations against exactly 19 remaining failures. Both are gated on a new `game:` key in the rom config (OOT default, MM), declared rather than sniffed from the cartridge title -- same reasoning as the CMPDMA opt-in. Shared code branches on it instead of forking, following the kArrayTypes precedent in OoTDListHelpers. OPTH 166/688 -> 688/688 total 47622 -> 48310 pass, 1600 -> 1078 fail OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five separate divergences, each found by walking a failing asset against OTRExporter's RoomExporter switch rather than by reading ahead. SetRoomBehavior: OoT writes gameplayFlags then gameplayFlags2 whole; MM unpacks that word into the five fields it encodes, six bytes against OoT's five. This alone was the "ours is one byte short" that every one of the 414 failing rooms showed. Room names: MM zero-pads the index, _room_00 not _room_0, which is also how its XML names the files. 0x19: MM has SetWorldMapVisited where OoT has SetCameraSettings, and MM writes no body at all. Torch was writing OoT's five bytes -- a reuse rather than an addition, so nothing warned. 0x1A/0x1B/0x1C/0x1E: SetAnimatedMaterialList, SetActorCutsceneList, SetMinimapList, SetMinimapChests do not exist in OoT. SetMinimapList has no count of its own and takes one entry per room, so SetRoomList now records the room count in the write context. SetCutscenes: MM carries a list of entries where OoT carries one pointer, and ZAPD rewrites the opcode to 0x1F when exporting it. Entries name their cutscene off the scene's base name, so an alternate header's cutscenes still belong to the scene rather than to the header. OROM 595 -> 21 failing EXTRA 132 -> 24 total 48310 -> 48884 pass OCVT rises 17 -> 294, which is not a regression: those cutscenes had no correct name before, so they were extras or missing rather than compared. They are now correctly named with OoT-format contents, which makes MM's cutscene command set the next piece of work. OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A cutscene that is declared somewhere already has a name; only undeclared ones should get a synthesized one. The MM list writer synthesized unconditionally, so 21 scenes referenced a name the reference spells differently -- gTerminaField... against Z2_00KEIKOKUCutsceneData_... Mirrors what the OoT path already did: resolve the pointer first, fall back to MakeAssetName. OROM 21 -> 13 failing EXTRA 24 -> 0 Extras reaching zero means torch no longer emits anything the reference does not have. OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Majora's Mask animated materials, which OoT has no equivalent of -- so a new MM-only factory rather than a branch inside shared code, and OoT cannot be affected by construction. Mirrors OTRExporter's TextureAnimationExporter and ZAPD's ZTextureAnimation. The entry list is 8 bytes per entry and runs until an entry whose segment is <= 0, that entry included. Five params shapes behind it: scrolling rows, three colour-change variants that size their lists by animLength or colorListCount depending on which, a texture cycle whose texture list is sized by the largest index that names it, and an empty form. Both sources are wired: XML-declared TextureAnimation elements go through the factory, and scenes discovering an animated material list now register the serialized companion rather than only writing its name. Note for anyone adding a factory here: CMake globs the source list at configure time, so a new .cpp links as an undefined vtable until cmake is re-run. OTAN 0/293 -> 293/293 OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OoT leaves an unresolvable display list as the raw G_DL with its address incremented. MM instead rewrites it to G_DL_OTR_INDEX, with the address replaced by (segment << 24) | (offset / 8) -- an index into that segment's gfx buffer. See the G_DL case in OTRExporter's DisplayListExporter, guarded on the segment not being registered. In practice this is segments 8 through 13, which are runtime-swapped and which the OoT path deliberately never tries to resolve. A zero segment offset keeps the plain opcode in both games. gsSPDisplayListOTRIndex already existed; the branch form did not, so gsSPBranchListOTRIndex is added alongside it. ODLT 438 -> 42 failing total 49805 -> 50092 pass (99.2%) OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MM's cutscene command set is separate from OoT's. Per ZCutscene::GetCommandMM everything funnels into eight shapes: a generic 8-byte entry; 0x0C-byte text, rumble, transition-general, fade-out-seq and time; a 0x30-byte actor cue (ids 100-149, 201, 450-599 and 300); and the spline group, whose header list terminates on a 0xFFFF marker and whose leading count is a byte length rather than an entry count. The output side mirrors OTRExporter_Cutscene::SaveMM. The segfault this was parked for was never in this code. While iterating, one build invocation ran with the wrong working directory and silently did nothing, so score.sh kept measuring a stale binary from an earlier broken state. The same source runs clean repeatedly once actually rebuilt, and the reader bounds checks added while hunting it never fire. They are kept anyway: they cost nothing and turn a future layout mistake into a logged skip rather than a crash. OCVT 318 -> 244 failing total 50092 -> 50166 pass OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four of the command constants were written from the enum's ordering rather than read off it, and three were wrong: CS_CMD_PLAYER_CUE 300 -> 200 (0x0C8) CS_CMD_TRANSITION_GENERAL 250 -> 155 (0x09B) CS_CMD_FADE_OUT_SEQ 254 -> 156 (0x09C) CS_CMD_TIME 255 -> 157 (0x09D) Id 200 is the player cue and carries 0x30-byte entries. Reading it as an 8-byte generic desynced the rom walk by 40 bytes per occurrence, which is where the runs of impossible "command id 0, zero entries" came from -- those were the middle of an actor cue being read as a command header. Found by decoding the reference's own tail: it had a command 200 whose entry was plainly 8 bytes where this code wanted 48, which pointed straight at the constant rather than at the layout. OCVT 244 -> 0 failing total 50166 -> 50393 pass (99.8%) OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ExportVtx looked the vertex up with GetNodeByAddr(ptr), where ptr had already been through PatchVirtualAddr. PatchVirtualAddr is ResolveVirtualAddr, and GetNodeByAddr resolves again, so a file with a `virtual:` mapping had its vram base subtracted twice. A correct address became a miss, and the miss fell through to the unresolved- virtual-segment path, which writes the OTR opcode with zeroed operands and no hash. Instrumenting said it plainly: w1=0x801BA550 resolved to ptr=0x80114A90, exactly where the vertex array is registered, followed by "direct lookup miss". The fix is to look up the unpatched address. Found only after a wrong guess -- that BaseAddress corresponds to RangeStart rather than to file offset 0, so the virtual base needed RangeStart subtracted. That broke overlays immediately and was reverted; BaseAddress is the vram address of offset 0 and RangeStart merely bounds what to extract. ODLT 6 -> 0 failing; display lists are now complete total 50432 -> 50438 pass OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rom struct is 14 bytes with a pad after type -- type @0, x @2, y @4, z @6, r @8, g @9, b @10, drawGlow @11, radius @12, per LightInfo in ZAPD's SetLightList.cpp. Torch read the fields back to back, so every field after type was shifted one byte and only 13 of the 14 were consumed. The write order was already correct, which is why the failing scenes were all exactly the right size with a smeared run of bytes. OoT is unaffected -- it shares the layout and the gate is unchanged at 35386/0, so it either never exercises a light list or exercises it identically. OROM 13 -> 0 failing total 50438 -> 50451 pass Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZAPD's surface type loop runs highestPolyType + 1 times unconditionally (ZCollision.cpp), so there is always at least one -- including when the header declares no polygons, and including when polyTypeDefAddress is null, in which case it reads from segment offset 0. Both cases occur in MM. object_fu_kaiten_Colheader_0037F8 has zero polygons and a null polyTypeDef, and the reference's surface type is the first eight bytes of the object file read as two big-endian words -- 002A0014 and 00130000, written back as data[1] then data[0]. Torch guarded on a non-empty polygon list and then on a non-null address, so it wrote an empty list and came up exactly eight bytes short. OCOL 4 -> 0 failing total 50451 -> 50455 pass OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Majora's Mask keyframe skeletons and animations, which OoT has neither of -- a new MM-only factory rather than a branch in shared code, so OoT cannot be affected by construction. Mirrors CKeyFrameExporter.cpp and ZAPD's ZCKeyFrame / ZCKeyFrameAnim. The animation is sized entirely by its skeleton, which is why the converter now carries the XML's `Skel` attribute through as skel_offset. Each limb's bit flags say which channels are animated: a set bit spends a kfNum, a clear one spends a preset value, over six bits of a byte for a standard skeleton and nine bits of a halfword for a flex one. The keyframe count is then the sum of the kfNums. Limb entries are 0xC bytes standard, 0x8 flex, and the flex form writes callbackIndex where the standard one writes a Vec3s translation. OKFA/OKFS 0/12 -> 12/12 total 50455 -> 50467 pass OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sample data buffer was sized min(0x500000, rest of rom). OoT's Audiotable fits under that; MM's is 0x548770, so everything past the cut got a header with no sample data at all -- the write is guarded on the data fitting in the buffer, and silently skipped when it does not. Instrumenting that guard printed it directly: offset=0x4FD5B0 size=0x2ACA end=0x50007A tableSize=0x500000, twenty times, for exactly the twenty failing samples. The other 662 sat below the cut. Every read into the buffer is bounds-checked, so the cap is not load bearing and the rest of the rom is safe to take. OSMP 662/682 -> 682/682 total 50467 -> 50487 pass, 4 failing OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A skeleton header's limb count and its limb table's length are separate values and can disagree. object_fsn's header claims 18 where only 17 limbs exist -- its xml comments on exactly that. ZAPD writes the header's count as-is but walks the table's, so torch now takes the table length from an optional limb_table_count while leaving the written count alone. Getting that split wrong the first time was visible immediately: applying the override to both made the length correct but wrote 17 in the header where the reference writes 18. gFsnSkel fixed; total 50487 -> 50488 pass OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An array of bare scalars. Each element repeats its own type tag before its value, which is what ArrayExporter writes -- 80 elements of (u32 type 3, u8 value) for this one, ZSCALAR_X8. The two other unsupported array kinds are deliberately not implemented; see the harness commit for why. total 50488 -> 50489 pass OoT gate: 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZAPD emits object_horse_link_child's skin-limb display list twice: once as object_horse_link_child_DL_00D500 from the XML, and again as object_horse_link_child_Skinlimb_00A138SkinLimbDL_00D500 from the skin-limb exporter. The two archive entries are byte-identical, self-hash included -- the copy carries CRC64 of the *original's* path, and the limb that references offset 0xD500 resolves to the original name, not the copy's. A plain second declaration at the same offset gets neither right: it hashes itself under its own name, and being registered later it takes over the address map, so the limb starts pointing at the copy. `duplicate_of: <path>` on a node says both. The node is skipped when the address map is populated, and the display list exporter hashes the named path instead of its own. MM 50489 -> 50490 of 50496. OoT pal_gc 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MM declares three arrays whose elements are neither vertices nor vectors: two CollisionPoly arrays in ovl_Dm_Char08 and a Pointer array in object_hanareyama_obj. OTRExporter has no writer for either kind, so it emits the array header and then one bare type word per element with no payload -- the whole body is determined by the count. SohArrayType mirrors ZAPD's ZResourceType, so CollisionPoly is 28 and Pointer 29, which is what the reference's header words say. The reference archive currently has uninitialized memory in those element words; briaguya0/OTRExporter deterministic-arrays fixes that at the source. Against a reference built with it these three match; against the current one they differ only in the words that fix zeroes. MM: all 50496 assets now generate. OoT pal_gc 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MM's sun is one 64x64 texture the rom stores in pieces, which ZAPD cannot
extract whole -- gameplay_keep.xml says exactly that above gSunSunsetTex. ZAPD
compensates by rewriting the t coordinate of every vertex the display list
named gSunDL loads (ZDisplayList.cpp, GfxdCallback_Vtx):
if (self->GetName() == "gSunDL")
vtx.t = (((vtx.t >> 5) - 1) / 2) << 5;
so the exported array deliberately does not match the rom: t 992 (31 texels)
becomes 480 (15), and 512 (16) becomes 224 (7). Reproduced verbatim, integer
division and all, behind a `sun_tc` node flag the harness sets by applying
ZAPD's own rule -- walk the display list named gSunDL, mark the arrays it
loads.
gameplay_keepVtx_07ACF8 was the last non-text mismatch. MM 50493 -> 50494 of
50496. OoT pal_gc 35386 matching, 0 mismatched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MM:TEXT was registered to OoTTextFactory, which writes OoT's format -- hence 86637 bytes against the reference's 448796. MM's differs throughout: the table entry holds the message offset at +4 with the segment in the top byte, each message carries its own 11-byte header (textboxType, textboxYPos, icon, nextMessageID, firstItemCost, secondItemCost), the terminator is 0xBF rather than 0x02, and the codes taking arguments are a different set (0x14 one byte; 0x1B-0x1F two). Layout and control codes from ZAPDTR/ZAPD/ZTextMM.cpp (ZTextMM::ParseMM); field order from OTRExporter/TextMMExporter.cpp. staff_message_data_static is its own format again -- no header, ends at 0x02, its own control codes -- selected by the file's name, as ZAPD selects it. Its textboxYPos keeps the low nibble of the table's type/position byte while textboxType is zeroed immediately after being computed; that asymmetry looks like an oversight but it is what the reference contains, so it is reproduced. MM 50494 -> 50496 of 50496, byte-identical. OoT pal_gc 35386 matching, 0 mismatched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
what this does
generates a matching o2r (to https://github.com/briaguya0/2ship2harkinian/tree/deterministic-extraction) when using an NTSC-U 1.0 rom
testing
https://github.com/briaguya0/zapd-to-torch-test-harness/tree/2ship
what's with the 2ship branch?
it's just Battler Alfa with a change to OTRExporter for determinism (ran into the same "we write garbage" thing when porting soh)
what's left to do/why is this a draft?
i still need to actually go through the code in here and make it not garbage. the goal of this pass was "get it to build a matching o2r" not "get it to a state where landing any of this code is a good idea"
sharing as a draft mostly so people can ping me on it and say "hey bria when are you gonna finish this?"