Phrase dictionary and dual-base tree topology, with verified tooling

[RadMageIRL]

Hi Markus,

I have been building DQ4 PSX tooling on top of your documentation
and wanted to send you what came out of it:

https://github.com/RadMageIRL/DQIV_PSX_TOOLS

Two things you may care about specifically. There is a per-block
phrase dictionary in 189 of the 1,528 text blocks, pointed to by
the sixth header int, which is 24 in exactly those 189 and 0 in the
other 1,339. Text from those blocks decodes with unexpanded 0x7Exx
codes unless it is handled. And your disassembly note about one
tree pointer targeting the start and another the middle turned out
to be the whole topology, which is what made the decoder work.

The repo is stdlib only Python with a gate suite you can run
against your own disc, and every format claim carries either a gate
number or an explicit INFERRED or UNKNOWN label. Round trip is
byte-exact on all 1,528 text sub-blocks.

Your documentation is the reason any of this was findable.

Thanks for putting it up.

RadMageIRL

[mschroeder-github]

Hi RadMageIRL,

thank you very much for your time and dedication to work on this project and to acknowledge my work.
You might be aware of the rom hacking forum where you could also present your work. I also saw that there is another group working on something.

My biggest interest is this question: Can we translate the game with your code project? Is it possible to insert longer English text than the Japanese script?

If you would like to discuss in depth you can contact me.

[RadMageIRL]

Can it translate? English is playing in game the full prologue across five archive blocks, the field and title menus, the examine and pickup families. Not a demo of one string. It can be done. It's a lot of work. And given the format (ISO/CD) it's harder to deal with as well, but possible, yes.

Can it insert longer text? Yes, and the interesting part is that it usually isn't longer. Measured at 1.922x characters against Enix's (Chuns/HB) own 1.944x in DW7, and blocks routinely come out smaller because English rebuilds the tree over approx. 51 symbols instead of 109. Where a block does overflow, reclaiming the per block kanji records fixes it, which is what Enix themselves did, measured on the DQ7 JP/EN pair the records collapse to a fixed 36 bytes, 837 of 877 blocks came out smaller, one gained a sector.

There are 2 Font tables. Font1 Menu, Font2 dialog.

[mschroeder-github]

Very nice. Are you in contact with translators, editors or a group? Can your tool be used by translators easily? Are you planing to publish it?

[RadMageIRL]

No group, it's me that speaks JP poorly and has some JP extended family.

On usability for translators, no, not fully yet, honestly. The extraction side is fine the whole script comes out as text with block and string indices. But conversely there's multiple EN versions of DQIV you can diff/compare.

The other side needs finishing, which I view as a tool that can easily insert, from a side by side JP to EN script/name/item/monster lists. The insertion side still needs a romhacker in the loop, because authoring has to respect per line cell budgets, substitution codes that reserve 6-9 cells at render time, and per block space limits.

It's also a nonobvious one as each block has its own huffman tree, using fewer distinct characters saves more space than writing shorter text. A translator shouldn't have to know all that.

Closing that gap is on my list, such as a tool that takes plain English and reports what fits, rather than requiring someone to already know the constraints.

I created a DQVI script/framework, plain EN that can be edited then inserted. Goal is same here, but some hurdles exist, and PSX is harder than SNES/SFC.

I have not looked at all menus such as battle and interaction with party/party chat (maybe some other stuff). Lofty goal is get prologue working end to end then investigate everything in Ch1, if no big gotchas exist, then it is possible things collapse down to bulk/volume/tedious side of work and engineering is in the rearview. But not fully known right now.

The tooling is already public and MIT, please enjoy.

[mschroeder-github]

Ok, then I look forward to your progress to have a ch1 roundtrip 👍