GoldenEye 007 Nintendo 64 Community, GoldenEye X, Nintendo 64 Games Discussion

[Wreck]

Always good to clear one hurdle and move onto the next. Any progress is good progress.

[zoinkity]

Those are the pointers to that data at runtime. The 0x21990 binary decompresses from 80020D90 to 8005D2E0.

Those address ranges refer to:

Code:

800484D0   0x27740   language 800484D4   0x27744   table of text call values for each text bank

Cross-referencing the 21990 doc against the address ranges doc can be helpful, especially when it references data outside of the compressed binary.

Oh, and there is a filesize limit on the compressed binary unless you start hacking ASM. There's clearly defined start and end addresses used to DMA the data that will be decompressed. Of course, if you shift that back everything needs to move in response.

Code:

0x10C8 + 0x10D0   start of compressed file 0x10D4 + 0x10DC   end compressed file

_________________
(\_/) Beware
(O.o) ze
(> <) Hoppentruppen!

[K1lo]

Quote:

Always good to clear one hurdle and move onto the next. Any progress is good progress. Smile

Yep! Feels good to be past that particular road block especially since I couldn't debug why it was going in to an infinite loop.

I kind of was shooting in the dark for a while.. but it's ok as I took the time to add lots of sanity checking code and review every line of code I wrote.

Quote:

Cross-referencing the 21990 doc against the address ranges doc can be helpful, especially when it references data outside of the compressed binary.

Yes I added a ton of sanity checking code against the 21990 document to make sure I was reading what I was expecting.

So, if I understand correctly 80xxxxxx addresses map to N64 / emulated memory at runtime?

As a result this is not something I will be editing as I only make 'offline' modifications to the ROM image itself.

Quote:

Oh, and there is a filesize limit on the compressed binary unless you start hacking ASM. There's clearly defined start and end addresses used to DMA the data that will be decompressed. Of course, if you shift that back everything needs to move in response.

Well I'm not adding anything to the 21990 table, just changing pointers. I easily compress the data block down to less than the space the original occupied so that not an issue I don't think.

Hopefully I won't come across these hard limits with setups though.. are you aware what the max setup compressed data size is?

Quote:

There's clearly defined start and end addresses used to DMA the data that will be decompressed. Of course, if you shift that back everything needs to move in response. Code: 0x10C8 + 0x10D0   start of compressed file 0x10D4 + 0x10DC   end compressed file

I don't follow - is the DMA is another table somewhere which defines where the 21990 is located and it's maximum size?

If that's the case I won't need to touch this for the reason above.

However if this also includes the individual setup blocks and individual text entries then perhaps I will have a problem since I am rearranging where these within the blocks that they were previously stored in.

[zoinkity]

Setup files--or really any of the files you see in that filetable--don't have a maximum compressed size. They don't actually need to be compressed. Only conflict you'll run into is needing to push back the image bank's initial offset.

Their decompressed size can be troublesome though, and that's why every map has its own memory allocation values. There's a number of factors influencing that: how many object instances there are, how many unique models are loaded, volume of textures loaded, size of the bg data, required memory for the loaded room buffer, etc.

Don't make too fine a distinction between ROM and rdram. Ignoring pointers and addressing can cause serious problems later on.

DMA is a direct memory access, allowing an I/O device to copy data directly to memory or vice-versa. In this case, it's loading data from a cart directly to rdram. I mentioned the offsets because Sub and I have managed to make radical enough changes to that compressed file to grow it larger than its original size. Pretty sure that was the original reason for removing the debug strings.
_________________
(\_/) Beware
(O.o) ze
(> <) Hoppentruppen!

[K1lo]

zoinkity wrote:

Setup files--or really any of the files you see in that filetable--don't have a maximum compressed size. They don't actually need to be compressed. Only conflict you'll run into is needing to push back the image bank's initial offset.

Well since I'm not touching image banks *hopefully* that means everything is quite simple.

zoinkity wrote:

Their decompressed size can be troublesome though, and that's why every map has its own memory allocation values. There's a number of factors influencing that: how many object instances there are, how many unique models are loaded, volume of textures loaded, size of the bg data, required memory for the loaded room buffer, etc.

Yes I already noticed with testing my changes.. I have to be careful with the degree of variation so as not to overload these constraints.

zoinkity wrote:

Don't make too fine a distinction between ROM and rdram. Ignoring pointers and addressing can cause serious problems later on.

In general I completely agree - however please correct me if I'm wrong, aside from pointers in the 21990 table which I update, I don't have to touch anything else right?

zoinkity wrote:

DMA is a direct memory access, allowing an I/O device to copy data directly to memory or vice-versa. In this case, it's loading data from a cart directly to rdram. I mentioned the offsets because Sub and I have managed to make radical enough changes to that compressed file to grow it larger than its original size. Pretty sure that was the original reason for removing the debug strings.

Aha makes sense!

[K1lo]

Hmm I've noticed something odd.. if I strip out the J text entries and update the 21990 entry to point to the E version block the game doesn't start.

However if I remove the J text block and just leave the entry in the 21990 now pointing to NULL data it runs.

Very odd

EDIT:

Actually after more digging I found that if I edit any of the ROM address pointers with the 21990 table the game doesn't start. I'm just met with a black screen... I'm not really sure how to debug this.

The most simple test case I can come up with is setting all the J pointers in the 21990 table to the first valid entry - 0x008e7420 (which is the first J entry 7e48:0000027f|8005b384|008e7420 LameJ).

I've already verified that the data doesn't have to be in the ROM for J entries as I removed all of the blocks referenced by J entries in the 21990 table (without actually editing the table) and the game started and worked fine.

Is there anything else I need to do with the 21990 table when I change a ROM address pointer value within it?

[Wreck]

Someone else can chime in with better details, but I thought the main file table needed to have incrementing addresses. I don't think you can direct it back earlier in the ROM, since it would break that ordering. Maybe I am wrong. We've relied on the Setup Editor for so many years now, that some specifics are difficult to remember.

[K1lo]

Effectively what I am doing is this:



I can strip out the J text blocks from without touching the entries in the 21990 table and it works.

However if I strip out the J text blocks and pack the Es together I need up update their 21990 entries - this causes the ROM to fail to start.

[SubDrag]

Have you tried comparing to doing the same function in Setup Editor?

[K1lo]

SubDrag wrote:

Have you tried comparing to doing the same function in Setup Editor?

Actually no because I'm not sure what other changes Setup Editor is doing 'under the hood'. I was trying to come up with the most minimal test case to understand what was going wrong.

Since I suspected the issue isn't the pointer value in the 21990 table (since these are correct) I thought the issue must be elsewhere. Since the Setup Editor makes a lot of changes I thought trying to track down the specific changes would be a tough task.

[SubDrag]

It does make changes, but focus on relating to the files. Ignore the rest. Look at those two areas. If there are any differences...that's probably why.

[kholdfuzion]

To help understand the 0x21990 section, check out my disassembly https://gitlab.com/kholdfuzion/Gold007.
The data section is slowly being migrated away from a single "monolithic" chunk to more logical locations, but for the stuff your touching, it's still mostly contiguous in src/tlbdata.s. The main file table uses macros obseg_file_Z and obseg_file_rz to add files to it (Z for untouched rom extract, rz for stuff recompressed from bin or source)


As far as the strings in LnameE/J thats in obseg/text. Text files should be 100% editale. You will need bass and mips64-elf-as to assemble.

[K1lo]

kholdfuzion wrote:

To help understand the 0x21990 section, check out my disassembly https://gitlab.com/kholdfuzion/Gold007. The data section is slowly being migrated away from a single "monolithic" chunk to more logical locations, but for the stuff your touching, it's still mostly contiguous in src/tlbdata.s. The main file table uses macros obseg_file_Z and obseg_file_rz to add files to it (Z for untouched rom extract, rz for stuff recompressed from bin or source) As far as the strings in LnameE/J thats in obseg/text. Text files should be 100% editale. You will need bass and mips64-elf-as to assemble.

All I see is assembly which is outside my skill set there. I don't understand how I can use this to understand the problem I am having?

[zoinkity]

A few things could be wrong.

1) Did you change the number of entries in the table at 0x27544? If you shrink the table at all and don't change the number of entries then this function:

Code:

7F0BCC28   F1758   generate resource data filesize entries

-goes haywire. Incidentally, the most entries that can be in the game total (not just the table) is 0x2E0, hardcoded at 7F0BD2CC and mandated by the other table above.

2) Did the total size of data in the filetable grow past its original size? If it ends past 0x8F7DF0 you'll need to change the offset for the image bank as well.

3) Did you change the language file lookup table starting at 0x27744 or fiddle with any of the strings for it at 0x3AAF0? Both would build the text banks wrong and cause a crash at lookup. Reason being is the language flag at 800484D0 is multiplied by the number of languages in the table (NTSC is 2, PAL 3) to do lookups. Removing the entries in the filetable though would have no effect unless a missing one were called.

Those are the three most obvious things. Otherwise couldn't tell you without actually looking at a sample.
_________________
(\_/) Beware
(O.o) ze
(> <) Hoppentruppen!

[K1lo]

zoinkity wrote:

A few things could be wrong. 1) Did you change the number of entries in the table at 0x27544? If you shrink the table at all and don't change the number of entries then this function: Code: 7F0BCC28   F1758   generate resource data filesize entries -goes haywire. Incidentally, the most entries that can be in the game total (not just the table) is 0x2E0, hardcoded at 7F0BD2CC and mandated by the other table above.

All I'm doing is changing the pointers for where the text entries are. I'm not adding or removing any entries to the 21990 table. For the J entries I've removed the data for from the ROM, I was just hoping to set the pointer to the equivalent E entry. It shouldn't matter anyway what these are set to since my understanding is they're never used in the NTSC game?

zoinkity wrote:

2) Did the total size of data in the filetable grow past its original size? If it ends past 0x8F7DF0 you'll need to change the offset for the image bank as well.

No, all I do is change a few DWORD pointer values so no net change in 21990 table size. Actually when compressing it's smaller than before, although I suspect this won't make any difference.

zoinkity wrote:

3) Did you change the language file lookup table starting at 0x27744 or fiddle with any of the strings for it at 0x3AAF0?

Nope

zoinkity wrote:

Both would build the text banks wrong and cause a crash at lookup. Reason being is the language flag at 800484D0 is multiplied by the number of languages in the table (NTSC is 2, PAL 3) to do lookups. Removing the entries in the filetable though would have no effect unless a missing one were called. Those are the three most obvious things. Otherwise couldn't tell you without actually looking at a sample.

I didn't have any more time to look at it today, let me post a patch. If you or someone else could dig deeper into it I'd really appreciate it as I have no idea how to debug the ROM itself.

UPDATE - I've uploaded patch files:

J text removed, no changes to 21990 table (works) This corresponds to the middle state pictured above.

J text removed, entries in 21990 table updated for J to point to E (not working)

Ultimately this is what I am trying to achieve:

J text removed, E entries moved and reordered, entries in 21990 table updated for E and J (not working) This corresponds to the right hand state in the image above.

Additionally, for clarity, this is my current code. It's not the prettiest or the most robust I've ever written, but it's a WIP until I can get to the bottom of what's going on with these pointers. Anyway I've included it in the hope it sheds more light into what I'm doing.

Code:

bool ROM::StripOutJapaneseText() {     //     // First parse table     //     const uint32_t OffsetToStartOfStagesWithinTable = 0x270AC;     // Determined by counting number of E and J entries from documented table at the URL below     // http://fgfc.ddns.net/PerfectGold/ZoinkDocs/GameShark%20-%20GE%20specific/multi%20load%20information.txt     const uint32_t numberOfTableEntries = 44;     uint32_t offset = OffsetToStartOfStagesWithinTable;     std::vector<InStageText> ist;     // Sanity check the first value to make sure it is what we expect     uint32_t entry = GetDWORDFromOffset( m_romData.block21990,  offset );     // The first entry should be this value     if( entry != 0x27e )     {         LogError( "StripOutJapaneseText(): First entry in 21990 table [" + std::to_string(entry) + "] is not what was expected!" );         return false;     }     bool shouldContinue = true;     while( shouldContinue )     {         // Structure of these entires is the same as for levels, however now we are combinting both the English and Japanese         // texts into a single entry. The reason for this is we want to discard the Japanese and rewrite the table         // Third DWORD has offset to English string block         const uint32_t offsetWithin21990ToRomDataPointerE = offset + ( sizeof( uint32_t ) * 2 );         // Third DWORD in second data block has ofset to English string block (3 + 2 jumps)         const uint32_t offsetWithin21990ToRomDataPointerJ = offset + ( sizeof( uint32_t ) * 5 );         // Offset to the English resource ID         const uint32_t gameAssetIDE = GetDWORDFromOffset( m_romData.block21990,  offset );                 // Get offset to English string data in ROM         const uint32_t textffsetWithinROME = GetDWORDFromOffset( m_romData.block21990,  offsetWithin21990ToRomDataPointerE );         // Offset to the Japanese resource ID         const uint32_t gameAssetIDJ = GetDWORDFromOffset( m_romData.block21990,  offset );                 // Get offset to Japanese string data in ROM         const uint32_t textffsetWithinROMJ = GetDWORDFromOffset( m_romData.block21990,  offsetWithin21990ToRomDataPointerJ );         if( gameAssetIDE <0x27e> 0x2d5 || gameAssetIDJ <0x27e> 0x2d6)         {             LogError( "StripOutJapaneseText(): First entry in 21990 table [" + std::to_string(entry) + "] is not what was expected!" );             return false;         }         // Store offsets and a copy of the E text data         InStageText i( offsetWithin21990ToRomDataPointerE, offsetWithin21990ToRomDataPointerJ, textffsetWithinROME, textffsetWithinROMJ );         // Load in raw string data for E only determining size by reading up until the Japanese which is the next entry         uint64_t size = i.m_startOffsetInROMForJ - i.m_startOffsetInROMForE;         m_f.GetAsVector( i.m_startOffsetInROMForE, size, i.m_rawDataE );         ist.push_back( i );         // Jump forward to next entry         offset += sizeof( uint32_t ) * 6;         if( ist.size() == numberOfTableEntries )         {             // Don't parse any more entries             shouldContinue = false;         }     }     // Sanity check the first stored data block     if(    ist[0].m_startOffsetInROMForE != 0x008e7410         || ist[0].m_startOffsetInROMForJ != 0x008e7420         || ist[0].m_OffsetToStartOffsetIn21990ForE != 0x00270b4         || ist[0].m_OffsetToStartOffsetIn21990ForJ != 0x00270c0)     {         LogError( "StripOutJapaneseText(): First entry in InStageText collection was not what was expected!" );         return false;     }     // Sanity check the last block     if(    ist[ist.size()-1].m_startOffsetInROMForE != 0x008f7dc0         || ist[ist.size()-1].m_startOffsetInROMForJ != 0x008f7dd0         || ist[ist.size()-1].m_OffsetToStartOffsetIn21990ForE != 0x00274bc         || ist[ist.size()-1].m_OffsetToStartOffsetIn21990ForJ != 0x00274c8)     {         LogError( "StripOutJapaneseText(): Last entry in InStageText collection was not what was expected!" );         return false;     }     LogInfo( "StripOutJapaneseText(): Finished parsing 21990 block and caching data" );     //     // Wipe entire block from ROM     //     // Offsets obtained from http://fgfc.ddns.net/PerfectGold/ZoinkDocs/GameShark%20-%20GE%20specific/multi%20load%20information.txt which     // is a little lazy since we can work it out from the data we have, but it should be fine     const uint32_t startOffsetForTextBlock = ist[0].m_startOffsetInROMForE;     const uint32_t endOffsetForTextBlock = ist[ist.size() - 1].m_startOffsetInROMForJ + ist[ist.size() - 1].m_rawDataE.size();  // 0x08f7de0     const uint32_t sizeOfTextBlock = endOffsetForTextBlock - startOffsetForTextBlock - 1;     memset( m_f.GetPtrToMemory(startOffsetForTextBlock, sizeOfTextBlock) , '\0', sizeOfTextBlock );     LogInfo( "StripOutJapaneseText(): Text block wiped" );     //     // Working backwards from the end, store the E data only and update the 21990 E and J pointers     //     uint32_t writingOffset = endOffsetForTextBlock;     // Iterate in reverse through collection so the last entry in the 21990 table will be the last in the text block     std::vector<InStageText>::const_reverse_iterator rit = ist.rbegin();     std::vector<InStageText>::const_reverse_iterator ritEnd = ist.rend();     uint32_t previousBlankSpace = 0x08e7a60; #if 0  // Test rewriting the data where it was before, don't reorder     for( ; rit != ritEnd ; ++rit )     {         const InStageText& i = *rit;         uint32_t currentOffsetE = GetDWORDFromOffset( m_romData.block21990, i.m_OffsetToStartOffsetIn21990ForE );         uint32_t currentOffsetJ = GetDWORDFromOffset( m_romData.block21990, i.m_OffsetToStartOffsetIn21990ForJ );         writingOffset = currentOffsetE;         memcpy( m_f.GetPtrToMemory(writingOffset, i.m_rawDataE.size()),             &i.m_rawDataE[0],             i.m_rawDataE.size() );         // Say both Japanese and English data is stored where we put the English         SetDWORDToffset( m_romData.block21990, i.m_OffsetToStartOffsetIn21990ForE, currentOffsetE );         SetDWORDToffset( m_romData.block21990, i.m_OffsetToStartOffsetIn21990ForJ, currentOffsetE );         //previousBlankSpace = currentOffsetJ;         LogInfo( "ROM::StripOutJapaneseText(): Updated E [" + ToHexString(currentOffsetE) + "],  J [" + ToHexString(currentOffsetJ) + "] -> [" + ToHexString(writingOffset) + "]" );     } #else  // Reorder the data     for( ; rit != ritEnd ; ++rit )     {         const InStageText& i = *rit;         writingOffset -= ( i.m_rawDataE.size() );         memcpy( m_f.GetPtrToMemory(writingOffset, i.m_rawDataE.size()),                 &i.m_rawDataE[0],                 i.m_rawDataE.size() );         // Say both Japanese and English data is stored where we put the English         uint32_t currentOffsetE = GetDWORDFromOffset( m_romData.block21990, i.m_OffsetToStartOffsetIn21990ForE );         uint32_t currentOffsetJ = GetDWORDFromOffset( m_romData.block21990, i.m_OffsetToStartOffsetIn21990ForJ );         SetDWORDToffset( m_romData.block21990, i.m_OffsetToStartOffsetIn21990ForE, writingOffset );         SetDWORDToffset( m_romData.block21990, i.m_OffsetToStartOffsetIn21990ForJ, writingOffset );         LogInfo( "ROM::StripOutJapaneseText(): Updated E [" + ToHexString(currentOffsetE) + "],  J [" + ToHexString(currentOffsetJ) + "] -> [" + ToHexString(writingOffset) + "]" );     } #endif     // Space saved is where we are within the block with respect to start of the block     uint32_t spaceSaved = writingOffset - startOffsetForTextBlock;     // Add an extra buffer byte to make sure we don't overwrite the new text with the setups     m_romData.endOfSetupWritableArea = writingOffset - 1;     // Show space saved     LogInfo( "StripOutJapaneseText(): Finished reorganising text data [" + std::to_string(spaceSaved) + "] bytes freed" );     return true; }

.. yikes.. it didn't look so long in Visual Studio .. sorry for the massive post.