Ys VIII Data Extraction

This is a place to post tech details and informative documents about romhacking PSP games.
flamethrower
Programmer
Posts: 995
Joined: Mon Mar 09, 2015 3:03 pm

Ys VIII Data Extraction

Post by flamethrower » Thu Aug 04, 2016 10:35 pm

Could not dump Ys8. It closes Near when you launch the game.
Tried the email version too.
I messed up the email version. I'll try again in a bit.
Ys8 won't open in the email app (it will open in Near). The email app behaves the same as Near. It closes when the game starts.

Zero Evo and Ao Evo are up on our FTP for staff members. 21,000 voice clips for Zero and 23,000 for Ao.

User avatar
Gu4n
Posts: 129
Joined: Wed Jul 08, 2015 5:25 am

Re: Thoughts on hacking Evolution voices into the PC version?

Post by Gu4n » Fri Aug 05, 2016 7:26 am

flamethrower wrote:Could not dump Ys8. It closes Near when you launch the game.
Tried the email version too.
I messed up the email version. I'll try again in a bit.
Ys8 won't open in the email app (it will open in Near). The email app behaves the same as Near. It closes when the game starts.

Zero Evo and Ao Evo are up on our FTP for staff members. 21,000 voice clips for Zero and 23,000 for Ao.
I got Ys 8 to open MolecularShell (what it's supposed to do), but it show any decrypted any files. All I can see is the encrypted file, which isn't even a .psarc. I also can't get Tokyo Xanadu to work. Not sure if these games don't work or whether I'm doing anything wrong.

awvnx
Posts: 15
Joined: Tue Mar 15, 2016 2:09 am

Re: Thoughts on hacking Evolution voices into the PC version?

Post by awvnx » Fri Aug 05, 2016 7:55 am

Gu4n wrote: I got Ys 8 to open MolecularShell (what it's supposed to do), but it show any decrypted any files. All I can see is the encrypted file, which isn't even a .psarc. I also can't get Tokyo Xanadu to work. Not sure if these games don't work or whether I'm doing anything wrong.
Have you been successful at dumping any game?

If you're dumping a physical game, make sure you use gro0 instead of ux0 in all parts of the instructions. If you enter ux0 to the browser URI then you'll be grabbing encrypted files from gro0 later.

Not sure what you mean by "all I can see is the encrypted file" because the folder structure and file names will be same whether or not you did everything correctly. It's the contents that are either encrypted or decrypted. Try grabbing images from the maunal folder to quickly check if decryption is working or not.

I'll try on my copy of Tokyo Xanadu later.

josejl
Programmer
Posts: 200
Joined: Mon Apr 13, 2015 6:49 pm
Location: Spain
Contact:

Re: Thoughts on hacking Evolution voices into the PC version?

Post by josejl » Fri Aug 05, 2016 8:19 am

I've read that some games aren't working (like Danganronpa Another Episode,for instance).

When they reported it to MajorTom (the one who came up with this trick), he asked for the param.sfo files. Other than that, I haven't found anything else so far.

User avatar
Gu4n
Posts: 129
Joined: Wed Jul 08, 2015 5:25 am

Re: Thoughts on hacking Evolution voices into the PC version?

Post by Gu4n » Fri Aug 05, 2016 8:24 am

But can you see anything on app0? When I had problems because of the lack of the near app on my PSTV, I got a different path on the FTP (can't remember it exactly, it contained app0 and more stuff,but it was empty).

I could browse on ux0:apps/ and see the encrypted game data (digital Cold Steel),but the folder/file structure was the same. Also,when the trick is working,you also can access via ux0:/apps. I have no physical game with me ,but perhaps you can try with gr0: instead.

Also, I think that sce_sys shouldn't have any actual game data.

josejl
Programmer
Posts: 200
Joined: Mon Apr 13, 2015 6:49 pm
Location: Spain
Contact:

Re: Thoughts on hacking Evolution voices into the PC version?

Post by josejl » Fri Aug 05, 2016 8:36 am

Awesome,it seems I somehow broke phpBB again, my message appears as Gu4n's, while replacing his own.
But can you see anything on app0? When I had problems because of the lack of the near app on my PSTV, I got a different path on the FTP (can't remember it exactly, it contained app0 and more stuff,but it was empty).

I could browse on ux0:apps/ and see the encrypted game data (digital Cold Steel),but the folder/file structure was the same. Also,when the trick is working,you also can access via ux0:/apps. I have no physical game with me ,but perhaps you can try with gr0: instead.

Also, I think that sce_sys shouldn't have any actual game data.

User avatar
Gu4n
Posts: 129
Joined: Wed Jul 08, 2015 5:25 am

Re: Thoughts on hacking Evolution voices into the PC version?

Post by Gu4n » Fri Aug 05, 2016 8:53 am

josejl wrote:Awesome,it seems I somehow broke phpBB again, my message appears as Gu4n's, while replacing his own.
But can you see anything on app0? When I had problems because of the lack of the near app on my PSTV, I got a different path on the FTP (can't remember it exactly, it contained app0 and more stuff,but it was empty).

I could browse on ux0:apps/ and see the encrypted game data (digital Cold Steel),but the folder/file structure was the same. Also,when the trick is working,you also can access via ux0:/apps. I have no physical game with me ,but perhaps you can try with gr0: instead.

Also, I think that sce_sys shouldn't have any actual game data.
What I understand from flame through chat is that Ys VIII uses up so much RAM, it disables Near from functioning in the background and thus doesn't have the files show up decrypted. That makes sense; Ys VIII can't be run alongside another application.

I'll leave this in you people's capable hands. If someone could share the (script) files with me, that would make me happy.

josejl
Programmer
Posts: 200
Joined: Mon Apr 13, 2015 6:49 pm
Location: Spain
Contact:

Re: Thoughts on hacking Evolution voices into the PC version?

Post by josejl » Fri Aug 05, 2016 9:21 am

Heh,it seems Flame already asked.

This response he got seems promising:

"I managed to dump those yesterday, you need to patch app.db.
With the help of someone from gbatemp, we noticed that the problem was either in key 2412347057 or key 1640794848 in the game entry in tbl_appinfo.
Simply delete those 2 keys if they exist (I mean, the full entry of the key). "

flamethrower
Programmer
Posts: 995
Joined: Mon Mar 09, 2015 3:03 pm

Re: Thoughts on hacking Evolution voices into the PC version?

Post by flamethrower » Fri Aug 05, 2016 11:28 am

I tried this and it didn't work. I'll try using the email trick on my 3.51 Vita and see if that works.
I had a key PCSG00881;2412347057;17 - delete entry, write changes, delete app.db, upload app.db, reboot. Same issue though, email (and other apps) close when you open Ys VIII.
I also have a key PCSG00246;2412347057;17 which is for Ao no Kiseki Evolution and that one will dump just fine.
So I don't think that's it.

I don't have any keys matching 1640794848 in my database.

flamethrower
Programmer
Posts: 995
Joined: Mon Mar 09, 2015 3:03 pm

Re: Thoughts on hacking Evolution voices into the PC version?

Post by flamethrower » Sun Aug 07, 2016 5:15 pm

josejl can you help?
The old (long, difficult) method I executed using my 3.51 Vita worked.
I got the game data unencrypted and I could use some help extracting the file.
This is the first 1MB of the file.
For some reason, you can see unencrypted data in SHIFT-JIS towards the end of this 1MB.
This game is 3.4GB (approx) that's why the table at the front is so huge, nearly 1MB in size.
rootast_1MB.7z
(251.5 KiB) Downloaded 171 times
The first four bytes of this file are XAST and my google-fu turned up nothing.

I uploaded the full thing to our ftp server for staff members.

josejl
Programmer
Posts: 200
Joined: Mon Apr 13, 2015 6:49 pm
Location: Spain
Contact:

Re: Thoughts on hacking Evolution voices into the PC version?

Post by josejl » Sun Aug 07, 2016 6:03 pm

Excellent, glad to see it worked after all, thanks for your time. I wasn't expecting Falcom to use a container file here, but well...

I haven't seen this format before either, and so far I haven't found anything. Perhaps it'll make sense with the full file, but I haven't found any relationship between the first binary table and the filename list which appears later.

I've been taking a quick look at ZnK Evo too,but I'm empty-handed so far. That uncompiled script certainly looks interesting. There are no direct mentions of playing voices,so I guess they are encoded somewhere (these "#0010000005V"-like values seem suspicious to me, although I don't know much about Kiseki script formats)

flamethrower
Programmer
Posts: 995
Joined: Mon Mar 09, 2015 3:03 pm

Re: Thoughts on hacking Evolution voices into the PC version?

Post by flamethrower » Sun Aug 07, 2016 7:02 pm

Regarding the Ys8 file.
Still no real ideas on this. The first decrypted file is at 0xE9F10, you see that value at 0x14 in the header so that might be important.
0x91550: You see 0xE9F10
0x91540: You see 0x4E9, this is the file size of the first file
0x91558: You see 0x4F0, this is the file size aligned to 0x10
The file name table starts at 0xAAD60. The first file path in the table is "actev/act0000.csv" It's extremely suspicious for the actual path name of this first file, because it is indeed a CSV and seems to enumerate some kind of event data. Beyond that I'm having trouble connecting the file paths and data.

The next file is at 0xEA400, file size 0x4E9.
You find that entry at 0x48AA0 (that's where the offset entry is).

The next file is at 0xEAA40, file size 0x63C.
You find that entry at 0x0x48B60.

Is it worth asking at Xentax about this? Maybe someone has heard of XAST.

josejl
Programmer
Posts: 200
Joined: Mon Apr 13, 2015 6:49 pm
Location: Spain
Contact:

Re: Thoughts on hacking Evolution voices into the PC version?

Post by josejl » Sun Aug 07, 2016 7:04 pm

Okay, a little update on Ys VIII container file.

I was able to extract one of the music files manually.

I only had to look for the header that the AT9 files use, and select up to the next header. This shows the data is uncompressed and can be used right away.

Also,thanks to this I could look for the offsets and file sizes in the first table. The offset is at 0x69860, and the file size is at 0x69868.

For some reason,the offset for the second music file is at 0x696B0, so they aren't sorted. I still have to figure out what the relationship between this table and the filenames that appear later.

Edit: Ok, at 0x69844 I see the pointer to the bgm/y8_b001.at9.

flamethrower
Programmer
Posts: 995
Joined: Mon Mar 09, 2015 3:03 pm

Re: Thoughts on hacking Evolution voices into the PC version?

Post by flamethrower » Sun Aug 07, 2016 7:21 pm

josejl wrote:Edit: Ok, at 0x69844 I see the pointer to the bgm/y8_b001.at9.
Nice work. Now it should be quite simple to extract the contents.
Seems like we have path, offset and size data for each entry. That should be all that's needed. All you need to do is go through each entry, detect if it's a valid entry, and if it is, dump the file. I saw some dummy entries in there.

awvnx
Posts: 15
Joined: Tue Mar 15, 2016 2:09 am

Re: Thoughts on hacking Evolution voices into the PC version?

Post by awvnx » Sun Aug 07, 2016 7:39 pm

Let's split off the general dumping and Ys VIII extracting stuff to another thread.

josejl
Programmer
Posts: 200
Joined: Mon Apr 13, 2015 6:49 pm
Location: Spain
Contact:

Re: Ys VIII Data Extraction

Post by josejl » Sun Aug 07, 2016 8:38 pm

Seems the script I made worked.

12776 files and 3.32 Gb,at least the numbers make sense.

http://pastebin.com/cGfPQB1i

It works on Python 3.5.1.

flamethrower
Programmer
Posts: 995
Joined: Mon Mar 09, 2015 3:03 pm

Re: Ys VIII Data Extraction

Post by flamethrower » Sun Aug 07, 2016 9:44 pm

Working for me too. Thanks for making this for me.

This is Python for read until sentinel value:

Code: Select all

from itertools import partial
b = b''
for char in iter(partial(f.read, 1), b'\x00'):
    b.append(char)
partial(f.read, 1) creates a function that reads a byte when it is called.
iter(partial(f.read, 1), b'\x00') will (over and over) call partial(f.read, 1) and do the body with the return value until the sentinel value b'\x00' is read.

josejl
Programmer
Posts: 200
Joined: Mon Apr 13, 2015 6:49 pm
Location: Spain
Contact:

Re: Ys VIII Data Extraction

Post by josejl » Sun Aug 07, 2016 10:04 pm

Ah,good to know, thanks.

I'm not too good at Python,so I end up programming stuff like if I was programming in C/C++.

It seems that Falcom has made a new revision of the ITP files,I tried to run the data with the Celceta EXE (I was hoping that it would load the title screen,like it does with the Ys Seven data), but it said something about revision not supported.

It's kind of funny that they're using AT9 files for the voices since ZnK Evo, now that we have a decoder for the HE-VAG files.

User avatar
NightWolve
Posts: 82
Joined: Fri Apr 17, 2015 10:32 am
Location: Chicago, IL, USA
Contact:

Re: Ys VIII Data Extraction

Post by NightWolve » Mon Aug 08, 2016 4:48 am

Ah, already looking at Ys VIII. :)

flamethrower
Programmer
Posts: 995
Joined: Mon Mar 09, 2015 3:03 pm

Re: Ys VIII Data Extraction

Post by flamethrower » Mon Aug 08, 2016 8:43 pm

Looking at the itps. I wasn't able to solve it. Maybe there is a swizzle thing going on again?
It's like a cross between regular itp and it3? I don't really know...
Look at \visual\chara\dr_aaron.it3
0x10 0x14 suspicious entries for height and width
0x40 IPAL (IDSTRING)
0x50 start of FALCOM3 compressed stream with the palette data
0x4C9 IDAT (IDSTRING)
0x5C9 start of FALCOM3 compressed stream with the bitmap data
The palette data when uncompressed is 0x400 which is consistent for BMP with a color table with 4 bytes per color and 256 colors. And the bitmap data when uncompressed is 0x10000. That's consistent with 256x256 image with 1 byte per pixel.

Here's the decompression program I wrote for Ys vs Sora no Kiseki skills table. It's capable of decompressing FALCOM3:
http://pastebin.com/WpWJHu6B
I could use some help making it easier to use, but... Feel free to modify as needed.

I don't really know how to put palette and bitmap data together. I tried a program called TiledGGD and you can try various settings and marry up palette and bitmap data. Nothing I tried worked, though.

ITP file:
dr_aaron.7z
(16.26 KiB) Downloaded 172 times
Linear mode render
dr_aaron_linear.png
dr_aaron_linear.png (21.82 KiB) Viewed 6673 times
16x8 tiles mode render
dr_aaron_16x8tiles.png
dr_aaron_16x8tiles.png (23.02 KiB) Viewed 6673 times

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest