R3: Music Extraction Discussion Topic

Talk about the Rayman music website.

Moderators: Droolie, English moderators

Forum rules
Please keep the forum rules and guidelines in mind when creating or replying to a topic.
gmsj0001
Électoon
Posts: 20
Joined: Wed Jul 23, 2008 1:26 pm
Location: Shanghai, China
Contact:
Tings: 0

R3: Music Extraction Discussion Topic

Post by gmsj0001 »

I'm the master of the Chinese Rayman Fan-club who played Rayman for ten years. And I loved the music in Rayman very much. Today I extracted some files from the data.hos and data.hst file in Rayman 3, But only the sound effects in data.hos is playable. All waves in data.hst is unable to play. I know perhaps it is because the compression cannot be recognised.

Now the master of this site extracted the music successfully. I really want to know the secrect of the data.hst file, or the compress method. I'm nearly crazy this days for the music... Help me!
Droolie
Mr Sax
Posts: 5162
Joined: Wed Apr 20, 2005 7:31 pm
Tings: 61830

Re: R3: How to extract the background music?

Post by Droolie »

Hello and welcome to Pirate-Community! :)

We have been doing the same for a very long time, but still without success. :(
This is why I extracted the music from the PS2 version of the game, which is much easier to extract.
That is the music that can be found on RayTunes.
You could try to ask support for that type of sound on this forum in this topic, though.
More experienced people can reply to this topic if they know how to play these ADPCM files. Don't expect much answers, though...

Drolpiraat.
gmsj0001
Électoon
Posts: 20
Joined: Wed Jul 23, 2008 1:26 pm
Location: Shanghai, China
Contact:
Tings: 0

Re: R3: How to extract the background music?

Post by gmsj0001 »

I've seen the post on XeNTaX several days ago by Google Search. Now it seems that we all have no ideas about the file...

But the file is correct. I use Filemon to see what Rayman3.exe do when I play and find that it really read the correct block of data.hst. Perhaps Ubi use a secrect function to coding the music. (perhaps mp3... every block ends up with the string "ump3") Can we search the computer RAM for some useful things?

Perhaps four years later, I graduate from the university and learn more about computer, such as ASM and DASM, memory...
gmsj0001
Électoon
Posts: 20
Joined: Wed Jul 23, 2008 1:26 pm
Location: Shanghai, China
Contact:
Tings: 0

R3: I've extracted one playable music from data.hst but...

Post by gmsj0001 »

I've extracted the 2d madness music from the data.hst file. After using WinHex to change the compression code from 02h to 01h, the music is playable. It is the correct music, but full of nosie. I think I've made some progress on data.hst because I really heard the music. But the noise is really...

I think that the data in data.hst may be mp3 format because the data is in many frames. One frame is 66Ah in size. I think perhaps we can learn some sound store formats, that we can know what's the correct compression method?
Droolie
Mr Sax
Posts: 5162
Joined: Wed Apr 20, 2005 7:31 pm
Tings: 61830

Re: R3: I've extracted one playable music from data.hst but...

Post by Droolie »

gmsj0001 wrote:I've extracted the 2d madness music from the data.hst file. After using WinHex to change the compression code from 02h to 01h, the music is playable. It is the correct music, but full of nosie. I think I've made some progress on data.hst because I really heard the music. But the noise is really...

I think that the data in data.hst may be mp3 format because the data is in many frames. One frame is 66Ah in size. I think perhaps we can learn some sound store formats, that we can know what's the correct compression method?
Yeah, I know, it's full of noise if you read it as RAW data as well / change the header of the file. Now you're at the point where I got stuck a few years ago... :(
And I don't think it's mp3 format... the music quality wouldn't be that good if it was. And I don't think they used any ( commonly ) known format... :|

And please, don't make so many topics for this. I'll merge the two topics you made... Done.
gmsj0001
Électoon
Posts: 20
Joined: Wed Jul 23, 2008 1:26 pm
Location: Shanghai, China
Contact:
Tings: 0

Re: R3: Music Extraction Discussion Topic

Post by gmsj0001 »

Thank you...

Er... It's not mp3 format... It's ADPCM... If I make a Mp3 Wave format file and read it as raw data, I cann't hear the music any more. But If I make a MS-ADPCM wave file then read it as raw, I can hear the music inside... of cause full of nosie and four times faster...

It is said the ADPCM encoding method can be modify by yourself, So R3 must have its own ADPCM encoding method... I consider if I can read some useful tip from the frame header... Hummmmmmm....
Droolie
Mr Sax
Posts: 5162
Joined: Wed Apr 20, 2005 7:31 pm
Tings: 61830

Re: R3: Music Extraction Discussion Topic

Post by Droolie »

Wait, I have an idea.
Could you send me that playable *.wav file you created?
With Yousendit, for example? ( my email address is [email protected] )
I might have found something that could make the noise disappear...
gmsj0001
Électoon
Posts: 20
Joined: Wed Jul 23, 2008 1:26 pm
Location: Shanghai, China
Contact:
Tings: 0

Re: R3: Music Extraction Discussion Topic

Post by gmsj0001 »

I've sent them to you now.

I think what we shoud do is not deal with the noise, but crack the ADPCM encode and decode method made by Ubi. The noise when read an ADPCM format file as raw is a normal thing, and four times faster is normal too because ADPCM compression rate is about 1/4.

If we discover the decode method, we are succeed...
Beebop
Électoon
Posts: 22876
Joined: Sat Jan 03, 2004 11:56 am
Location: Home, England
Contact:
Tings: 5

Re: R3: Music Extraction Discussion Topic

Post by Beebop »

gmsj0001 wrote:I've sent them to you now.

I think what we shoud do is not deal with the noise, but crack the ADPCM encode and decode method made by Ubi. The noise when read an ADPCM format file as raw is a normal thing, and four times faster is normal too because ADPCM compression rate is about 1/4.

If we discover the decode method, we are succeed...
I am sure it will work out.
Droolie
Mr Sax
Posts: 5162
Joined: Wed Apr 20, 2005 7:31 pm
Tings: 61830

Re: R3: Music Extraction Discussion Topic

Post by Droolie »

Yeah, I guess. The Rayman 3 for GameCube soundtrack ( which has more tracks, possibly? It's the original version after all... ) has a higher priority for me though.
Synthesis
Barbara
Posts: 47
Joined: Fri Jan 02, 2009 7:48 pm
Location: France
Tings: 10235

Re: R3: Music Extraction Discussion Topic

Post by Synthesis »

Sorry for posting a reply so long after the previous one.


Maybe like a lot of Rayman2/3 fans, I liked the soundtrack a lot, and really wanted to get the original music files. So I managed to code a little command-line tool to get the Rayman 2 soundtrack, encoded in ADPCM format. The program opens any .apm file and converts it to a standard .wav file, readable with any media player.
The only thing left to do is to merge the created .wav files to get a complete song (as you may know, the soundtrack of Rayman 2 is sliced in parts of 1 or 2 bars, so that the music changes depending on where you are in the level).
Some parts of songs are not played at all in the game ! (for example song endings)

I'll post the executable here in a few days.
Maybe I'll write another tool to get the .bnm files extracted, but I'm not planning to do it now.
=> Already done by Rene Smit, the tool's called F1BNM.


I've discovered Rayman 3 only a few days ago, and I've already taken a look in all these .bnh and .hxc files. Here are the results of the discoveries (maybe some of you have already noticed that):

The .bnh files are just text files renamed to .bnh, and each line of the file looks like:
cuuid( [event type, in hex], [event ID, in hex] ) * fixe [yes/no] * level [yes/no] * transition [yes/no] * file [filename]
The corresponding .hxc files seem to be structured binaries: the file names in the bnh files appear in these files. For each file, there is the name of an event, and a file name. I still don't know what the other bytes mean.

The musics/soundeffects are not in these files. They are all packed in 2 files:
Data.hos [~40 MB], which is, just like .bnm files, a concatenation of standard .wav files;
Data.hst [~498 MB], which contains a lot of sound files in .wav format, and encoded in Microsoft ADPCM.

For those like me who can't manage to play these files, I'm planning to make an extractor-decoder.
The only thing is that I'd like to have the right names for the sound files that will be extracted (at least something more interesting than "sample0001.wav").
I know the file names are in the .bnh/.hxc files, but... which name corresponds to what file?

One more thing. For those who are interested in decoding ADPCM files, I've found good algorithms on MultimediaWiki (they are the ones I use): the standard IMA-ADPCM format and how to decode it, and Microsoft's ADPCM format (0x02 in wav files).

Sorry for this --really-- long reply, but I hope this will help...

EDIT: Corrected some mistakes in this post, now that I know more about BNM and HXC (these formats are alike).
Last edited by Synthesis on Sun Jan 25, 2009 6:37 pm, edited 1 time in total.
Synthesis
Barbara
Posts: 47
Joined: Fri Jan 02, 2009 7:48 pm
Location: France
Tings: 10235

Re: R3: Music Extraction Discussion Topic

Post by Synthesis »

Finally, here it is!
Ray2Get, the command-line ADPCM decoder for Rayman 2 .apm files!

It took me a little time to finish it, and it still doesn't convert mono files properly (that's why "outro1.apm" is not in the .bat file).

Download link:
Ray2Get (zip, 13.6 kiB) (removed for terms of service violation or something)
Ray2Get (zip, 9.45 kiB) (new .bat file included)

For those who are interested in having an mp3 version of the original soundtrack:
Rayman 2 OST - Pack 1 (zip, 78.47 MB)
Rayman 2 OST - Pack 2 (zip, 79.15 MB)


I'm currently starting to code the Rayman 3 version of this tool...
Last edited by Synthesis on Sun Jul 10, 2011 9:41 pm, edited 1 time in total.
Droolie
Mr Sax
Posts: 5162
Joined: Wed Apr 20, 2005 7:31 pm
Tings: 61830

Re: R3: Music Extraction Discussion Topic

Post by Droolie »

Image

Wow, just wow! It's absolutely amazing! And you included all the unused ends of the tracks ( and an unused beginning for some of them! )! :D
You deserve 5000 tings for this. Here they are!
You'll get 5000 more when you release the R3 tool. :D

Can I put this on RayTunes instead of the current one? ;)

EDIT -> I was right when I thought that I had heard track 89 ( the "ray" bonus track ) before... it's a jingle from the N64 version!
Also, it seems that the Precipice - Part 2 is completely new! o_O

EDIT 2 -> I love the beginning of the Precipice - Part 1. I would have used it as the music of an introduction of the level. :)
Attachments
ray2get.zip
Ray2Get, the command-line ADPCM decoder for Rayman 2 .apm files!
(13.66 KiB) Downloaded 498 times
Synthesis
Barbara
Posts: 47
Joined: Fri Jan 02, 2009 7:48 pm
Location: France
Tings: 10235

Re: R3: Music Extraction Discussion Topic

Post by Synthesis »

Thanx Drolpiraat for the Tings!

Maybe you should not replace RayTunes's current Rayman 2 Soundtrack (the quality is better than my PC-version rips).
But you can add my files in a new "category" (or in "Other Rayman Soundtracks", it's up to you).

Actually my external hard-disk crashed a few months ago, and I had to re-code it all from scratch! But this second version creates standard wav files (the previous one only created .raw files that I had to convert to wav).

__________
I've also started to extract sound effects from Rayman 2 .bnm files (using F1BNM), but there are so many BNMs, and I can't automate the extraction process. But it'll be completed in a few weeks, maybe.

__________
Now I'm working hard on Ray3Get.
I've already "reverse-engineered" the HXC file format and wrote a .txt file containing some of its specs (just ask me if you want it).

The program now extracts the standard PCM files in Data.hos (MS-ADPCM will be implemented soon).
The final version will extract every file indexed in the .hxc files (actually yeah, they're indexes). So we'll have musics and also sound effects, with their original file names!
I really want to have it finished...

__________
EDIT: Sorry, I didn't mention it:
The "ump3" tags that can be seen in Data.hst's adpcm files have nothing to see with mp3 format, nor m3u or so.
These seem to be special subchunk tags for wav files, just like "data", "datx", "plst", "list", "labl" and so on... For more info, check out The Sonic Spot.

EDIT 2: Sorry for these over-long posts, I have lots of things to say :D
Synthesis
Barbara
Posts: 47
Joined: Fri Jan 02, 2009 7:48 pm
Location: France
Tings: 10235

Re: R3: Music Extraction Discussion Topic

Post by Synthesis »

Whew!
Didn't know the ADPCM files could be so hard to rip!

In fact, the files are split into very small parts (exactly 1536 bytes each).
That's why there are a lot of "cue ", "LIST" and "labl" tags at the end of each file. For more info, see link in previous post.
Synthesis
Barbara
Posts: 47
Joined: Fri Jan 02, 2009 7:48 pm
Location: France
Tings: 10235

Re: R3: Music Extraction Discussion Topic

Post by Synthesis »

News about ray3get:

The format is really tricky to decode: it's not basic IMA-ADPCM format.
It seems to be something like MS-ADPCM.

Do you know any form of compression for which a flat signal is coded by value 0x7 ?

Also, I still don't know what the block headers are used for...
Droolie
Mr Sax
Posts: 5162
Joined: Wed Apr 20, 2005 7:31 pm
Tings: 61830

Re: R3: Music Extraction Discussion Topic

Post by Droolie »

Synthesis wrote:Do you know any form of compression for which a flat signal is coded by value 0x7 ?
I wish I knew, but sadly, I don't. :(

I will give you another 5000 tings for your hard work though ( and for the great silence-removing tool you made ). Here they are! ;)
Synthesis
Barbara
Posts: 47
Joined: Fri Jan 02, 2009 7:48 pm
Location: France
Tings: 10235

Re: R3: Music Extraction Discussion Topic

Post by Synthesis »

Thanx again.
Can't wait for the RayTunes site to be updated :)
Drolpiraat wrote:
Synthesis wrote:Do you know any form of compression for which a flat signal is coded by value 0x7 ?
I wish I knew, but sadly, I don't. :(
I think it's MS-ADPCM-like, but this format is quite complicated, and when I try to decode I get either distortion or some almost-silent noise.
I'm sure the key is the block headers, but I still don't know what the bytes corresponds to. I googled for some kind of specification, but never found anything that really fits...
stan423321
Tarayzan
Posts: 7312
Joined: Sat Jan 12, 2008 3:22 pm
Location: Poland, of Union of Socialistic European Republics
Tings: 63250

Re: R3: Music Extraction Discussion Topic

Post by stan423321 »

In Poland, it is legal to disassmeble program to see what it do. Maybe go to Poland, disassemble R3 part responsible for playing songs and rewrite it in your programming language?
Synthesis
Barbara
Posts: 47
Joined: Fri Jan 02, 2009 7:48 pm
Location: France
Tings: 10235

Re: R3: Music Extraction Discussion Topic

Post by Synthesis »

Well, that's a really good idea, but I wish I had enough knowledge about asm...
If anybody here could help...
Post Reply