CRP file Unpacker prototype

[tony56a1 79]

If anyone's interested( probably not lol, since Modtools does all of this in-game anyways), I've built a small prototype CRP unpacker, so you can extract things like meshes, textures, and some metadata out of the CRP file without having to open the game. Unfortunately, since I can't import any of the Unity libraries, whoever builds the next version should probably move this over to a unity app instead of the current solution( plain old .NET app ). 

In order to use this, you'll need .NET framework 4.5, and to copy Assembly-CSharp.dll, ColossalManaged.dll, ICities.dll, and UnityEngine.dll into the app folder. Once you've done that, just drag the .crp file into CrpParser.exe, and it should output a folder containing:

• Steam Preview images( if any )(.png files)
• Object Meshes(.obj files)
• Textures(.dds files)
• LOD meshes/textures( if any )
• some object metadata( asset name, steamid of the author, etc )( .json files)
  
  Download link here: https://github.com/tony56a/crp-parser/releases/download/v1.0.3/CrpParser.zip
  Source link:https://github.com/tony56a/crp-parser

Hopefully, we can get some interested people working on something for writing a standalone CRP/content packer,  so we can get pack stuff like animated textures and the like.

[Judazzz 1,936]

Holy moly dude, that's beyond awesome, pretty much what I was looking for!
Don't have time right now, but I can't wait trying it out.

[Judazzz 1,936]

Exporting the various files works fine (as far as my newbie eye can tell), but unfortunately there's nothing I can do with the .obj file that's being spit out by your tool.
Importing a new asset based on the .obj file (I have a "{obj-filename}_d.png" texture file in same folder) doesn't work - it shows up in the import list, but I get neither preview nor can I continue (button disabled), so I reckon it needs to be converted first. I've tried Autodesk's FBX Converter, but it's not working for me: I get a 'Not enough parameters' message and that's the end of it  

Am I missing something here?

Edit: I suspect there may be something with the exported .obj file. When I use ModTools to dump the mesh of the same prop, I can convert it to .fbx with FBX Converter without problems. However, importing that .fbx gives me nothing (still no preview, and the 'Continue' button is clickable once, then disappears...)

[Darf 1,439]

You can't convert it 1:1. The import adds some vertexes at the bottom for slopes. If you import it in 3ds max you can easily convert it to fbx.

[Judazzz 1,936]

All right, thanks for the info. What I need for some new features for American RoadSigns is a set of import-ready .fbx files (road sign props) so I can create additional ones to include in the mod (basically re-textured versions of the signs already included), and I hoped to be able to achieve that with a few (simple) tools. Knowing that it takes more than that, I guess I'll have to to rethink my approach (I have a sneaking suspicion that importing and exporting with 3D Max won't be as simple as just that, which means having to spend more time for trial-and-error, reading additional resources, etc. - something I have no desire to do).

[tony56a1 79]

4 hours ago, Darf said:

You can't convert it 1:1. The import adds some vertexes at the bottom for slopes. If you import it in 3ds max you can easily convert it to fbx.

Similarly, importing to Blender seems to work OK for me, there are some settings you'll have to configure to make the model game-ready (as I recall, mainly just moving the center of the model ).

[Darf 1,439]

Nah, you got to invert one of the axis of the pivot of the model and cut off the bottom row of vertices. You can adjust the pivot or import using the center option. You also have to split one of the maps using colorchannels.

[Tailgunner 8]

I have tinkered around with the code a bit. My fork mainly introduces a AssetInfo object that gets returned by parseFile and bundles the parsed data.

So you can say

var info = new CrpDeserializer(path).parseFile();
var name = info.metadata["name"];
var preview = info.images[1];

etc.

Here's the link: https://github.com/clausthaler-devel/crp-parser/tree/master/CRPParser

[Tailgunner 8]

And some more tinkering. While above tinkerings focus the developer sense of smugness, the following changes will appeal to the common user.

The code now contains a windows app, as well as a (now working) command line tool to extract crp files. With both you can chose the output directory.

Also some minor improvements to the code. Oh and I added a fincy fancy belly whistely installer for the clicky people.

screenie:

 

[Tailgunner 8]

And some more tinkering. I removed the dependency of the ImageMagick Library, it was 10 MB fat! Now uses byte[]. new installer: https://github.com/clausthaler-devel/crp-parser/blob/master/Installer/CRP-Extractor-0.2-Setup.exe?raw=true

  Edited July 6, 2016 by Tailgunner  

[Judazzz 1,936]

Sounds like a great optimization, but when I try to extract a .crp file, I get an ImageMagick-related error ('Could not load file or assembly 'Magick.Net-Q16-AnyCPU ....').
Also, you accidentally linked to the 0.1 version (this is the correct url: https://github.com/clausthaler-devel/crp-parser/blob/master/Installer/CRP-Extractor-0.2-Setup.exe?raw=true - your link text is correct, the href links to the 0.1 version).

[Tailgunner 8]

Works fine for me. There are no references to the Magick library anymore, so they hardly can throw an error. I'm pretty sure something has been mixed up during install. Try re-installing while making sure the extractor isn't running. 

Meh, I packed outdated files into the installer. That is now corrected.

Also: corrected link above.

[tony56a1 79]

Looks great! Thanks so much for the GUI!

A note about the ImageMagick dependency though, it's in the first revision so that the app can convert the dds textures to png, in case anyone using the app can't use a dds viewer/editor or something. Something to think about...

On 7/6/2016 at 11:32 AM, Tailgunner said:

And some more tinkering. I removed the dependency of the ImageMagick Library, it was 10 MB fat! Now uses byte[]. new installer: https://github.com/clausthaler-devel/crp-parser/blob/master/Installer/CRP-Extractor-0.2-Setup.exe?raw=true

[Tailgunner 8]

those textures are png format anyway. turning the byte array into a usable image is as simple as

        private static BitmapImage LoadImage( byte[] imageData )
        {
            return LoadImage( new System.IO.MemoryStream( imageData ) );
        }

        private static BitmapImage LoadImage( Stream stream )
        {
            try
            {
                var i = new BitmapImage();
                i.BeginInit();
                i.StreamSource = stream;
                i.EndInit();
                i.Freeze();
                return i;
            }
            catch
            {
                return null;
            }
        }

[Netuno 97]

8 hours ago, tony56a1 said:

Looks great! Thanks so much for the GUI!

A note about the ImageMagick dependency though, it's in the first revision so that the app can convert the dds textures to png, in case anyone using the app can't use a dds viewer/editor or something. Something to think about...

 

With version 0.2, some png images are not displayed and they can't be opened.

[Tailgunner 8]

Mmh. Allright, I will think about that.

[tony56a1 79]

12 hours ago, Tailgunner said:

those textures are png format anyway. turning the byte array into a usable image is as simple as

        private static BitmapImage LoadImage( byte[] imageData )
        {
            return LoadImage( new System.IO.MemoryStream( imageData ) );
        }

        private static BitmapImage LoadImage( Stream stream )
        {
            try
            {
                var i = new BitmapImage();
                i.BeginInit();
                i.StreamSource = stream;
                i.EndInit();
                i.Freeze();
                return i;
            }
            catch
            {
                return null;
            }
        }

Depending on the version of the package(probably, I only tested like 4-5 assets[also, sort of why I'm weary about adding unit tests where the test is just parsing a file]), the textures may be stored as a DDS file, preview images are always PNGs though. You probably don't need that dependency if you want save the file as-is, but I found it inconvenient to have to open visual studio or something to verify them.

[Tailgunner 8]

allright i will change it to byte[] for the previews and back to magick for the textures and make it load lazy so it only croaks when somebody accesses textures and the magick lib isn't there. that way you can keep the parser slick and when somebody needs the texture stuff /(s)?he/ can add the magick lib. i'll do installers with and without.

btw, tony if you want to  continue on, please feel free to merge my changes. I refactored quite a bit and it's quite more friendly for unit tests imho. edit: oh, and because i added one unit test already (along with the test project).

[Frontwards 0]

Could this work for color correction LUTs as well, eventually?.

[tony56a1 79]

Latest commit adds this functionality. I'll release a new binary after getting some other changes in.
 @Tailgunner: Looks good! Can you submit them as a pull request? Makes it easier for me.

[Frontwards 0]

Looking forward to it, thank you.

[ruoy 2]

Man this is awsome! I uploaded a asset about a week ago and then lost everything (Game Install, Asset Meshes and Textures) because of a Trojan that royally screwed my PC up. 

 

With this tool now I have a chance of getting it back and make an update that users have asked for. 

 

Cheers

[Netuno 97]

I wish this application also had the reverse operation - packer

[Frontwards 0]

Hello, are you still going to release a binary at some point? could really use that LUT feature still. Thanks.

[Ronyx69 2,413]

7 hours ago, Frontwards said:

Hello, are you still going to release a binary at some point? could really use that LUT feature still. Thanks.

Pretty sure you can dump the LUT with modtools somewhere in camera controller or some kind of render manager, there is a color correction manager or properties thing.

[OLFced 0]

Hello,  i could not view or edit .json  on my blender. is any way for converse .json to .fbx or .obj?

[boformer 3,169]

6 hours ago, OLFced said:

Hello,  i could not view or edit .json  on my blender. is any way for converse .json to .fbx or .obj?

json files are not model files. They only contain plain text properties, like descriptions. You can open them with notepad.

[charlesnew 262]

Hi I've been trying to use the extractor but it comes up with an error saying it "could not load file or assembly". Is there any fix to this? I notice that Judazzz has had a similar issue and something got fixed, but I still have it.

EDIT: Forgot to mention, I tried a reinstall, and the error isn't related to ImageMagick

[Jnkyrd 0]

Everything seems to be unpacking correctly except I don't know what to do with this file -> Jnks Casino Lot One_Data_3_obj.bin

I assume this is the .obj file? I don't understand the .bin

Can't open it with anything that I have tried. Can't convert it either.

Please help me..

 

Jnk

[Ronyx69 2,413]

6 minutes ago, Jnkyrd said:

Everything seems to be unpacking correctly except I don't know what to do with this file -> Jnks Casino Lot One_Data_3_obj.bin

There should be a seperate regular .obj file which is the mesh.