Super Easy to Use 3D Engine (Su3De) – Binary File Format Part 3 – File Format

In part 2, we talked about vertices in the Su3De binary model format. In this post I want to talk about the “overall” structure of the binary file and then add texture coords and texture maps to the format…in the NEXT post.  I know I said we would talk about textures here, but I got ahead of myself (hard hat area – be flexible and watch out for falling ideas).  We still need to talk a little more about the basic binary format first.  Jumping right in, the Portable Network Graphics (PNG) file format is an exceptionally well thought out binary format and was designed primarily with “network transmission” in mind and as a result has quite a few features geared to that. The PNG format is also equally well suited to being stored on a file system. You can probably tell by now where I am going with all of this talk about PNG. The plan is to use the PNG file format as a basic blueprint for the Su3De binary model format. You can learn more about the PNG format here. If you are knowledgeable in the ways of the PNG, the following will be eerily familiar to you.

Su3De File Header

In all good binary file formats, there should be a few bytes in the beginning that tell you that what you think you are reading is indeed actually what you are reading. For PNGs, this sequence is “0×89, 0×50, 0x4E, 0×47, 0x0D, 0x0A, 0x1A, 0x0A”. These 8 bytes tell you convincingly that you are dealing with a PNG file. Take a look at the hex file dump below of a very small PNG file.

PNG File Hex View

PNG File Header Table

Bytes Purpose
0×89 The high bit is set which is useful to identify a network transmission environment that supports 7-bit instead of 8-bit characters.  In this case, you would see a 0×09 and know that the rest of the file is bogus.
0×50, 0x4E, 0×47 The three ASCII characters representing the characters “PNG”.
0x0D, 0x0A ASCII “Carriage Return” (0x0D) and a “Line Feed”  (0x0A)
0x1A ASCII “Substitute” character which also happens to be a DOS text file “end of file” character
0x0A Another ASCII “Line Feed” (0x0A) perhaps for Unix / Linux text file environments?

A good bit of thought went into this sequence to insure that network transmission systems don’t fiddle with the data as it is transmitted and received.  For example, some file transfer systems change the 0x0D(Carriage Return), 0x0A(Line Feed) into just a Line Feed for example – stripping out the Carriage Return.  Some file transfer systems might ADD a Carriage Return to to the header ending Line Feed… At any rate, if you read the first 8 bytes and you get (in hexadecimal): 89 50 4E 47 0D 0A 1A 0A– then you can pretty much guarantee that the file isn’t being “tinkered with” – especially across a network.

So, what should the Su3De file header look like?  Should it copy the PNG format above or should it “be its own thing?”  I am highly inclined to go ahead and use the PNG file header format and simply replace the ASCII “PNG” with ASCII “SU3”. Why? The reason is simple…I intend to use many of the other elements of a PNG file as well moving forward…so this will make writing a SU3 “reader” simpler because you can use existing PNG reader code to read the SU3 files as well.

The Su3De file header will be:

0×89 – 0×53 – 0×55 – 0×33 – 0x0D – 0x0A – 0x1A – 0x0A

That was easy…wasn’t it?

Chunks of Data Goodness

Now, we are going to talk a bit about “chunks”. Chunks are “groups” of data that fall into a predefined category and are, for the most part, self contained “entities”. Chunks consist of a chunk header usually followed by a data field and ending with a CRC-32 value to insure that the chunk was read correctly (i.e. not corrupted in some way).  So the basic format of a PNG chunk (and now SU3 chunks) looks like the following:

[sourcecode language="c"]
typedef struct{
unsigned int chunkLength;
unsigned int chunkType;
unsigned char chunkData[];
unsigned int chunkCRC;
}CHUNK;
[/sourcecode]

chunkLength is a unsigned 32 bit value telling you how many bytes are in the chunkData field. A length of 0 means that there is no chunkData which is considered valid for some chunkTypes.

chunkType is a four character ASCII code.  The values can be any upper or lower case ASCII letter value – A-Z or a-z or any upper / lower case combination…although in some cases specific upper and lower case characters are meaningful – I will explain later.

chunkData is obviously the data that is meaningful relative to the chunkType. If chunkLength is 0, then there is no chunkData in the chunk.

chunkCRC is a CRC-32 value that protects the chunkType and chunkData fields.  This value is used to determine whether the chunk data that was read is valid or corrupted.  The CRC algorithm used (according to the “official” PNG specification) is defined by ISO 3309 and ITU-T V.42. Sample source code to generate this CRC value can be found here - among many other places I am sure.  It will, of course, be part of the Su3De engine.

NOTE: In PNGs, multi-byte sequences (shorts, ints, floats, etc.) are in big endian format.  In SU3 files, and as we discussed in a previous post, we will “stick to our guns” and use little endian for multi-byte sequences.

The great thing about this chunk architecture is that it is easily extensible.  You can add new chunks to the file format as needed without affecting previous readers / parsers.  If a reader doesn’t know about a particular chunk, it just ignores it.  As long as the “basic” required chunks are present…the model can be used. It is plain to see that you could easily add all kinds of features in the future without breaking the first binary format implementations.

Su3De Chunks

At this point, we are going to switch away from talking about PNG and start talking about Su3De’s binary model format SU3.  The chunk formats will be based on PNG and more about the actual PNG format (especially “chunking” can be found here and here.

Model Header Chunk

The first chunk that must be located in the SU3 is the header chunk identified as “MHDR“.  It is the first chunk in the file and will contain basic information about the model. It is a required chunk. In its initial defined form, it looks like this (subject to change – you are wearing your hardhat right?):

[sourcecode language="c"]
typedef struct{
unsigned int chunkLength;
unsigned int chunkType;
unsigned int faceCount;
unsgined int modelHints;
unsigned byte textureFormat;
unsigned byte vertexFormat;
unsigned int chunkCRC;
}SU3_MHDR;
[/sourcecode]

faceCount represents the number of faces in the model. Each face is comprised of three vertices with each vertex having a x, y and z coordinate. The vertices are defined in the MVTX chunk.

modelHints is a 32-bit bitfield that contains “hints” about the chunks that are in the SU3 model file.  This provides a kind of “heads up” to the model reader about the nature of the model.

textureFormat is a single byte value that indicates what format the texture data (if there is any) is in.  For example, the texture data could be 8-bit palettized, 16-bit direct color (555 or 565), 24-bit direct color, 32-bit direct color with alpha, some compressed data format, etc.

vertexFormat is a single byte value that indicates what format the vertex information is in.  Currently two formats are supported, floating point and fixed point.

Model Vertices Chunk

You read about this “chunk” in the last post, however, we weren’t calling them chunks then.  This chunk is identified as “MVTX” and is also a required chunk. By way of a refresher, here is the chunk format:

[sourcecode language="c"]
typedef struct{
float x;
float y;
float z;
}SU3_VTX3F;

typedef struct{
unsigned int chunkLength;
unsigned int chunkType;
SU3_VTX3F su3Vertices[3]; //x, y, z
unsigned int chunkCRC;
}SU3_MVTX;
[/sourcecode]

su3Vertices represent the polygonal mesh of the model.  Each entry in the SU3_VTX3 array represents a single vertex of a triangle.  The vertex entries are in x, y, z order so su3Vertices[0] would be the x entry of the first vertex of the first triangle, su3Vertices[1] would be the y entry of the first vertex of the first triangle, su3Vertices[2] would be the z entry of the first vertex of the first triangle, su3Vertices[3] would be the x entry of the second vertex of the first triangle…and so on.  It takes nine su3Vertices entries to fully describe one triangle.

Putting it all together “so far”…

Ok, so let’s use a basic 3D cube as an example.  Let’s build a SU3 file and look at it step by step.

First, we will have the file header:
0×89 – 0×53 – 0×55 – 0×33 – 0x0D – 0x0A – 0x1A – 0x0A

Next, we will have a MHDR chunk that will contain some key information:

[sourcecode language="c"]
SU3_MHDR su3mhdr;

su3mhdr.chunkLength = 10; //number of bytes in the data field
su3mhdr.chunkType = 0x5244484D; //ascii MHDR
su3mhdr.faceCount = 12; //a cube will have two triangles per side – and six sides
su3mhdr.modelHints = 0×0001; //bit 0 = mesh present (which is required)
su3mhdr.textureFormat = 0×0; //0 indicates no texture
su3mhdr.vertexFormat = 0×01; //1 indicates floating point format
su3mhdr.chunkCRC = crcval; //calculated CRC value for type + data fields
[/sourcecode]

After the MHDR chunk, we will have our vertex data in the form of a MVTX chunk:

[sourcecode language="c"]
SU3_VERTICES su3verts;
su3verts.chunkLength = numTriangles * (3 * 12); //3 verts per tri and (3 entries per vert (xyz) * 4 bytes per float
su3mhdr.chunkType = 0x5854564D; //ascii MVTX
//triangle 1
su3verts.vertices[0] = -10f; //tri 1 vertex 1 x
su3verts.vertices[1] = 0.0f; //tri 1 vertex 1 y
su3verts.vertices[2] = 10.0f; //tri 1 vertex 1 z

su3verts.vertices[3] = -10f; //tri 1 vertex 2 x
su3verts.vertices[4] = 0.0f; //tr1 1 vertex 2 y
su3verts.vertices[5] = -10.0f; //tri 1 vertex 2 z

su3verts.vertices[6] = 10f; //tri 1 vertex 3 x
su3verts.vertices[7] = 0.0f; //tri 1 vertex 3 y
su3verts.vertices[8] = -10.0f; //tri 1 vertex 3 z

//triangle 2
su3verts.vertices[9] = 10f; //tri 2 vertex 1 x
su3verts.vertices[10] = 0.0f; //tri 2 vertex 1 y
su3verts.vertices[11] = -10.0f; //tri 2 vertex 1 z

su3verts.vertices[12] = 10f; //tri 2 vertex 2 x
su3verts.vertices[13] = 0.0f; //tri 2 vertex 2 y
su3verts.vertices[14] = 10.0f; //tri 2 vertex 2 z

su3verts.vertices[15] = -10.0f; //tri 2 vertex 3 x
su3verts.vertices[16] = 0.0f; //tri 2 vertex 3 y
su3verts.vertices[17] = 10.0f; //tri 2 vertex 3 z

//…and so on for all 12 triangles
su3verts.chunkCRC = crcval; //calculated CRC value for type + data fields
[/sourcecode]

As with the MHDR chunk, a MVTX chunk is required.

The file ends with a chunk appropriately named MEND and it is a mandatory chunk with no data field that signals that “that’s all there is and there isn’t any more.”

[sourcecode language="c"]
typedef struct{
unsigned int chunkLength;
unsigned int chunkType;
unsigned int chunkCRC;
}SU3_MEND;

SU3_MEND su3End;

su3End.chunkLength = 0;
su3End.chunkType = 0x444E454D; //ascii MEND
su3End.chunkCRC = crcval; //calculated CRC value for type + data fields
[/sourcecode]

Quick and Easy Development Environment

I have started using the Bloodshed Dev-C++ development environment for quick code prototyping and testing.  Why…because it is AWESOME that’s why – despite the weird name.  You can find it here.  I use the Dev-C++ 4 version and haven’t tried Version 5 yet.

I wrote a “quick and dirty” SU3 file writer that will write out a 3D cube to a SU3 file using all the chunk stuff we talked about above.  This was done on a Windows XP system…but the code should compile pretty much anywhere. Here is the full source code to that writer:

[sourcecode language="c"]
#include <stdio.h>
#include <stdlib.h>

#define MAKEFOURCC(ch0, ch1, ch2, ch3)
((unsigned int)(unsigned char)(ch0) | ((unsigned int)(unsigned char)(ch1) << 8) |
((unsigned int)(unsigned char)(ch2) << 16) | ((unsigned int)(unsigned char)(ch3) << 24 ));

#define FOURCC_MHDR MAKEFOURCC(‘M’,’H’,’D’,’R’)
#define FOURCC_MVTX MAKEFOURCC(‘M’,’V’,’T’,’X’)
#define FOURCC_MEND MAKEFOURCC(‘M’,’E’,’N’,’D’)

#pragma pack(1)
/* structure of floating point xyz coordinates */
typedef struct{
float x;
float y;
float z;
}SU3_VTX3F;

typedef struct{
unsigned int chunkLength;
unsigned int chunkType;
}SU3_CHUNK;

typedef struct{
unsigned int faceCount;
unsigned int modelHints;
unsigned char textureFormat;
unsigned char vertexFormat;
}SU3_MHDR;

//header sequence including ‘SU3′
static unsigned char fileHeader[] = {0×89, 0×53, 0×55, 0×33, 0x0D, 0x0A, 0x1A, 0x0A};

static SU3_VTX3F FlexVerts[] = {
{-10.000000f, 0.000000f, 10.000000f}, //triangle 1
{-10.000000f, 0.000000f, -10.000000f}, //triangle 1
{10.000000f, 0.000000f, -10.000000f}, //triangle 1
{10.000000f, 0.000000f, -10.000000f}, //triangle 2
{10.000000f, 0.000000f, 10.000000f}, //triangle 2
{-10.000000f, 0.000000f, 10.000000f}, //triangle 2
{-10.000000f, 20.000000f, 10.000000f}, //triangle 3
{10.000000f, 20.000000f, 10.000000f}, //triangle 3
{10.000000f, 20.000000f, -10.000000f}, //triangle 3
{10.000000f, 20.000000f, -10.000000f}, //triangle 4
{-10.000000f, 20.000000f, -10.000000f}, //triangle 4
{-10.000000f, 20.000000f, 10.000000f}, //triangle 4
{-10.000000f, 0.000000f, 10.000000f}, //triangle 5
{10.000000f, 0.000000f, 10.000000f}, //triangle 5
{10.000000f, 20.000000f, 10.000000f}, //triangle 5
{10.000000f, 20.000000f, 10.000000f}, //triangle 6
{-10.000000f, 20.000000f, 10.000000f}, //triangle 6
{-10.000000f, 0.000000f, 10.000000f}, //triangle 6
{10.000000f, 0.000000f, 10.000000f}, //triangle 7
{10.000000f, 0.000000f, -10.000000f}, //triangle 7
{10.000000f, 20.000000f, -10.000000f}, //triangle 7
{10.000000f, 20.000000f, -10.000000f}, //triangle 8
{10.000000f, 20.000000f, 10.000000f}, //triangle 8
{10.000000f, 0.000000f, 10.000000f}, //triangle 8
{10.000000f, 0.000000f, -10.000000f}, //triangle 9
{-10.000000f, 0.000000f, -10.000000f}, //triangle 9
{-10.000000f, 20.000000f, -10.000000f}, //triangle 9
{-10.000000f, 20.000000f, -10.000000f}, //triangle 10
{10.000000f, 20.000000f, -10.000000f}, //triangle 10
{10.000000f, 0.000000f, -10.000000f}, //triangle 10
{-10.000000f, 0.000000f, -10.000000f}, //triangle 11
{-10.000000f, 0.000000f, 10.000000f}, //triangle 11
{-10.000000f, 20.000000f, 10.000000f}, //triangle 11
{-10.000000f, 20.000000f, 10.000000f}, //triangle 12
{-10.000000f, 20.000000f, -10.000000f}, //triangle 12
{-10.000000f, 0.000000f, -10.000000f}, //triangle 12
};
#pragma pack()

int main(int argc, char *argv[])
{
char *ptrVerts;
unsigned int crc;
SU3_CHUNK chk;
SU3_MHDR mHDR;

FILE *fp = NULL;

fp = fopen("c:\file.SU3", "wb");

fwrite(fileHeader, sizeof(fileHeader), 1, fp);

// build and write the model header chunk
chk.chunkLength = sizeof(mHDR);
chk.chunkType = FOURCC_MHDR;
fwrite(&chk, sizeof(SU3_CHUNK), 1, fp);
mHDR.faceCount = 12;
mHDR.modelHints = 0×01;
mHDR.textureFormat = 0×00;
mHDR.vertexFormat = 0×01;
fwrite(&mHDR, sizeof(SU3_MHDR), 1, fp);
crc = 0×12345678; //bogus placeholder value
fwrite(&crc, sizeof(crc), 1, fp);

//build and write the model vertex chunk
chk.chunkLength = sizeof(FlexVerts);
chk.chunkType = FOURCC_MVTX;
fwrite(&chk, sizeof(SU3_CHUNK), 1, fp);
fwrite(&FlexVerts, sizeof(FlexVerts), 1, fp);
crc = 0×12345678; //bogus placeholder value
fwrite(&crc, sizeof(crc), 1, fp);

//build and write the model end chunk
chk.chunkLength = 0;
chk.chunkType = FOURCC_MEND;
fwrite(&chk, sizeof(SU3_CHUNK), 1, fp);
crc = 0×12345678; //bogus placeholder value
fwrite(&crc, sizeof(crc), 1, fp);

system("PAUSE");
return EXIT_SUCCESS;
}
[/sourcecode]

Here is a hex dump of the file written by the code above:

SU3 File Hex Dump

If we had an actual reader / renderer (…like the Su3De engine itself), you would see the following:

Rendered Cube – as if from SU3 file

Wrap up for this post

We covered a lot of ground with this post.  In the next one, we will add texture coordinates and texture images to the SU3 format and start working on the .OBJ file converter that will build SU3 files from .OBJ and .MTL files. This will, of course, take several more posts to cover.  I hope you can see where we are going with all of this…and it is starting to make sense.

Thanks for reading and see you in Su3De!  Please add any comments if you think I am off in the weeds somehow and other things you might like to see here…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>