Object persistence
In order to simplify the process of loading and saving data of various types to and from the user's hard-drive, ZenGin implements a simple object persistence system using the zCArchiver
class and its derivatives that allow the individual engine classes to implement a routine specifying which data should be saved or loaded from disk and in which manner.
An object that is derived from the zCObject
class may overload the Archive
and Unarchive
virtual methods. The class may then call on an interface provided by the zCArchiver
class within these methods which allows it to directly read from or write to a stream using several modes. Those are ASCII and BinSafe by default. There are, however, more options, as is explained below.
Archive format
In order to better understand how this process works, it would be best to look at an example of a .ZEN file containing an instance of an oCWorld
object.
Header
When you open up a ZenGin archive, you will see the following at the start of the file:
Let's look at each of these properties and what they mean:
ZenGin Archive
This simply specifies that the following data is an zCArchiver
archive.
ver 1
Version specification. Can be either 0
or 1
. Both Gothic 1 and 2 are already on version 1, although version 0 archives can also be occasionally found.
zCArchiverGeneric
Specifies which derived zCArchiver
class should be used to read this archive. Accepted values are zCArchiverGeneric
for ASCII and Binary archives, and zCArchiverBinSafe
for BinSafe archives. More info below.
This property might not be present in older archives.
ASCII
This is the most important part of the header as it specifies in which format should the data be stored. There are 4 different modes:
- ASCII - The simplest one. It stores data in human-readable ASCII notation (not unlike JSON for example). This is usually used when saving data during development and/or testing, while the final version of said data will most likely be stored as BIN_SAFE.
- ASCII_PROPS - Same as ASCII except with more additional data that the developer can specify for visual clarity. In practice, it is not used anywhere and mostly serves only to prettify debug info (try typing
ZWORLD VOBPROPS
in the console and look in zSpy ;) ). - BINARY - Binary representation of the class instance, which mostly copies the data 1:1 into/from the stream. In practice, this format is only used to store savefiles (.SAV).
- BIN_SAFE - BinSafe, short for Binary Safe, is an extended version of Binary which stores type information along with the data itself. This is meant to make error checking for invalid data easier. There are other changes which are explained below. Most, if not all world files (.ZEN), are stored in this format.
saveGame 0
Specifies if this archive is a savefile. This property might not be present in older archives.
date 7.1.2001 23:9:19
The date at which this archive was created.
user roeske
The user which created the archive. This property might not be present in older archives.
END
Tells the parses that this is the end of the header.
We may additionally find a property called csum
in version 0 archives which stores the checksum of the whole archive. This property is, however, unused and equals 00000000
by default.
In order to correctly read the archive's header across varying engine versions, one should not count on the properties always being in the same order or even being there at all.
If the archive utilizes zCArchiverGeneric
then this header will also be followed by a short section specifying the number of object instances in this archive. This value will be used to initialize the objectList, which is an array of pointers where the addresses of loaded objects will be stored for later referencing. This property would be directly part of the main header in older versions.
If the archive is created using zCArchiverBinSafe
, this data will be stored in the following binary structure:
Contents
Looking further into the archive, we see what appears to be a nested structure.
We primarily differentiate between chunks and properties within ZenGin archives:
Chunks
A chunk is a structure that groups properties together. For most of the time, a chunk represents a class instance. This is, however, not always true as classes may arbitrarily create chunks as is needed. For example, the sample above contains a chunk called VobTree
, which does not represent a class instance, but only serves to make the reading of the archive easier.
While in ASCII mode, the start of a chunk is represented using square brackets.
[% oCWorld:zCWorld 64513 0]
There are 4 pieces of data separated by spaces inside the start of each chunk, which are:
- Object name - The name of the chunk to use while reading. If the chunk has no name, then it will be simply equal to
%
. - Class name - The name of the class which this chunk represents. Class names are stored with their full inheritance hierarchy (e.g.
oCMobLadder:oCMobInter:oCMOB:zCVob
). In case the chunk is not an object, but an arbitrary chunk, then this field will be equal to%
(%
can also mean that this chunk is a nullptr). In some cases you may encounter the symbol§
instead. This means that the object already exists and that the parser should look for it in the objectList using the object index. Using this mechanism, a single instance can be referenced multiple times without worrying about duplicity. - Class version - Used to ensure that the data being read is compatible with the current game/engine version, so that there are no mismatches in the data pattern. This value is different for every class and varies between game versions.
- Object index - An index into the objectList under which this object will be stored. If the class name is equal to
§
, then this value will be used to retrieve an existing instance from the objectList.
If this is a Binary archive, the same data will be stored in the following binary structure:
Oddly enough, if the archive is BinSafe, then the data will be encoded the same way as in ASCII mode, except that it will be stored as a type-checked property.
In ASCII mode []
represents the end of the current chunk.
Properties
We find properties inside the chunks which are key-value pairs that classes use to store the actual data. Each property stores its name, type and value. In ASCII mode the format for this isname=type:value
.
For example:
visual=string:SURFACE.3DS
By default, zCArchiver
allows to store properties of the following types:
-
Int - A regular 32-bit integer. In ASCII mode, int gets stored as
name=int:1
, while in Binary mode, it's just the raw value stored as 4 bytes. -
Byte - A 8-bit integer. ASCII mode doesn't differentiate between Int and Byte, so this will be stored as
name=int:1
regardless. Binary mode stores only the single byte. -
Word - A 16-bit integer. ASCII mode doesn't differentiate between Int and Word, so this will be stored as
name=int:1
regardless. Binary mode stores only the 2 bytes. -
Float - A standard IEEE 754 32-bit floating point number. In ASCII mode the format is
name=float:1.0
, while in Binary mode the float gets stored raw as 4 bytes. -
Bool - Stores a single-byte boolean value. In ASCII mode its
name=bool:1
and in Binary mode it's a single byte. -
String - An ASCII encoded string. While in ASCII mode, strings are stored as
name=string:value
. In Binary mode, strings are NULL terminated. -
Vec3 - A three component vector, mainly used to store positional data. The ASCII mode format is
name=vec3:1.0 1.0 1.0
. In Binary mode the three components of the vector are stored in series, which equals to a total size of 12 bytes. -
Color - A 32-bit color value stored as BGRA. In ASCII mode the color is stored as
name=color:255 255 255 255
while in Binary mode it's just 4 raw bytes. -
Raw - Raw binary data. In order to maintain readability, in ASCII mode this gets stored as a hex encoded string such as
name=raw:63D15B07
. In Binary mode, only the data itself is stored, without any other info. Be aware that due to this you must know the size of the data beforehand. -
RawFloat - An array of floats, mainly used to store bounding boxes. In ASCII mode, the floats are stored as
name=rawFloat:1.0 1.0 1.0 1.0 1.0 1.0
. In Binary mode the floats are stored in series as raw bytes. Same as withRaw
, you must know the size of the array beforehand. -
Enum - An enum value. In ASCII mode, it gets stored as
name=enum:1
. In Binary mode, it behaves the same asInt
.
As you might have noticed, binary mode doesn't perform any kind of checks on if it's reading the right property or even data of the correct type. This is why BinSafe mode exists, as it stores the property type in along with the data itself.
Looking at the enumeration of types, you might notice that BinSafe mode has an additional property type called Hash. BinSafe archives include a hash table which is stored in the following manner:
Instead of storing the raw value, properties may save a hash instead, which is then used to look up the corresponding value from the hash table.
Implementation
As mentioned in the opening paragraph, classes may use the described functionality by overloading the Archive
and Unarchive
virtual methods, which pass an instance of zCArchiver
by reference. When the class instance is then serialized and/or parsed, these methods are called and perform the desired serialization/parsing work.
The class uses methods provided by the zCArchiver
instance within these routines. These methods return/accept a value of a specific type (e.g. ReadInt/WriteInt), while they do the actual reading/writing work behind the scenes based on the current mode (ASCII/Binary/BinSafe). The programmer writing the class then does not care whether the final archive will be saved as ASCII, Binary or BinSafe, as they only use the zCArchiver
Read* and Write* methods.
A practical example
Let's propose that we have a class which is declared like so:
The hypothetical class then implements these virtual functions:
We then initialize the class in the following way:
If we now serialized, or to use the engine's term "archived", this instance into an ASCII archive, the result would look like this:
Notice how secondPointerToMyObject
doesn't have any contents. The character §
tells the parser that this object already exists in the objectList, and that instead of creating a new instance, it should return an existing instance which is stored under index 1
in the objectList.
This allows an instance to be referenced from multiple places, without the need to worry about duplicity.
If we used Binary or BinSafe mode, we would see a big blob of binary data instead. This would, of course, store the exact same data, although in a slightly less human-readable format.
Final thoughts
We hope this helps you better understand the inner workings of ZenGin. If you want to see how Piranha Bytes went about implementing a much more advanced version of this system for their next engine, check out Genome's object persistence system.