MDL file

From Wiki

Jump to: navigation, search
(Primitive Channels)
 
(58 intermediate revisions by 18 users not shown)
Line 1: Line 1:
{{Under construction}}
+
'''MDL''' is the file format employed by [[Sword of Moonlight]] when something is to be animated. It's counterpart is [[MDO]]. Tools were provided by [[From Software]] for working with MDO files. However, no analogous tools were ever provided for MDL. MDO appears to be incapable of locomotive animation, despite having some simple texture based animation ability, which MDL may well not enjoy. MDL does the rest.
__TOC__
+
 
 +
==History==
 +
 
 +
Despite having tools to work with MDO files, the format itself remains a black box. Since the MDL format on the other hand had been completely inaccessible for nearly a decade, greater priority has been given to figuring it out. The file format was first reverse engineered sometime in the latter half of 2010. There has been no official statement, however there has been a claim that on at least one occasion, a From Software customer relations spokesperson did state via email, paraphrasing, any attempt to reverse engineer Sword of Moonlight file formats is OK with the company.
 +
 
 +
Even though importers and exporters (export from MDL would come later) were developed around the same time the format had become well understood, there was no party invested enough in creating original MDL compatible models until the mid months of 2011. Efforts so far remain in the immature stages while a fully realized outcome appears inevitable.
 +
 
 +
==Characteristics==
 +
 
 +
An MDL file is a relatively simple, piecewise model, with limited texture mapping possibilities. There are two flavor of animation, [[Tinman]] and [[Scarecrow]]. Tinman animations move like an animated suit of armor, or a wooden puppet come to life. Scarecrow on the other hand, move like a sack doll, or anything really with some degree of flexibility.
 +
 
 +
==Specification==
 +
This section describes the file format in an intimate way, which the average reader might prefer to dismiss. It should be understood that the specification below, is more like documentation, and a reverse engineered one at that. Note that some more technical aspects may be offloaded to subpages linked to from within. If you want to make MDL files, it will probably be a favor to yourself to look this over well.
 +
 
 
<fieldset class="spec">
 
<fieldset class="spec">
 
<legend class="caps">
 
<legend class="caps">
==Header==
+
===Header===
 
</legend>
 
</legend>
 
{| class="wikitable"
 
{| class="wikitable"
 +
!colspan=11|16 8bit bytes in total
 
|-
 
|-
|LF||HA||SA||TB||PC||U0||-16-||PCSZ||-16-||HASZ||SASZ
+
|LF||HA||SA||TB||PC||00||-16-||PCSZ||-16-||HASZ||SASZ
 
|}
 
|}
 
<ifieldset class="spec">
 
<ifieldset class="spec">
Line 25: Line 39:
 
|PC:||primchans||''one or more required (meshes)''
 
|PC:||primchans||''one or more required (meshes)''
 
|-
 
|-
|U0:||unknown00||''perhaps always one''
+
|00:||unknown00||''perhaps always one''
|-
 
| -16-
 
 
|-
 
|-
 
|PCSZ:||primchanwords||''dword anims block offset (could be 32bit)''
 
|PCSZ:||primchanwords||''dword anims block offset (could be 32bit)''
|-
 
| -16-
 
 
|-
 
|-
 
|HASZ:||hardanimwords||''dword sizeof anims block''
 
|HASZ:||hardanimwords||''dword sizeof anims block''
Line 58: Line 68:
 
<legend class="tech">1st PRIMITIVE CHANNEL
 
<legend class="tech">1st PRIMITIVE CHANNEL
  
==Primitive channels==
+
===Primitive channels===
 
</legend>
 
</legend>
 
{| class="wikitable"
 
{| class="wikitable"
 +
!colspan=7|7 32bit words in total
 
|-
 
|-
||- -VB- -||- -VC- -||- -NB- -||- -NC- -||- -PB- -||- -PC- -||- -U0- -
+
||- -VB- -||- -VC- -||- -NB- -||- -NC- -||- -PB- -||- -PC- -||- -00- -
 
|}
 
|}
 
<ifieldset class="spec">
 
<ifieldset class="spec">
Line 82: Line 93:
 
|PC:||primcount||
 
|PC:||primcount||
 
|-
 
|-
|U0:||unknown00||''probably always zero''
+
|00:||unknown00||''probably always zero''
 
|}
 
|}
 
</ifieldset>
 
</ifieldset>
  
 
  VB ¦ NB ¦ PB
 
  VB ¦ NB ¦ PB
These three 32bit (the alignment is 32bit, The values themselves may be 16bit) values are offsets which should be added to the end of the header (or the beginning of the first primitive channel ''sub header'') in order to reach the beginning of the Vertex/Normal/Primitive (face) data belonging to this primitive channel. Part of the [[#Per vertex location]], [[#Per vertex lighting]], and [[#3D primitives]] blocks respectively. The units are in 32bit intervals. In other words, a value of 4, indicates 16 bytes. The offsets are absolute, which is to say, not relative to one another.
+
These three 32bit{{dagger}} values are offsets which should be added to the end of the header (or the beginning of the first primitive channel ''sub header'') in order to reach the beginning of the Vertex/Normal/Primitive (face) data belonging to this primitive channel. Part of the [[#Per vertex location]], [[#Per vertex lighting]], and [[#3D primitives]] blocks respectively. The units are in 32bit intervals. In other words, a value of 4, indicates 16 bytes. The offsets are absolute, which is to say, not relative to one another.
  
 
  VC ¦ NC ¦ PC
 
  VC ¦ NC ¦ PC
These three 32bit (the alignment is 32bit, The values themselves may be 16bit) values account for the number of elements expected to be found within the blocks indicated by VB, NB, PB respectively.
+
These three 32bit{{dagger}} values account for the number of elements expected to be found within the data indicated by VB, NB, PB respectively.
 +
 
 +
{{dagger|The alignment is 32bit, the values themselves may be 16bit (this possibility remains untested.)}}
  
 
</fieldset>
 
</fieldset>
Line 101: Line 114:
 
<fieldset class="spec">
 
<fieldset class="spec">
 
<legend class="tech">3D PRIMITIVES BLOCK
 
<legend class="tech">3D PRIMITIVES BLOCK
==3D primitives==
+
 
 +
===3D primitives===
 
</legend>
 
</legend>
 
{{main|3D Primitive}}
 
{{main|3D Primitive}}
 +
{{Under construction}}
 
</fieldset>
 
</fieldset>
 
<fieldset class="spec">
 
<fieldset class="spec">
 
<legend class="tech">PER VERTEX LOCATION BLOCK
 
<legend class="tech">PER VERTEX LOCATION BLOCK
==Per vertex location==
+
===Per vertex location===
 
</legend>
 
</legend>
 
{{main|3D Vertex#Location}}
 
{{main|3D Vertex#Location}}
 +
This block is packed with consecutive 3D vertex coordinates side by side and nothing else. See [[#Primitive channels]] and [[#3D primitives]] for more information.
 +
{| class="wikitable"
 +
!colspan=11|two 64bit vertices
 +
|-
 +
||-VX-||-VY-||-VZ-||-00-||⋯||-VX-||-VY-||-VZ-||-00-
 +
|}
 +
VX ¦ VY ¦ VZ
 +
These are signed 16bit whole number values. The units are in millimeters, indicating a three dimensional point in space.
 +
00
 +
VX/Y/Z cover all the bases for a 3D vertex. These bits are not used (vertices must begin on 32bit word boundaries)
 
</fieldset>
 
</fieldset>
 
<fieldset class="spec">
 
<fieldset class="spec">
 
<legend class="tech">PER VERTEX LIGHTING BLOCK
 
<legend class="tech">PER VERTEX LIGHTING BLOCK
==Per vertex lighting==
+
===Per vertex lighting===
 
</legend>
 
</legend>
 
{{main|3D Vertex#Lighting}}
 
{{main|3D Vertex#Lighting}}
 +
This block is packed with consecutive 3D lighting vectors (surface normals) side by side and nothing else. See [[#Primitive channels]] and [[#3D primitives]] for more information.
 +
{| class="wikitable"
 +
!colspan=11|two 64bit normals
 +
|-
 +
||-NX-||-NY-||-NZ-||-00-||⋯||-NX-||-NY-||-NZ-||-00-
 +
|}
 +
NX ¦ NY ¦ NZ
 +
These are signed 16bit whole number values indicating a three dimensional surface normal. The convention is 4096:1. Not enough attention has been payed to if/when the values are normalized prior to use.
 +
 +
00
 +
NX/Y/Z cover all the bases for a 3D normal. These bits are not used (normals must begin on 32bit word boundaries)
 
</fieldset>
 
</fieldset>
 
<fieldset class="spec">
 
<fieldset class="spec">
 
<legend class="tech">Tinman ANIMATIONS BLOCK
 
<legend class="tech">Tinman ANIMATIONS BLOCK
==Tinman animations==
+
===Tinman animations===
 
</legend>
 
</legend>
 
{{main|Tinman animation}}
 
{{main|Tinman animation}}
 +
 +
{| class="wikitable"
 +
!colspan=2|32bits
 +
|-
 +
||-NC-||-00-
 +
|}
 +
 +
NC
 +
This 16bit value accounts for the total number of animation channels. A channel may have primitives associated with it or not. Channels are arranged hierarchically / outwardly from a single top-level channel. When one channel is animated, the effect is applied accumulatively to the subordinate channels.
 +
 +
00
 +
This 16bit value is not understood. It may always be 1.
 +
 +
<ifieldset class="spec">
 +
<legend class="tech">1st Tinman ANIMATION
 +
====Animations by ID====
 +
</legend>
 +
{| class="wikitable"
 +
!colspan=2|32bits
 +
|-
 +
||-ID-||-NF-
 +
|}
 +
 +
ID
 +
This 16bit value identifies the context of the animation. For instance, 0 for an "idle" animation. 4 for an "open" animation (eg. a door.) The meanings of the IDs appear to be implicit rather than conventional. Ie. they are hard wired into Sword of Moonlight itself. For a complete list of known IDs, see [[MDL (file format)/Appendix A (animation IDs)]].
 +
 +
NF
 +
This 16bit value accounts for the total number of sequential frames involved in the animation. The frames are not interpolated over time, and appear at regular intervals. There is no standard reference in terms of timing. However traditional (eg. original Sword of Moonlight files) playback speed is approximately 30 frames per second (reports exist that [[PRF]] files are useful for modulating playback in some way unclear / not well documented)
 +
 +
<ifieldset class="spec">
 +
<legend class="caps">
 +
====The animation frame buffer====
 +
</legend>
 +
{{main|/Algorithm A (Tinman codec)}}
 +
 +
The Tinman frame buffer model lacks a definite structure. It is instead a kind of specialized compression algorithm. For a precise definition, there is [[MDL (file format)/Algorithm A (Tinman codec)]].
 +
 +
How it works in a nutshell; there is a sequence of variable length frames, which are themselves comprised of a sequence of variable length channel operations. Unaffected channels are omitted per frame. And in general the affect upon the channel for that frame is proportional to the amount of bytes consumed by that operation. Each operation upon a channel is accumulated frame by frame until the animation runs its course.
 +
 +
The word length of the frame buffer in total is not part of the MDL format. It is necessary to traverse the coarsest level of granularity afforded the frame buffer model in order to compute it. Or in fact to arrive at the end of the the animation itself. Which is where you will find the next animation, and so on, until the last animation indicated in the [[#Header]] is encountered.
 +
 +
The first channel operation sequence of the first frame of the first animation includes some additional information which describes a skeletal hierarchy by ID. IDs up to the number of [[#Primitive channels]] indicated by the header refer to the primitive channels themselves per each ''sub header''. Subsequent IDs refer to until hence undefined animation-only channels, ie. lacking geometry. 
 +
 +
Each channel operation has the net effect of transforming (eg. translating and rotating) its assigned channel little by little something like stop-motion animation.
 +
</ifieldset>
 +
 +
</ifieldset>
 +
<ifieldset class="spec">
 +
<legend>⋯</legend>
 +
</ifieldset>
 +
<ifieldset class="spec">
 +
<legend>Nth Tinman ANIMATION</legend>
 +
</ifieldset>
 
</fieldset>
 
</fieldset>
 
<fieldset class="spec">
 
<fieldset class="spec">
 
<legend class="tech">Scarecrow ANIMATIONS BLOCK
 
<legend class="tech">Scarecrow ANIMATIONS BLOCK
==Scarecrow animations==
+
 
 +
===Scarecrow animations===
 
</legend>
 
</legend>
 
{{main|Scarecrow animation}}
 
{{main|Scarecrow animation}}
 +
{{Under construction}}
 
</fieldset>
 
</fieldset>
 
<fieldset class="spec">
 
<fieldset class="spec">
 
<legend class="tech">1st PlayStation .TIM IMAGE
 
<legend class="tech">1st PlayStation .TIM IMAGE
==TIM imaging==  
+
===TIM imaging===
 
</legend>
 
</legend>
 +
{{main|TIM (file format)}}
 +
This block is an embedded TIM image which works as a [[3D Texture]]. The TIM specifications are not fully implemented by Sword of Moonlight. 24bit color mode is not supported. Dimensions are restricted to 256x256 pixels or less. It's generally not known to what extent the PlayStation frame buffer is implemented. Up to four 256x256 textures are known to work (some speculations are conceivable.)
 
</fieldset>
 
</fieldset>
 
<fieldset class="spec">
 
<fieldset class="spec">
Line 140: Line 233:
 
<legend>Nth PlayStation .TIM IMAGE</legend>
 
<legend>Nth PlayStation .TIM IMAGE</legend>
 
</fieldset>
 
</fieldset>
 +
 +
==See also==
 +
*[[/Appendix A (animation IDs)]]
 +
*[[x2mdl]]
  
 
[[Category:Sword of Moonlight]]
 
[[Category:Sword of Moonlight]]
 
[[Category:File formats]]
 
[[Category:File formats]]

Latest revision as of 05:19, 21 August 2013

MDL is the file format employed by Sword of Moonlight when something is to be animated. It's counterpart is MDO. Tools were provided by From Software for working with MDO files. However, no analogous tools were ever provided for MDL. MDO appears to be incapable of locomotive animation, despite having some simple texture based animation ability, which MDL may well not enjoy. MDL does the rest.

History[edit]

Despite having tools to work with MDO files, the format itself remains a black box. Since the MDL format on the other hand had been completely inaccessible for nearly a decade, greater priority has been given to figuring it out. The file format was first reverse engineered sometime in the latter half of 2010. There has been no official statement, however there has been a claim that on at least one occasion, a From Software customer relations spokesperson did state via email, paraphrasing, any attempt to reverse engineer Sword of Moonlight file formats is OK with the company.

Even though importers and exporters (export from MDL would come later) were developed around the same time the format had become well understood, there was no party invested enough in creating original MDL compatible models until the mid months of 2011. Efforts so far remain in the immature stages while a fully realized outcome appears inevitable.

Characteristics[edit]

An MDL file is a relatively simple, piecewise model, with limited texture mapping possibilities. There are two flavor of animation, Tinman and Scarecrow. Tinman animations move like an animated suit of armor, or a wooden puppet come to life. Scarecrow on the other hand, move like a sack doll, or anything really with some degree of flexibility.

Specification[edit]

This section describes the file format in an intimate way, which the average reader might prefer to dismiss. It should be understood that the specification below, is more like documentation, and a reverse engineered one at that. Note that some more technical aspects may be offloaded to subpages linked to from within. If you want to make MDL files, it will probably be a favor to yourself to look this over well.

<legend class="caps">

Header[edit]

</legend>

16 8bit bytes in total
LF HA SA TB PC 00 -16- PCSZ -16- HASZ SASZ

<ifieldset class="spec"> <legend> swordofmoonlight.h </legend>

LF: leadflags perhaps bitwise flags
HA: hardanims JointMIMe-like animation (PSOne)
SA: softanims Vertex/normalMIMe-like animation
TB: timblocks Textures (PSOne TIM image format)
PC: primchans one or more required (meshes)
00: unknown00 perhaps always one
PCSZ: primchanwords dword anims block offset (could be 32bit)
HASZ: hardanimwords dword sizeof anims block
SASZ: softanimwords dword sizeof diffs block

</ifieldset>

LF

This byte is not super well understood. It is either a straight value, or a bitwise combination of flag values. Known values are 0, 1, and 4. For not animated, Tinman style animation, and Scarecrow style animation respectively.

HA ¦  SA

These bytes are the number of hard and soft (aka. Tinman/Scarecrow) animations contained within the file.

TB

This byte is the number of PlayStation TIM images embedded within the file. These images are used as textures. It seems that the EneEdit tool somehow overrides the use of the embedded images, in favor of external TXR images which it is able to generate (please note that the author himself is uncertain the accuracy of this claim!!)

PC

This byte is the number of sub headers in the file, of which there is one of per each separate section of geometry (a MDL may include many such sections.)

PCSZ ¦ HASZ ¦ SASZ

These three 16bit values indicate the size of the first three data blocks. The units are in 32bit intervals. In other words, a value of 4, indicates 16 bytes. Each block begins after the previous block, therefore it is necessary to sum the preceding blocks in order to arrive at the offset to any given block. There is a fourth block, which runs to the end of the file. The four blocks respectively pertain to 3D geometry, Tinman animation, Scarecrow animation, and TIM images. The TIM images themselves can consecutively be thought of as 4th, 5th, and so on blocks. It's a matter of personal interpretation.

<legend class="tech">1st PRIMITIVE CHANNEL

Primitive channels[edit]

</legend>

7 32bit words in total
- -VB- - - -VC- - - -NB- - - -NC- - - -PB- - - -PC- - - -00- -

<ifieldset class="spec"> <legend> swordofmoonlight.h </legend>

VB: vertsbase dword offset to vertex block
VC: vertcount
NB: normsbase dword offset to normal block
NC: normcount
PB: primsbase dword offset to face block
PC: primcount
00: unknown00 probably always zero

</ifieldset>

VB ¦ NB ¦ PB

These three 32bit+ values are offsets which should be added to the end of the header (or the beginning of the first primitive channel sub header) in order to reach the beginning of the Vertex/Normal/Primitive (face) data belonging to this primitive channel. Part of the #Per vertex location, #Per vertex lighting, and #3D primitives blocks respectively. The units are in 32bit intervals. In other words, a value of 4, indicates 16 bytes. The offsets are absolute, which is to say, not relative to one another.

VC ¦ NC ¦ PC

These three 32bit+ values account for the number of elements expected to be found within the data indicated by VB, NB, PB respectively.

+The alignment is 32bit, the values themselves may be 16bit (this possibility remains untested.)

<legend>⋯</legend>

<legend>Nth PRIMITIVE CHANNEL</legend>

<legend class="tech">3D PRIMITIVES BLOCK

3D primitives[edit]

</legend>

<legend class="tech">PER VERTEX LOCATION BLOCK

Per vertex location[edit]

</legend>

This block is packed with consecutive 3D vertex coordinates side by side and nothing else. See #Primitive channels and #3D primitives for more information.

two 64bit vertices
-VX- -VY- -VZ- -00- -VX- -VY- -VZ- -00-
VX ¦ VY ¦ VZ

These are signed 16bit whole number values. The units are in millimeters, indicating a three dimensional point in space.

00

VX/Y/Z cover all the bases for a 3D vertex. These bits are not used (vertices must begin on 32bit word boundaries)

<legend class="tech">PER VERTEX LIGHTING BLOCK

Per vertex lighting[edit]

</legend>

This block is packed with consecutive 3D lighting vectors (surface normals) side by side and nothing else. See #Primitive channels and #3D primitives for more information.

two 64bit normals
-NX- -NY- -NZ- -00- -NX- -NY- -NZ- -00-
NX ¦ NY ¦ NZ

These are signed 16bit whole number values indicating a three dimensional surface normal. The convention is 4096:1. Not enough attention has been payed to if/when the values are normalized prior to use.

00

NX/Y/Z cover all the bases for a 3D normal. These bits are not used (normals must begin on 32bit word boundaries)

<legend class="tech">Tinman ANIMATIONS BLOCK

Tinman animations[edit]

</legend>

32bits
-NC- -00-
NC

This 16bit value accounts for the total number of animation channels. A channel may have primitives associated with it or not. Channels are arranged hierarchically / outwardly from a single top-level channel. When one channel is animated, the effect is applied accumulatively to the subordinate channels.

00

This 16bit value is not understood. It may always be 1.

<ifieldset class="spec"> <legend class="tech">1st Tinman ANIMATION

Animations by ID[edit]

</legend>

32bits
-ID- -NF-
ID

This 16bit value identifies the context of the animation. For instance, 0 for an "idle" animation. 4 for an "open" animation (eg. a door.) The meanings of the IDs appear to be implicit rather than conventional. Ie. they are hard wired into Sword of Moonlight itself. For a complete list of known IDs, see MDL (file format) / Appendix A (animation IDs).

NF

This 16bit value accounts for the total number of sequential frames involved in the animation. The frames are not interpolated over time, and appear at regular intervals. There is no standard reference in terms of timing. However traditional (eg. original Sword of Moonlight files) playback speed is approximately 30 frames per second (reports exist that PRF files are useful for modulating playback in some way unclear / not well documented)

<ifieldset class="spec"> <legend class="caps">

The animation frame buffer[edit]

</legend>

The Tinman frame buffer model lacks a definite structure. It is instead a kind of specialized compression algorithm. For a precise definition, there is MDL (file format) / Algorithm A (Tinman codec).

How it works in a nutshell; there is a sequence of variable length frames, which are themselves comprised of a sequence of variable length channel operations. Unaffected channels are omitted per frame. And in general the affect upon the channel for that frame is proportional to the amount of bytes consumed by that operation. Each operation upon a channel is accumulated frame by frame until the animation runs its course.

The word length of the frame buffer in total is not part of the MDL format. It is necessary to traverse the coarsest level of granularity afforded the frame buffer model in order to compute it. Or in fact to arrive at the end of the the animation itself. Which is where you will find the next animation, and so on, until the last animation indicated in the #Header is encountered.

The first channel operation sequence of the first frame of the first animation includes some additional information which describes a skeletal hierarchy by ID. IDs up to the number of #Primitive channels indicated by the header refer to the primitive channels themselves per each sub header. Subsequent IDs refer to until hence undefined animation-only channels, ie. lacking geometry.

Each channel operation has the net effect of transforming (eg. translating and rotating) its assigned channel little by little something like stop-motion animation. </ifieldset>

</ifieldset> <ifieldset class="spec"> <legend>⋯</legend> </ifieldset> <ifieldset class="spec"> <legend>Nth Tinman ANIMATION</legend> </ifieldset>

<legend class="tech">Scarecrow ANIMATIONS BLOCK

Scarecrow animations[edit]

</legend>

<legend class="tech">1st PlayStation .TIM IMAGE

TIM imaging[edit]

</legend>

This block is an embedded TIM image which works as a 3D Texture. The TIM specifications are not fully implemented by Sword of Moonlight. 24bit color mode is not supported. Dimensions are restricted to 256x256 pixels or less. It's generally not known to what extent the PlayStation frame buffer is implemented. Up to four 256x256 textures are known to work (some speculations are conceivable.)

<legend>⋯</legend>

<legend>Nth PlayStation .TIM IMAGE</legend>

See also[edit]