So, my own rendering project at home has been stalled of late : between Mass Effect 3, F1 races, going back to my home town and being ill I've not had the time to work on it too much.

The weekend before last I wrote some code to setup a simple Win32 window so that I could crack on but beyond that things haven't progressed too far.

While I do intend to carry on with the project over time (because there are some things I want to try out) I've also shifted my attention to properly learning about the Dynamic Language Runtime (DLR) in .Net4.

My motivation behind this is down to a growing frustration at work with our material data system and the amount of work it takes to work with them.

Broadly speaking out materials are broken up into 3 parts;

1. templates
   
2. types
   
3. materials
   

Templates are the lowest level; they define all the input data and configuration options that can be applied to a shader.
Types are a set of configurations for a shader and are used to control shader compiling.
Finally the 'materials' themselves are an instance of a type/template which holds the parameter settings and what textures should be applied to them.

As a system it works very well however there are still some issues with it mostly to do with redeclarations of information.

You see there is a 4th, slightly hidden, component to the above; a python script.
This python script is used to help process the information in a material into a 'game ready' state.

The build system uses it on a per-material basis to work out what operations should be done to textures to make them 'game ready' as well as how parameters should be packed together so that the game team can control what entries go where in each float4 block.

This flexibilityis nice however it brought with it some problems, mostly with textures but also with parameters.

With textures there is no longer a 1:1 mapping between what the artist attaches and what the shader reads. The pipeline could, for example, take two textures and combine them to produce a 3rd which is used in the game. It could split a texture into two. The most extreme example we could think of was that a texture could be processed, it's average lum. value worked out and this feedback into the material as a parameter.

The first problem however was the lack of 1:1 mapping and this was 'solved' in the script by having it remap the inputs to the correct outputs when it was a pass thru operation. This was done, everyone was happy and on we went.

The parameters came next and they had the same problem; as we could pack parameters together now there 1:1 mapping was once again removed so once again the python script performed the remap and pack operation and all was good.

Then another problem appeared; live link.
Via a connection to the game it is possible to change material parameters on the fly however with the broken 1:1 mapping we now needed a way to tell the editor how to pack data together to send it over the wire and, more importantly, which artist inputs trigger updates on what parameters in the shader.

And thus was born a 'live link' map which could be used to figure out what inputs affected what outputs, would let you run a python script (via IronPython) to condition the data if required and all was good once more.

Until the max shaders needed to be dealt with, at which point the lack of 1:1 mapping for the textures once again appeared. As the max shader could be heavier than the final in-game shaders there was no requirement to build the data for visualisation however it still needed to know what inputs mapped to what shader inputs and thus the 'max map' was born.

So now a template is defining

• config options
  
• artist inputs
  
• material outputs
  
• 'live link' mappings for parameters
  
• max mappings for parameter and textures
  

But at the same time the python was also defining the mapping between parameters and textures for input & outputs so already the information is duplicated and easy to get out of step.

Finally the system isnt as easy to use as I had hoped. Yes, I did the python work and in my head I saw the idea of having centralised 'packing' functions for the scripts however this hasn't happened; instead the game team have been duplicating functions and copying and pasting things around.

Between the duplication, the 'logic' in the XML, the scripting issues and the general slowing of the build pipeline I've finally snapped.

Which brings us to the DLR.

What I would like to produce, using the DLR, is a domain specific language which can replace all of the above.
Our build system, tools and max plugin are all based on .Net4.0 now so the DLR is a key part of that so being able to load and run a 'script' which defines a template seems like a good fit.

My hope is that, given some time, a language can be designed which allows you to declare the various inputs, outputs & transformations and that, by using a plugable backend, the same script can be used by the tools and build system to a different end.

For example if you were to write:

declare parameter "foo" float

then the tools would know to create an input in the UI for a single float with the label/name foo while the build system would know to look for an input named 'foo' in the material file it was passing.

Now, that syntax is by no means final but a more declarative syntax would be a better fit I feel and with some decent library support for data packing/transform built in (and expandable via .Net dlls) could make a viable replacement.

Now, I only had the idea around 48 hours ago and I've only recently started working on learning the DLR however as projects go I think this would be worthwhile.