Developing an Image Post-Processor plug-in for OpenFX

Introduction

The plug-in Post-Processor  is defined in the Animator as either a universally applied process of every rendered frame or as an Actor that applies the effect over a range of frames in an animation. The Actor defined image processor can have its parameters morph over a range of  frames. It is possible for multiple image processors to be applied one after the other in succession, each processor applies its effect to the image modified by any predecessor.  The image processors can be very complex as they can gain access to the Software Renderer's depth buffer and thus an image processor could render a whole scene and blend it in with the OpenFX scene rendered in the usual way. This is how the atmospheric effect of volume fog is achieved. Access to the dept buffer ca also allow glowing effects, and lens flares to appear to go behind objects in the scene.

As well as providing access to the output buffers (colour, depth etc.) the interface with the renderer provides access to the camera parameters, lights and all mesh objects in the scene.  Access to the Renderer's data, and to some functions in the renderer is provided through a structure called an XIMAGE structure. A pointer to the this structure is passed to the function performing the effect, called:

long _RenderImageProcess(char *PrmList, XIMAGE *lpXimage);

The main structures that the post processors use is the output image buffer and the lights in the scene.  The XIMAGE structure is defined in file "ximage.h" (located in the "common\postprocess"  folder and the framebuffer and light's structure is defined in file "struct.h" which is located in the "postprocess" folder.

Storing and using effect settings

Plug-in  postprocessors  (and shaders/animation effects) all have user configurable parameters. The number and type of the configurable parameters necessarily support different numbers of and different types of data.  To be able to store these in the .MFX and .OFX files OpenFX stores these parameters as a text string - each plug-in is expected to begin by reading its parameters from the parameter text string (if it exists, or create it from defaults, if it does not exist.) In that part of the plug-in used by the Designer or Animator, before it terminates it is expected to write into the text string its parameters. This can normally be done using the C functions sscanf() and sprintf().  By adopting this strategy OpenFX imposes no constraints on the type or number of shader (etc.) parameters.

Building and Files

Each image postprocessor is built as a Windows DLL, it will have one or more C or C++ language source file, one or more project specific header files , a Visual Studio project file, a resource file (.RC)  and a file with filename extension .XFX.  The project will build to a .DLL of the same name as the XFX file.  The build files can have any location, but the XFX and DLL files must be placed in the "POSTPROCESS" folder.

The plugin source files need to include the following files;

• struct.h

• ..\common\postprocess\ximage.h

• local.h  (which includes the memory management functions)

• utils.h (included utility functions from file  xtils.c)

It is useful to include some helpful ancillary routines by including the files:

• paint.c

The file "xutils.c" should be added to the project source file list and compiled along with the image processor's files. Functions in "utils.c" can gain access to all the Animator's internal data structures through the defines in "dstruct.h"  (See the "lens flare" and "volume rendering atmospherics" plugins for examples of how to assess these animator data structures to allow the post process to select specific Actors.)

The "struct.h" header is located in the "effects" folder and it defines the "fullscreenbuffer" and the "light" data structures.  The file "ximage.h"  defines a data structure, (XIMAGE)   that gives the postprocessor access to the Renderer's internal data that could be useful in implementing a postprocess. (All lights and the screen buffers are accessed through the use of this header.)   Before calling a postprocessor the Renderer fills an instance of the XIMAGE structure and passes it to the _RenderImageProcess() function which is the function in the post processor that implements the filter.

Two recursively included headers define the memory allocation functions used by the current OpenFX, these are located in the "animate" folder. The "common\mem.h" file provides system specific memory allocation function defines. File "animate\memory.h" provides a structure that has a void pointer member which the Animator will set to point to its ANI_STRUCTURE before it calls the effect set-up functions. (See  XUTILS.C for an example of how to access and use an Actor's names.)

The resource file (.RC) will contain a dialog box description to allow the processor's parameters to be set by the user.  To get a feel for an image processor we shall look at a plugin for a simple  filter that adjusts the dynamic range in the output image to make it suitable for display on TVs  rather than a computer's monitor.

The effect dialog presents a number of options:

The files associated with the wave effect are:

• tv.c
• tv.h
• tv.rc
• tv.xfx
• def.def

these are located in the "postprocess" folder. The build project  file is also located in the "postprocess" folder and there is a SLN file that includes all the image processor projects.

The build process will create a DLL file called tv.dll in the postprocess folder.  The wave.efx is a dummy file containing nothing, it is used for the external  texture selection box to provide a list of all effects, it can be regarded as a simple place holder.   If you want to start writing a new plugin then we suggest you copy the files from the example plugin to a new set.  It's possible to copy the .VCPROJ files and modify it using notepad  replacing all instances of the string  "TV" with the name of your new texture.

The DEF.DEF  file is common to all postprocessors and it identifies all the functions in the postprocessor that are exported for use in the Animator and the Renderer.

This concludes all we have to say about building. Most of the other plugins in OpenFX follow a similar strategy.

The plugin-code

The plugin code follows the basic design of a Windows DLL . In this section we will examine the parts of the source file that are common to all postprocessor plugins. The most important items to be aware of are the names of the functions called by OpenFX's modules.  These functions use pointers to data structures through which access to the OpenFX main module's (designer/animator/renderer) memory management, and other functions may be made.    

The OpenFX method of accessing global variables and main program functions   (VERY IMPORTANT)

OpenFX uses a pointer mechanism for allowing the plugins to access functions in the main program and program global variables. Since all OpenFX plugins are DLLs that are loaded into the main applications address space. The plugin DLLs are NOT loaded when the main application starts (many of the built-in actions are also implemented as DLLs that ARE loaded when the program starts) they are loaded when required and unloaded when they have finished.
In the case of plugins that are used in the Designer and Animator AND are needed by the Renderer (e.g. shaders) then the Renderer loads all the needed plugins when it reads the script file during initialisation. The Renderer unloads the plugins when it is finished with the script.

When the Animator requires to configure a postprocess  it calls  the function _SetExternalParameters(), with  arguments:

char *_SetExternalParameters(char *Op, ,HWND hWnd, long ruler, char *name, X__MEMORY_MANAGER *lpevi);

• The first parameter is a character pointer to the parameter string for the texture (this was mentioned above and we will render to it below.
• The second argument is a handle to the container's window. This handle is used in the generation of the modeless dialog box that allow the user to enter the image processor's parameters.
• The third parameter is the size (in integer units) of the equivalent of 1.0 distance units (whatever we decide those unit to be, cms, miles, etc..)
• The fourth parameter is a character string giving the name of the DLL supplying the effect.
• The third parameter, the  "lpEVI" parameter is a pointer to the X__MEMORY_MANAGER data structure. This structure is a global structure that contains pointers to OpenFX's equivalent of the "malloc()" and "free()" funcitons. Actually we use the Windows API for memory allocation but this may change in future. This structure looks like this:

typedef struct tagX__MEMORY_MANAGER {
void *(*fpMalloc)(long size);
void (*fpFree)(void *buffer);
void *lpAni;
void *lpMod;
} X__MEMORY_MANAGER;

The void structure members "lpAni" and "lpMod"   are used to pass pointers to the internal structures ANI_STRUCTURE(Animator)   and X__STRUCTURE(Designer) when appropriate!  (When not used these pointers are set to NULL.)

The Designer, Animator and Renderer  keep only one copy of this structure, it is initialised at start-up and passed to all the plugins.  By using the members of this structure any plugin can access the memory functions used internally.   This method is very simple and very flexible for extension and allowing plug-ins to access global data and functions in the calling programs.

In order that the contents of structures like this can be made available to all functions and files in a plugin the pointers passed in to the functions should normally  be copied into the global variable:

Accessing variables and functions in a DLL module through direct use of a pointer to the X__MEMORY_MANAGER  is very tiresome.  For example to use the Designer's number of selected  vertices variable (called "NvertSelect")  it would be necessary to use the code (  (*(lpevi->NvertSelect))    )  to  gain access to this variable.   This is very messy!  To overcome this issue and provide an interface to the functions that make them look (in the code) just like normal function calls, a number of #defines are included in the header file "defines.h" , which is one of the ".h" files that every plugin must include.

Using these defines, calls to functions and use of global variables can be used as if they were part of the module code.

Keeping this very important information in mind we can return to considering the Checker texture plugin and look at the key code constructs.

An image post-processors primarily requires access to the framebuffers, this is achieved through the "lpXimage" pointer parameter passed to the "_RenderIamgeProcess(..)" function. The screen pixels are addressed at "lpXimage->Screen"  Each pixel is recorded as an instance of the "fullscreenbuffer" structure defined in file "struct.h". Thus for example to work through the entire output buffer setting it to a black colour could be done with code like this:

long _RenderImageProcess(char *PrmList, XIMAGE *lpXimage){
 int i,j; 
 fullscreenbuffer *S;
 S=lpXimage->Screen;
 for(i=0;i<lpXimage->Ymax;i++){
   for(j=0;j<lpXimage->Xmax;j++){
     S->R=S->G=S->B=0;
     S++;
   }
 }
 return 1;
}

The required elements in a plugin postprocess are:

#include <stdlib.h>
#include <windows.h>
#include "struct.h" /* general structures */
#include "..\common\postprocess\ximage.h"
#include "local.h" 
#include "utils.h"
static HINSTANCE hDLLinstance=NULL; /* use to pick up resources from DLL */ 

BOOL WINAPI DllMain(HANDLE hDLL, DWORD dwReason, LPVOID lpReserved){
switch (dwReason) {
case DLL_PROCESS_ATTACH:
  hDLLinstance = hDLL; /* handle to DLL file */
  break;
case DLL_PROCESS_DETACH:
   break;
}
return (int)TRUE;
}

char * _SetExternalParameters(
   char *Op, /* string for the parameters */
   HWND hWnd, /* parent window */
   long ruler, /* ruler scale value to facilitate scaling */
   char *name, /* name of DLL file with the effect */
   X__MEMORY_MANAGER *lpEVI /* pointer to structure with memory functions */
){
 /* output name and buffer should be as long as necessary to hold full string */
 char buffer[256];
 if(Op != NULL){ /* parameters exist so read them off the list */
   sscanf(Op,"%s %  ...",buffer, ...);
 }
 /* Do the user interface as required to set up the effect, may use a */
 /* dialog box etc. Return old string if effect is cancelled (no change) */
 if(DialogBox(hDLLinstance,MAKEINTRESOURCE(DLG_....),hWnd, (
    DLGPROC)DlgProc) == FALSE)return Op;
 /* Free space occupied by old parameter string */
 if(Op != NULL)CALL_FREE(Op); /* free the old string */
 /* print the parameters into the buffer */
 sprintf(buffer,"%s % ...", ...);
 /* Prepare the output buffer to take copy of parameter list */
 if((Op=(char *)CALL_MALLOC(strlen(buffer)+1)) == NULL)return NULL;
 /* Copy the parameter string to the output buffer */
 strcpy(Op,buffer);
 return Op;
}

long _RenderImageProcess(char *PrmList, XIMAGE *lpXimage){
 /* In the data passed via the lpXimage Structure                  */
 /* Z is UP, Y is forward (in front of observer, X to the RIGHT    */
 /* the camera is located at 0,0,0. All real numbers are floats    */
 /* xx and yy are screen co-ords of light (0,0) is at top left     */
 /* Zbuffer values are distances from Camera (at 0,0,0) to surface */
 /* visible in pixel. Z buffer addressing is similar to Screen     */
 /* buffer addressing.                                             */

 fullscreenbuffer *S;

 sscanf(PrmList,"%s ", ...);
 S=lpXimage->Screen;
 for(i=0;i<lpXimage->Ymax;i++){
   for(j=0;j<lpXimage->Xmax;j++){
     // process the screen buffer
     S++;
   }
 }
 return 1;
}


long _RenderGLexternal(char *PrmList, XIMAGE *lpXimage){
 //Nothing implemented 
 return 1;
}

The TV process example

In the TV process example the colour values are converted to the HSV colour system and the Value parameter adjusted to lie within given parameters, then the colour values are converted back to RGB and written back into the image buffer. See comments in the code below

 ...
 fullscreenbuffer *S;
 BOOL bMorph;
 // read the processors parameters
 sscanf(PrmList,"%s %d %d %d",name_path,&ftype,&id1,&id2);
 d1=(double)id1;
 d2=(double)id2;
 // If the effect is changing along a timeline then it is
 // interploated over time.
 if(lpXimage->Morph && lpXimage->mParameters != NULL){
   bMorph=TRUE; mr=lpXimage->MorphRatio;
   sscanf(lpXimage->mParameters,"%s %d %d %d",
          name_path,&mftype,&mid1,&mid2);
 }
 d1/=255.0; d2/=255.0;
 range=(d1-d2);
 // get pointer to start of main output image buffer
 S=lpXimage->Screen;
 // go through the image buffer pixel by pixel
 // convert to HSV and clamp and scale and convert
 // back to RGB
 for(i=0;i<lpXimage->Ymax;i++){
   for(j=0;j<lpXimage->Xmax;j++){
     Rgb2Hsv(S->R,S->G,S->B,&h,&s,&v);
     s=max(0.0,(s-d2))*range+d2; 
     v=max(0.0,(v-d2))*range+d2;
     Hsv2Rgb(h,s,v,&(S->R),&(S->G),&(S->B));
     S++;
   }
 }
 return 1;
}

Go back to the developer page...