Cg Programming/Unity/Outlining Objects

This tutorial covers a vertex transformation that moves each vertex along its surface normal vector to create a larger outline of an object. While this vertex transformation is quite simple, creating a reasonable rendering of the outline requires use of the stencil buffer, which is also discussed.

Creating an Outline by Moving Vertices along their Normal Vector
One method to create an outline that is around an object is to enlarge the object by moving its vertices along their surface normal vectors. Given the position and the normal vector in object coordinates, this is quite straightforward in a vertex shader: This vertex shader converts the surface normal vector  into a   vector, then multiplies it with the user-specified uniform   (to allow for an adjustable thickness of the outline) and adds it to the position of the vertex in. All this happens in object coordinates; thus, we have to convert to clip coordinates by multiplying with  before returning the position.

There is not much more to say about this specific vertex transformation – except that it works best with smooth surfaces. Surfaces with hard edges (i.e., discontinuous surface normal vectors) do not work well. Sometimes it helps to use  to render frontfaces and backfaces, but in general the approach just doesn't work well with hard edges.

To give the outline a uniform green color, we can use a simple fragment shader like this:

Avoiding Occlusions of the Outlined Object by its Outline
The more challenging part of rendering this kind of outline is to avoid occlusions of the outlined object by the outline: since we have enlarged the object to create the outline, the larger outline will usually occlude the object that we want to outline. On the other hand, the outline should occlude other objects in the background and it should be occluded by objects in the foreground, i.e., it should behave like any other opaque object, except when it is in front of the outlined object.

If we assume that we first render a regular version of the object that we want to outline, and after that render the outline, we can state the challenge in this way: the outline should only be rasterized for pixels that are not covered by the outlined object. In this way, the challenge sounds like something that can be solved with the stencil test, because the stencil test is used to limit the rasterization to certain parts of the framebuffer; in our case to the parts of the framebuffer that are not covered by the object that we want to outline.

Our strategy is then to first mark all pixels that are covered by the object that we want to outline in the stencil buffer. After that, when rendering the outline, we can use a stencil test to rasterize the outline only in pixels that haven't been marked.

To mark all pixels that are covered by an object with a value of 1 in the stencil buffer, we can use this ShaderLab syntax in a  block before  : The  keyword specifies that we want to activate a stencil test, which is necessary because writing to the stencil buffer technically is part of the stencil test. sets the reference value of the stencil test; in this case the value that we want to write into the stencil buffer.

specifies that we want to work with all fragments regardless of the pixels' value in the stencil buffer. In general,  specifies a comparison for a pixel's value in the stencil buffer. If that comparison fails, the fragment is discarded and the pixel is not rasterized. specifies a comparison that always passes, i.e., none of the fragments are discarded.

specifies that the  "operation" should be applied to the stencil buffer if the comparison has been passed, i.e., the value of a pixel in the stencil buffer should be replaced by the reference value that is specified with.

A complete pass to render a black object and at the same time mark the stencil buffer with 1 in all pixels that are covered by the object could look like this:

The second part of our approach to to render the outline only where the stencil buffer has not been marked with a 1. Such a stencil test could be specified in this way: Again,  sets the reference value of the stencil test, but in this case the value is used for the comparison.

specifies a comparison for a pixel's value in the stencil buffer that is passed only if the value is not equal to the reference value. If that comparison fails (i.e., if the value in the stencil buffer is equal to 1) the fragment is discarded and the pixel is not rasterized, which is what we want for our outline such that it doesn't occlude the object that we want to outline.

specifies that the  "operation" should be applied to the stencil buffer if the comparison has been passed, i.e., we don't change the stencil buffer but simply keep whatever values are already in the stencil buffer. (This is important because multiple triangles of the mesh might cover the same pixel.)

This stencil test is applied to the shader above for the outline that had its vertices moved along the surface normal vector.

Complete Shader Code
Putting everything together and defining a properties block for the  uniform variable, we have this shader: This shader renders opaque green outlines for uniformly black objects. However, you can easily replace the vertex and fragment shaders to render transparent outlines of other colors or shaded objects – as long as you keep the stencil tests and the outline in the 2nd pass is larger than the object in the 1st pass.