OpenGL Programming/Stencil buffer

When you draw something with OpenGL, you see colors on the screen, but keep in mind that there are other buffers than the color buffer. You are already familiar with the depth buffer, that prevent background pixels from being displayed if there is a closer pixel already. Now it's time to introduce the stencil buffer.

= Concept =

The stencil buffer is another buffer for your custom use: you can store per-pixel information there and direct OpenGL to act differently depending on that information.

There are quite a few points to understand to manipulate the stencil buffer correctly, so let's take time to learn them.

= Bitplanes =

This is mostly a matter of vocabulary: the per-pixel information is a number of bitplanes - that is, of bits.

For instance, it is common to have 8 bitplanes, which means one byte is associated with each pixel on screen, and you can store up to $$2^8=256$$ different values. You can inspect the number of bitplanes using.

= Tests and Operations =

The stencil buffer can be manipulated using  and  :   specifies a test to apply to each pixel of the stencil buffer, then   specifies the action to apply depending on the test result.

takes 3 parameters, to build the following test :
 * OP: one of GL_NEVER, GL_ALWAYS, GL_EQUAL, GL_NOTEQUAL, GL_LESS, GL_LEQUAL, GL_GEQUAL, GL_GREATER
 * ref: a fixed integer used in the comparison
 * mask: a mask applied to both ref and the stencil pixel; you can use 0xFF (if you have 8 bitplanes) to disable the mask

takes 3 parameters: Each of these 3 parameters is an action to perform on the stencil buffer, one of GL_KEEP, GL_ZERO, GL_REPLACE, GL_INCR, GL_INCR_WRAP, GL_DECR, GL_DECR_WRAP, GL_INVERT (default GL_KEEP).
 * sfail: the test from  failed
 * dpfail: the test from  passed, but the depth buffer test failed
 * dppass: the test from  passed, and the depth buffer passed or is disabled

So, we see that two different tests are chained, resulting in 3 situations: Note that with sfail and dpfail, the color buffer is left unmodified.
 * first, if it fails we apply sfail and stop
 * then depth-test (if depth buffer is available and enabled), if it fails we execute dpfail and stop
 * if both tests work, we evaluate the fragment shader in the color buffer and apply dppass

If you draw a volumetric shape, keep in mind that OpenGL may draw the same stencil pixel several times, depending on the order of overlapping triangles and/or the availability of the depth buffer; in such cases, your action will be applied several times.

= Sample =

In our example, we'll draw a moving circle, and the scene will be clipped inside that circle:

draws the color cube from tutorial 05, and  draws a circle in 2d mode. We won't discuss these functions because they are not related to the stencil feature, but you're invited to look at the source code (see at the link at the bottom of this page).

Note that there's no need to use a separate program to draw in the stencil buffer. In this case it makes sense, because we draw a 2D pattern, but it's perfectly possible to draw a 3D pattern in the same scene, as in the Mini-Portal tutorial.

= Working on color, depth and/or stencil buffers specifically =

In the sample, we see that we first worked exclusively on the stencil buffer to draw the stencil shape, then exclusively on the color buffer to use it as a mask.

When you want to modify the stencil buffer without modifying the color buffer and/or depth buffer, you can mask these buffers: Don't forget to set them back. You can save the previous values this way:

If you want to apply changes on the color buffer without modifying the stencil buffer, you can either:
 * set  - this means nothing will be written in any condition
 * set  - which applies a no-op for all conditions

= Debugging =

It is difficult to understand what is wrong when working on the stencil buffer, because we have no direct way to see it.

If you use OpenGL non-ES, you can call  to grab the whole stencil buffer and inspect it externally, but it's tedious.

Another method is to fill the screen using two triangles, and with a stencil test (for instance, to match only non-0 stencil pixels). You may need to enable/disable the color buffer when doing so.

Then swap the OpenGL color buffer and pause, so you can inspect the result visually:

Your  function will preferably use a dedicated program that will involve no MVP matrix, though you can reuse an existing program and pass   and.

= Performances =

We saw there are three possible actions in :
 * sfail
 * dpfail
 * dppass

It is true that it's counter-intuitive to express your action on the stencil buffer using the sfail action, because of the double-negation (instead of "do this when this condition is met", you have to write "don't do this when this condition is not met).

However, if you use dppass to express your action, keep in mind that OpenGL will check both the stencil and depth test, and will also compute the pixel value using the fragment shader. This is a terrible performance hit if you are only drawing a shape in the stencil buffer - because you don't need to call the fragment shader at all.

A suggestion is to first implement your drawing algorithm using dppass for clarity, and once it works, reverse the conditions in  and use the sfail. sfail will not try to evaluate the pixel fragment when the test fails, so it's faster.

Example:
 * First prototype with:
 * then optimize:

= Limitations =

The OpenGL 4.2 core profile specification mentions:
 * However, when both depth and stencil attachments are present, implementations are only required to support framebuffer objects where both attachments refer to the same image.

This means that you probably cannot attach two renderbuffers to a single frame buffer.

In OpenGL ES 2.0 in particular, combined formats are not supported in   (only   and   are supported, typically not   or  ).

If you need to combine post-processing and stenciling, you'll need to avoid framebuffer objects and use the technique instead.

= References =