GLSL Programming/Unity/Lighting of Bumpy Surfaces

This tutorial covers normal mapping.

It's the first in a series of tutorials about texturing techniques that go beyond two-dimensional surfaces (or layers of surfaces). In this tutorial, we start with normal mapping, which is a very well established technique to fake the lighting of small bumps and dents — even on coarse polygon meshes. The code of this tutorial is based on and.

Perceiving Shapes Based on Lighting
The painting by Caravaggio depicted is about the incredulity of Saint Thomas, who did not believe in Christ's resurrection until he put his finger in Christ's side. The furrowed brows of the apostles not only symbolize this incredulity but clearly convey it by means of a common facial expression. However, why do we know that their foreheads are actually furrowed instead of being painted with some light and dark lines? After all, this is just a flat painting. In fact, viewers intuitively make the assumption that these are furrowed instead of painted brows — even though the painting itself allows for both interpretations. The lesson is: bumps on smooth surfaces can often be convincingly conveyed by the lighting alone without any other cues (shadows, occlusions, parallax effects, stereo, etc.).

Normal Mapping
Normal mapping tries to convey bumps on smooth surfaces (i.e. coarse triangle meshes with interpolated normals) by changing the surface normal vectors according to some virtual bumps. When the lighting is computed with these modified normal vectors, viewers will often perceive the virtual bumps — even though a perfectly flat triangle has been rendered. The illusion can certainly break down (in particular at silhouettes) but in many cases it is very convincing.

More specifically, the normal vectors that represent the virtual bumps are first encoded in a texture image (i.e. a normal map). A fragment shader then looks up these vectors in the texture image and computes the lighting based on them. That's about it. The problem, of course, is the encoding of the normal vectors in a texture image. There are different possibilities and the fragment shader has to be adapted to the specific encoding that was used to generate the normal map.



Normal Mapping in Unity
The very good news is that you can easily create normal maps from gray-scale images with Unity: create a gray-scale image in your favorite paint program and use a specific gray for the regular height of the surface, lighter grays for bumps, and darker grays for dents. Make sure that the transitions between different grays are smooth, e.g. by blurring the image. When you import the image with Assets > Import New Asset change the Texture Type in the Inspector View to Normal map and check Generate from greyscale. After clicking Apply, the preview should show a bluish image with reddish and greenish edges. Alternatively to generating a normal map, the encoded normal map can be imported (don't forget to uncheck the Generate from greyscale box).

The not so good news is that the fragment shader has to do some computations to decode the normals. First of all, the texture color is stored in a two-component texture image, i.e. there is only an alpha component $$A$$ and one color component available. The color component can be accessed as the red, green, or blue component — in all cases the same value is returned. Here, we use the green component $$G$$ since Unity also uses it. The two components, $$G$$ and $$A$$, are stored as numbers between 0 and 1; however, they represent coordinates $$n_x$$ and $$n_y$$ between -1 and 1. The mapping is:

$$n_x = 2 A - 1$$  and   $$n_y = 2 G - 1$$

From these two components, the third component $$n_z$$ of the three-dimensional normal vector n$$=(n_x,n_y,n_z)$$ can be calculated because of the normalization to unit length:

$$\sqrt{n_x^2+n_y^2+n_z^2} = 1\quad \Rightarrow$$ $$n_z = \pm\sqrt{1 - n_x^2 - n_y^2}$$

Only the “+” solution is necessary if we choose the $$z$$ axis along the axis of the smooth normal vector (interpolated from the normal vectors that were set in the vertex shader) since we aren't able to render surfaces with an inwards pointing normal vector anyways. The code snippet from the fragment shader could look like this:

The decoding for devices that use OpenGL ES is actually simpler since Unity doesn't use a two-component texture in this case. Thus, for mobile platforms the decoding becomes: However, the rest of this tutorial (and also ) will cover only (desktop) OpenGL.

Unity uses a local surface coordinate systems for each point of the surface to specify normal vectors in the normal map. The $$z$$ axis of this local coordinates system is given by the smooth, interpolated normal vector N in world space and the $$x-y$$ plane is a tangent plane to the surface as illustrated in the image. Specifically, the $$x$$ axis is specified by the tangent attribute T that Unity provides to vertices (see the discussion of attributes in ). Given the $$x$$ and $$z$$ axis, the $$y$$ axis can be computed by a cross product in the vertex shader, e.g. B = N × T. (The letter B refers to the traditional name “binormal” for this vector.)

Note that the normal vector N is transformed with the transpose of the inverse model matrix from object space to world space (because it is orthogonal to a surface; see ) while the tangent vector T specifies a direction between points on a surface and is therefore transformed with the model matrix. The binormal vector B represents a third class of vectors which are transformed differently. (If you really want to know: the skew-symmetric matrix B corresponding to “B×” is transformed like a quadratic form.) Thus, the best choice is to first transform N and T to world space, and then to compute B in world space using the cross product of the transformed vectors.

With the normalized directions T, B, and N in world space, we can easily form a matrix that maps any normal vector n of the normal map from the local surface coordinate system to world space because the columns of such a matrix are just the vectors of the axes; thus, the 3×3 matrix for the mapping of n to world space is:

$$\mathrm{M}_{\text{surface}\to \text{world}} = \left[ \begin{matrix} T_x & B_x & N_x \\ T_y & B_y & N_y \\ T_z & B_z & N_z \end{matrix} \right]$$

These calculations are performed by the vertex shader, for example this way: The factor  in the computation of   is specific to Unity, i.e. Unity provides tangent vectors and normal maps such that we have to do this multiplication.

In the fragment shader, we multiply the matrix in  with n. For example, with this line:

With the new normal vector in world space, we can compute the lighting as in.

Complete Shader Code
This shader code simply integrates all the snippets and uses our standard two-pass approach for pixel lights. Note that we have used the tiling and offset uniform  as explained in the  since this option is often particularly useful for bump maps.

Summary
Congratulations! You finished this tutorial! We have look at:
 * How human perception of shapes often relies on lighting.
 * What normal mapping is.
 * How Unity encodes normal maps.
 * How a fragment shader can decode Unity's normal maps and use them to per-pixel lighting.