Skewmesh Technique

Normal map baking is an art unto itself. In our role as an art outsrouce team, we experiment a lot with new ideas to improve our work, and give better results to our clients. As such, I developed a technique to remove skewing/strange waviness from normal map bakes: Enter the "skewmesh" technique.  This tutorial assumes you're using 3dsmax, but it should work in any equivalent application.

So what is it we're trying to solve? 

When baking a normal map , it's important to understand that the vertex normals of your low poly model are averaged in relation to the face normal, so we get skewed results for our normal map details, as seen here on the right:

Our preference would be no skewing of the details, like so:

in order to achieve this clean bake effect, we use an intermediate mesh known as a "skew mesh". A skewmesh is just your low poly model with a tesselate modifier added to it. Please note that the tension is set to 0. We do this to prevent strange mesh smoothing on the sew mesh. 

Once you've created the skewmesh, you can use it to bake your normal map from your high polygon model. Make sure to bake an object space map (local space/XYZ).

Once you've baked your object space map unto your skewmesh, you'll notice that due to the extra points/raycasts, the skewing is gone. The next step is to rebake this local space map to a tangent space map on your actual low polygon model. When doing this transfer, you may want to turn off filter maps in the render setup tp prevent graininess in your normal map(found in "Render Setup -> renderer" tab). Also, make sure to choose "tangent" space maps un the baking options window.

When you've rebaked , you will transfer the clean bake from your skewmesh to your low poly, and all of the skewing will be gone. 

You can do this in max, or Xnormal. Both of which I've done a video tutorial for. Ignore the wierd bake at the end of the Xnromal video, I messed up the settings, but the process is the same.  This works great on complex shapes as well:

Use this technique, and you'll never have awful waviness again.

Peter Kojesta
http://www.ExisInteractive.com

3ds Max:

Xnormal: