MathJax example

Intro

The Inverse Hull method is a technique used in computer graphics to create outline shading for 3D objects. It is often employed to give objects a more illustrative or stylized appearance by emphasizing their silhouettes or edges.

The core concept involves extending the initial geometry along the directions of its polygon normals and subsequently only rendering the polygons that are facing away from the camera.
3D character model with no outline3d character model with an outline

Implementation

Numerous rendering engines include their own predefined Toon or Stylized shading models that can automatically generate outlines for specific 3D objects. For instance, the Eevee render engine offers a material backface culling option, streamlining and optimizing the outline rendering process. Nonetheless, diverse render engines and 3D software demand distinct shader configurations. In this technique, I will outline a process to generate outlines at the geometry level, ensuring compatibility across various render engines without necessitating extra setup. Note that the drawback of this approach is its relatively slower performance compared to built-in methods.

Here is an example code that groups front facing polygons, which we can then delete leaving the outline geometry.
View VEX code
 
// Get the transformation matrix of the specified camera object
matrix Matrix = optransform(chs("camera"));

// Calculate the camera's world space position by transforming the origin (0,0,0) using the matrix
vector camPos = cracktransform(0, 0, 0, {0, 0, 0}, Matrix);

// Calculate the normalized vector representing the view direction from the camera to the current point
vector viewRay = normalize(@P - camPos);

// Define a threshold angle (in radians) for visibility determination
float threshold = chf("limit");

// Check if the dot product between the view direction and the point's normal is below the threshold
if (dot(viewRay, v@N) < threshold) {
    i@group_invisible = 1; // Mark the point as invisible
} else {
    i@group_invisible = 0; // Mark the point as visible
}