# Sobel Edge Detection on Depth Texture

I'm currently trying to implement contour shading via edge detection on a depth texture.

I'm rendering the color and depth information onto two textures. In my post processing shader I do the following:

``````out vec4 FragColor;
in vec2 TexCoords;
uniform sampler2D colorTexture;
uniform sampler2D depthTexture;
uniform float far;
uniform float near;
``````

Then the matrices for the sobel filter:

``````mat3 sobel_y = mat3(
1.0, 0.0, -1.0,
2.0, 0.0, -2.0,
1.0, 0.0, -1.0
);

mat3 sobel_x = mat3(
1.0, 2.0, 1.0,
0.0, 0.0, 0.0,
-1.0, -2.0, -1.0
);
``````

The function to linearize the Depthvalue

``````float LinearizeDepth(float z)
{
float n = near;
float f = far;
return (2.0 * n) / (f + n - z * (f - n));
}
``````

In the main I'm filling the matrix to calculate the gradients and then substract the result of the diffuseColor to get Black if the gradient is high (an edge) and just output the diffuseColor if the gradient is low.

``````void main()
{
vec3 colorDiff = texture(colorTexture, TexCoords).rgb;
mat3 I;
vec3 texel;
for (int i=0; i<3; i++) {
for (int j=0; j<3; j++) {
float depth = LinearizeDepth(texture(depthTexture, TexCoords + vec2(i-1, j-1)).r);
I[i][j] = depth;
}
}

float gx = dot(sobel_x, I) + dot(sobel_x, I) + dot(sobel_x, I);
float gy = dot(sobel_y, I) + dot(sobel_y, I) + dot(sobel_y, I);

float g = sqrt(pow(gx, 2.0)+pow(gy, 2.0));

FragColor = vec4(colorDiff - vec3(g), 1.0);
}
``````

My problem is that g is always 0 since i just the the normal color as output.

If i just output the depth values it does look correct: Any Idea what I'm doing wrong here?

• Here's your problem:

``````for (int i=0; i<3; i++) {
for (int j=0; j<3; j++) {
float depth = LinearizeDepth(texture(depthTexture, TexCoords + vec2(i-1, j-1)).r); I[i][j] = depth;
}
}
``````

Texture coordinates are usually in the range 0...1, where 0 on the x axis is the left edge of the texture and 1 is the right edge, regardless of the resolution of the texture.

So when you add/subtract 1 from the texture coordinate, you're not nudging the sample over by one pixel, you're jumping the entire width of the texture. If the wrap mode is set to repeat, then you're wrapping all the way around the texture and back to the same spot.

So naturally your Sobel filter doesn't find an edge: it's presented with a grid of 9 identical samples, the same as it would see in a field of a single flat colour.

Instead, you'll want to offset your samples by ±1/(texture size) to move only one pixel over.

• I'm not sure I entirely understand your question, but I think you can take the gradient of the depth that you've calculated in `g` and use that as an overlay for the color you sampled from the color channels. Perhaps something like this (untested):

``````float inkAmount = g / kMaxGradientValue;
``````

Then mix the color texture with black (or whatever color your outline ink is) via something like this:

``````FragColor = mix(colorDiff, inkColor, inkAmount);
``````

In the above `kMaxGradientValue` is the maximum value that the gradient can have. It's been a while since I've worked with Sobel operators, but I think the max is something like `sqrt(18) = ~4.24`, assuming your R, G, & B are all in the 0-1 range.

You might need or want to adjust the white and black points of the ink amount to deal with noise and small variations in the gradient.