Uniformly Darken Areas in the Scene Using a Triangle List, How?

[hytel]

I have many things in my VSG scene. Some at ground level, some above. I have a triangle list, coplanar with the ground, that I'm currently using to color areas on the ground. You can think of it as shading where on the ground there are pipes and other features below the surface at all depths. So I draw the ground in the scene, then I draw the opaque triangles on top, then I draw the rest of the scene. Looks good. But what I really need is to use the color to "shade" the ground slightly where the triangles are, and not overlay it. So, I decided to enable blending. That doesn't work because many of the triangles are coplanar. When they are draw opaque, this isn't obvious because they just overwrite each other (I have depth test off). But, when I draw them semi-transparent, they blend the ground darker, but more so in areas with greater triangle overlap. I need a CONSISTENT blending across all pixels in the triangle list.

I believe I'm going to need to do two passes for the scene to solve this, once to set the stencil buffer for areas where the triangles are, and then another to use the stencil pixels to blend a color with the existing pixels. Is this the only way to do this? Is there an easier way? If I wanted to do this two pass algorithm with the stencil buffer, how would I set this up in VSG?

Note: I'm also using the ViewDependentState shadowing infrastructure, so I don't want to interfere with that.

[AnyOldName3]

This sounds very much like the kind of situation the stencil buffer exists for, so that would probably be the most sensible approach. As a first attempt, I'd try adding state to the state group for your triangle list to disable depth and colour writes but enable stencil writes, then add another drawable that gets drawn later which does a stencil test and covers the whole screen with a single large triangle. It shouldn't be too complicated.

[hytel]

Wow! This would be INFINITLY easier to do than what I was going to do. Gosh, I hope this will work. Ok, so I first added a clear stencil to the scene graph early on with a:

    VkClearValue clearStencilValue{};
    clearStencilValue.depthStencil = {0.0f, 0};
    VkClearAttachment stencil_attachment{VK_IMAGE_ASPECT_STENCIL_BIT, 1, clearStencilValue};
    VkClearRect rect{m_camera->getRenderArea(), 0, 1};
    auto clearAttachments = vsg::ClearAttachments::create(vsg::ClearAttachments::Attachments{stencil_attachment},
                                                                                             vsg::ClearAttachments::Rects{rect});

Not sure if when the screen is resized, the attachment rect automatically changes - do I have to manage this?  I don't see any of the VSG examples doing this?

Next, I turned off writing to the color buffer by:

     VkPipelineColorBlendAttachmentState colorBlendAttachment = {};
     colorBlendAttachment.blendEnable = VK_TRUE;
     colorBlendAttachment.colorWriteMask = 0;
     auto colorBlendState = vsg::ColorBlendState::create(vsg::ColorBlendState::ColorBlendAttachments{colorBlendAttachment});

Then I draw my triangles with a vsg::DepthStencilState that should write to the stencil buffer, but not the depth buffer:

                dss.depthTestEnable = VK_FALSE;
                dss.depthWriteEnable = VK_FALSE;
                dss.stencilTestEnable = VK_FALSE;
                dss.front.writeMask = 0xFF;
                dss.front.compareMask = 0x0;
                dss.front.failOp = VK_STENCIL_OP_KEEP;
                dss.front.passOp = VK_STENCIL_OP_REPLACE;
                dss.back.writeMask = 0xFF;
                dss.back.compareMask = 0x0;
                dss.back.failOp = VK_STENCIL_OP_KEEP;
                dss.back.passOp = VK_STENCIL_OP_REPLACE;

Can't tell if this so far is working - though the black triangles are indeed not writing to the color buffer. Finally, I need to do the draw over the screen with a large rectangle using the stencil buffer as a guide. I'm pretty sure I know how to do exactly this EXCEPT how to get the geometry to fill the whole screen. The VSG examples use a different projection to make this easy. But I don't want to change my perspective projection. How do I create a flat sheet to cover my window viewport without changing my projection? I will have the scene elements added in order draw terrain using depth, clear stencil, draw triangles to stencil, draw rectangle blending using stencil, draw even more objects above the ground with depth. To do this, I need these child nodes in the scene together, so I don't want to create other camera, etc, if possible.

[AnyOldName3]

The easiest way is probably just to make a triangle with vertex coordinates at

$\begin{pmatrix}-1\\-1\end{pmatrix}$, $\begin{pmatrix}3\\-1\end{pmatrix}$, $\begin{pmatrix}-1\\3\end{pmatrix}$

and render it with a vertex shader that just passes those coordinates straight through to the rasteriser. If you don't use the view or projection matrices, then the current view and projection details don't matter.

[hytel]

Interesting. But, up until now, I've let VSG handle most of the vertex/fragment shading (using the phong and flat shaders) because it knows about things like blending which I'll need in the final stencil blending step - though I guess I could do this too in the custom shaders. Also, why a triangle and not a quad? What is the 3 for? What are the raw coordinates of a quad fitting the four corners of the viewport?

[AnyOldName3]

The corners of the screen from the rasteriser's point of view range from -1 to 1, so to cover all of that with a single triangle, you need to go beyond the edge, and that's where the threes come from. It's pretty common that people suggest drawing a fullscreen quad instead of a single triangle, and it shouldn't make much difference here, but for more expensive effects, there are more cache misses and worse occupancy for the regions of the screen that cover the edges of the two triangles making up the quad, so it's better to get in the habit of using one big triangle.

For something as simple as this, your shaders could simply be

#version 450

layout(location = 0) in vec2 vsg_Vertex;

out gl_PerVertex{
    vec4 gl_Position;
};

void main()
{
    gl_Position = vec4(vsg_Vertex, 0.0, 1.0);
}

for the vertex shader and

#version 450

layout(location = 0) out vec4 outColor;

void main()
{
    outColor = vec4(/* your colour here*/);
}

for the fragment shader.

[hytel]

Wow! That vertex/fragment shader with the SINGLE triangle works! I got blending on the ground! This is SOOO much simpler than how I was going to do it. Thanks very much!

[hytel]

I have a problem, and I can't seem to track down why something drawn earlier in the scene graph would "show through" something later in the scene graph. All I'm trying to do is have shadow casting models in the scene on top of the terrain, but the "ground darkening" I'm doing using the stencil buffer is showing through. My scene graph looks like this:

1. Skybox Group - doesn't write to the depth buffer

2. Sun Group - Draws a sun on the periphery

3. Terrain Group - Draws tiles using the depth buffer - terrain is flat

4. Shadow Group children:

   a) Clear Stencil Buffer

   b) Draw to stencil buffer areas of the ground that need to be darkened using terrain co-planer triangles with:
   depthTestEnable = VK_FALSE
   depthWriteEnable = VK_FALSE
   depthBoundsTestEnable = VK_FALSE
   stencilTestEnable = VK_TRUE
   depthCompareOp = VK_COMPARE_OP_GREATER
   Stencil Op Front:
   Fail Op: VK_STENCIL_OP_KEEP
   Pass Op: VK_STENCIL_OP_REPLACE
   Depth Fail Op: VK_STENCIL_OP_KEEP
   Compare Op: VK_STENCIL_OP_DECREMENT_AND_WRAP
   Compare Mask: 000000ff
   Write Mask: 000000ff
   Reference: 00000001
   Stencil Op Back:
   Fail Op: VK_STENCIL_OP_KEEP
   Pass Op: VK_STENCIL_OP_REPLACE
   Depth Fail Op: VK_STENCIL_OP_KEEP
   Compare Op: VK_STENCIL_OP_DECREMENT_AND_WRAP
   Compare Mask: 000000ff
   Write Mask: 000000ff
   Reference: 00000001
   Min Depth Bounds = 0.00
   Max Depth Bounds = 1.00

   c) Use a screen covering triangle and the stencil buffer to shade areas on the ground that need to be darkened using:
   depthTestEnable = VK_FALSE
   depthWriteEnable = VK_FALSE
   depthBoundsTestEnable = VK_FALSE
   stencilTestEnable = VK_TRUE
   depthCompareOp = VK_COMPARE_OP_GREATER
   Stencil Op Front:
   Fail Op: VK_STENCIL_OP_KEEP
   Pass Op: VK_STENCIL_OP_KEEP
   Depth Fail Op: VK_STENCIL_OP_KEEP
   Compare Op: VK_STENCIL_OP_REPLACE
   Compare Mask: 000000ff
   Write Mask: 00000000
   Reference: 00000001
   Stencil Op Back:
   Fail Op: VK_STENCIL_OP_KEEP
   Pass Op: VK_STENCIL_OP_KEEP
   Depth Fail Op: VK_STENCIL_OP_KEEP
   Compare Op: VK_STENCIL_OP_REPLACE
   Compare Mask: 000000ff
   Write Mask: 00000000
   Reference: 00000001
   Min Depth Bounds = 0.00
   Max Depth Bounds = 1.00
   d) Clear the stencil buffer again (though not using in my geometry after this

5. Model Group: Draw many models on the terrain using VSGs reader and a vsg::MatrixTransform to place them. They do use the depth buffer.

I have used one of the VSG examples to write out the depth buffer and confirmed that up until #5, the scene only has depth for round Sun and terrain. The depth buffer isn't affected by the stencil buffer darkening process.

But, even though the models are drawn very late (after skybox, sun, terrain, shadow darkening sequence), all use the depth buffer, and none use the stencil buffer, the darkened areas are STILL BLEEDING THROUGH THE MODELS. What am I missing? Anything I can check to track this issue down?

[AnyOldName3]

How are you enforcing the draw order? It's not necessarily the same as the order the nodes were added to the scenegraph, e.g. if you use a renderlayer or anything with depth sorting.

[hytel]

I don't see any renderlayer in VSG, but there is vsg::Layer which I'm not using anywhere in the code. I do have shadows active and I do believe it does do "passes" somehow, but I've turned off shadow casting for the terrain and the stencil blend groups just in case. The self-shadowing is working great. The darkened areas on the ground are being draw to the color buffer, and they are definitely not being drawn to the depth buffer. It could very well be a draw order, but how can I easy force the models to draw late - which is should be already?

[hytel]

Unbelievable. I figured it out. I wrote my own cool scene graph printer (example one only does top-level node elements) and it shows me that the scene group is perfect. But in two different places in the code, I add the scene to the vsg::View. So the view ran through the scene twice I guess? I'd have to think about it, but I guess it rendered over the previous one. Dunno. I need to be careful. You can addChild to the vsg::View, but you also get the scene added to it by vsg::View::create(camera, scene). Didn't realize it added it as a courtesy! Sheesh.

[robertosfield]

I haven't followed this thread closely as I've had lots of other work on, but perhaps I can chip in here with a general suggestion.

Using RenderDoc and vsg::GpuAnnotation together can be really useful in showing what is being rendered and what parts of the scene graph they are being rendered from. So if you have a subgraph that is rendered twice then hopefully you'd spot this.

[hytel]

Very cool. I tried adding the debugAPI boolean but the output is unreadable. Next time I'll try your two debugging aids. The complexity in Vulkan is daunting. The one problem I have left right now is during an offscreen snapshot of the view described above with the shadows and terrain darkened areas. Maybe you can help. As described above, to render the darkened areas, I do a three step in my scene graph: clear stencil, draw to stencil, use stencil to shade areas of terrain. This works like a champ. Love it. The terrain is being darkened in certain areas perfectly. To do the offscreen rendering/save, I'm use a refactored version of your offscreen rendering example with the offscreen render buffer sized differently from the main view. With a few tweaks, this mostly works. But, because I share the scene graph (main view and offscreen), I'm getting various warning and issues concerning the Clear Attachment that I put in place to clear the stencil bits. It doesn't like the size of the rect for the attachment which makes sense because each view is sized differently and yet they are sharing the same scene graph. Any easy way around this?