[BUG]: Shadow of Rome: missing/delayed character shadows in Software Mode

[Dreadmoth]

Describe the Bug

Shadow of Rome [SLES-52950]

When software rendering is used. shadows of NPCs and/or the player character are not visible (or are only partially visible) in some areas of the game (such as "The Forum" during daytime).
In some other scenes (such as the first arena battle), shadows appear several seconds after entering the area.

Reproduction Steps

Run the attached dump, observe missing shadows of NPCs on software renderer.

To reproduce in the game itself:

A. Using the attached memory card (to observe missing NPC shadows in "The Forum"):

1. Launch game.
2. Select "Continue" at main menu.
3. Walk down stairs behind and to the left of the player character.
4. Walk up stairs ahead to the left.
5. Continue forward, turn right when possible.
6. Walk up stairs until a "Leave this area?" prompt appears, select "Yes".
7. Select "The Forum" location, press ✖.
8. Walk forward, observe missing shadows of NPCs on software renderer. The shadow of the head/helmet of one NPC may be visible.

B. To observe shadows appearing after a delay in the first arena battle:

1. Launch game.
2. Select "New Game" at main menu.
3. Select a difficulty level.
4. Watch, or skip intro FMV by pressing START.
5. Advance through dialogue by pressing ✖ several times.
6. Turn right, approach the guard near the gate, press ⭘ to talk.
7. Advance through dialogue by pressing ✖.
8. Select "Of course.".
9. Advance through dialogue by pressing ✖.
10. After the loading screen, observe missing shadows of NPCs and player character on software renderer.
11. Close the tutorial pop-up by pressing ✖.
12. Wait about 5 seconds, observe as shadows appear. (may coincide with the appearance of a second tutorial pop-up, depending on how quickly the first was closed)

Expected Behavior

Character shadows rendered, without delays - as seen on hardware renderers and PS2.

Character shadows rendering in "The Forum" area have been confirmed on a real PS2.
Screen captures by refraction: one, two

I can't currently test the first arena battle on my PS2, but gameplay videos that seem to be from real hardware (e.g. this one, which was uploaded before PCSX2 could run the game without issues) show shadows rendered without a delay.

PCSX2 Revision

1.7.5939

Operating System

Windows 10 (64bit)

If Linux - Specify Distro

No response

CPU

AMD Ryzen 7 7800X3D

GPU

Nvidia GeForce GTX 1070 Ti

GS Settings

Renderer: Software

Emulation Settings

No response

GS Window Screenshots

"The Forum" area:

Software renderer (NPC shadows always missing):

Hardware renderer:

---

First arena battle:

Software renderer (player character and NPC shadows appear after a delay):

Shadow.of.Rome_SLES-52950_Software.mp4

Hardware renderer:

Shadow.of.Rome_SLES-52950_hardware.mp4

Logs & Dumps

GS dump

Memory card (SLES-52950)

[TJnotJT]

The main character and NPC shadows appear to be unrelated issues.

NPC shadows: caused by incorrect rounding when SW renderer does fogging. Game draws a texture for shadows but leaves fogging enabled with F = 255 for all vertices. This should be a NOP but SW renderer's fog is off by 1. Can be fixed by disabling fog in this case:

if (PRIM->FGE)
{
	gd.sel.fge = 1;

	gd.frb = env.FOGCOL.U32[0] & 0x00ff00ff;
	gd.fga = (env.FOGCOL.U32[0] >> 8) & 0x00ff00ff;
}

to

if (PRIM->FGE && !(m_vt.m_min.p.w == 255.0f && m_vt.m_max.p.w == 255.0f))
{
	gd.sel.fge = 1;

	gd.frb = env.FOGCOL.U32[0] & 0x00ff00ff;
	gd.fga = (env.FOGCOL.U32[0] >> 8) & 0x00ff00ff;
}

in GSRendererSW.cpp. Not fully tested but shouldn't have side effects.

Main character shadows: caused by overflow when casting float -> int with infinite ST coordinates (Q is 0 in the draw and clamped to FLT_MIN, which gives 2^TW * S/Q = infinty). Can be fixed by clamping infinite ST to a finite value. Also not tested, could have breaking side effects.

[refractionpcsx2]

I feel like the SW fogging should be fixed if it's off by one, rather than hacking the end value, if all the other values are going to be wrong too.

[TJnotJT]

Agreed, it would be good for the overall accuracy to fix it in the SW rasterizer also (though note that the fix I suggested doesn't hack the end value; it disables fogging in the draw setup if F = 255 across the whole draw). It's strange cause the fogging equation is not exactly complicated and the fog should be constant in this case. Need to invesigate more thoroughly.

[TJnotJT]

The fogging issue seems to be is caused by the rounding behavior of GSVector4::modulate16<0>(). It seems this was already known because changing this in GSDrawScanline.cpp fixes the issue (with scanline JIT code gen disabled):

rb = frb.lerp16<0>(rb, fog);
ga = fga.lerp16<0>(ga, fog).mix16(ga);

/*
fog = fog.srl16<7>();

VectorI ifog = VectorI::x00ff().sub16(fog);

rb = rb.mul16l(fog).add16(frb.mul16l(ifog)).srl16<8>();
ga = ga.mul16l(fog).add16(fga.mul16l(ifog)).srl16<8>().mix16(ga);
*/

to

// rb = frb.lerp16<0>(rb, fog);
// ga = fga.lerp16<0>(ga, fog).mix16(ga);

fog = fog.srl16<7>();

VectorI ifog = VectorI::x00ff().sub16(fog);

rb = rb.mul16l(fog).add16(frb.mul16l(ifog)).srl16<8>();
ga = ga.mul16l(fog).add16(fga.mul16l(ifog)).srl16<8>().mix16(ga);

I guess it was assumed that off by 1 in fogging wouldn't break anything so it was optimized a bit.

Edit: Appears to be the same as this issue: #5898.

[TellowKrinkle]

BTW just so you know, that commented out code in the C scanline draw is actually not correct either according to the hardware tests in #5898.

[TellowKrinkle]

When I test the SW renderer with fog fixed by #13171, the player character is still missing a shadow. Is that just an issue with the dump or is the game like that as well?

[TJnotJT]

BTW just so you know, that commented out code in the C scanline draw is actually not correct either according to the hardware tests in #5898.

I see, thanks for explaining.

When I test the SW renderer with fog fixed by #13171, the player character is still missing a shadow. Is that just an issue with the dump or is the game like that as well?

Yes, this is expected as the player character shadow and NPC shadows are caused by different things (player shadow is due to infinite S in vertices).

[Dreadmoth]

#12880 appears to be when the player character shadow disappeared from "The Forum" area:

v2.3.440: no shadows

v2.3.439: PC shadow