This function is called too often wrongly to retrieve immediate values, displacements or offsets in an instruction. You should use a MEM_READ_BYTE (3 T-states) and not an OPCODE_READ_M1 (4 T-states) for them!
There is no M1 = 0 and no refresh when retrieving immediate values, displacements or offsets in an instruction!
See here a complex example here: https://floooh.github.io/2021/12/06/z80-instruction-timing.html#dd-and-fd-prefixes. Be aware the table are showing steps per half-cycle (half T-state) so a MEM_READ_BYTE would be 6 rows for 3 T-states.