gh-135700: Fix instructions in __annotate__ have incorrect code positions

[AndPuQing]

• Issue: some instructions in __annotate__ have incorrect code positions #135700

cc @15r10nk @JelleZijlstra @iritkatriel

[iritkatriel]

This test passes for me on main (without this PR).

[15r10nk]

The problem is the indentation. @AndPuQing indented my code in the strings which (for some unknown reason for me) caused the test to pass. a textwrap.dedent should help.

[AndPuQing]

Yes, I fixed the indent problem, but it seems that the current fix also cannot pass the test.

======================================================================
FAIL: test_issue135700 (test.test_pdb.PdbTestCase.test_issue135700)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/data/xxx/dev/cpython/Lib/test/test_pdb.py", line 3842, in test_issue135700
    self.assertNotIn("(1)__annotate__()", result[0])
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '(1)__annotate__()' unexpectedly found in '> ...

../cpython/python -m dis testmod.py                                                                            (base) 
  --           MAKE_CELL                0 (__conditional_annotations__)

   0           RESUME                   0

   1           LOAD_CONST               2 (<code object __annotate__ at 0x798bd679cd50, file "testmod.py", line 1>)
               MAKE_FUNCTION
               STORE_NAME               2 (__annotate__)
               BUILD_SET                0
               STORE_NAME               0 (__conditional_annotations__)

   3           LOAD_BUILD_CLASS
               PUSH_NULL
               LOAD_CONST               0 (<code object ClassVar at 0x798bd6bb1940, file "testmod.py", line 3>)
               MAKE_FUNCTION
               LOAD_CONST               1 ('ClassVar')
               CALL                     2
               STORE_NAME               1 (ClassVar)

   5           LOAD_NAME                0 (__conditional_annotations__)
               LOAD_SMALL_INT           0
               SET_ADD                  1
               POP_TOP
               LOAD_CONST               3 (None)
               RETURN_VALUE

Disassembly of <code object ClassVar at 0x798bd6bb1940, file "testmod.py", line 3>:
  3           RESUME                   0
              LOAD_NAME                0 (__name__)
              STORE_NAME               1 (__module__)
              LOAD_CONST               0 ('ClassVar')
              STORE_NAME               2 (__qualname__)
              LOAD_SMALL_INT           3
              STORE_NAME               3 (__firstlineno__)

  4           LOAD_CONST               1 (())
              STORE_NAME               4 (__static_attributes__)
              LOAD_CONST               2 (None)
              RETURN_VALUE

Disassembly of <code object __annotate__ at 0x798bd679cd50, file "testmod.py", line 1>:
  1           RESUME                   0
              LOAD_FAST_BORROW         0 (format)
              LOAD_SMALL_INT           2
              COMPARE_OP             132 (>)
              POP_JUMP_IF_FALSE        3 (to L1)
              NOT_TAKEN
              LOAD_COMMON_CONSTANT     1 (NotImplementedError)
              RAISE_VARARGS            1
      L1:     BUILD_MAP                0

  5           LOAD_SMALL_INT           0
              LOAD_GLOBAL              0 (__conditional_annotations__)
              CONTAINS_OP              0 (in)
              POP_JUMP_IF_FALSE       10 (to L2)
              NOT_TAKEN
              LOAD_GLOBAL              2 (ClassVar)
              COPY                     2
              LOAD_CONST               1 ('__dataclass_fields__')

  1           STORE_SUBSCR
      L2:     RETURN_VALUE

[15r10nk]

hmm, @AndPuQing your fix solved the problem which I have reported in the issue (the positions in the bytecode and column information) but not the problem with pdb because the line number of the synthetic code in __annotate__ is still 1. I don't know if there is a way to express an non existing code location (-1 maybe), but this is what would be needed here. Another solution might be to use the location of the first ast-node which is handled by the code (line 5 in this case). This would cause no line change when the code is executed. I thing this might be the most practical solution here.

[iritkatriel]

Cc @gaogaotiantian

[AndPuQing]

Hi everyone, here is the progress. I think this is because the loc passed to codegen_process_deferred_annotations is the first line of AST/module body.

So I experimentally got the first annotation stmt through deferred_anno in codegen_process_deferred_annotations and then corrected the position of annotations "setup" opcode. From the test, the breakpoint no longer stops at (1)__annotate__. The following is the execution output.

./python -m pdb ../playground/main.py                                                          (base) 
> /mnt/data/xxx/dev/playground/main.py(1)<module>()
-> import testmod
(Pdb) b testmod.py:1
Breakpoint 1 at /mnt/data/xxx/dev/playground/testmod.py:1
(Pdb) c
> /mnt/data/xxx/dev/playground/testmod.py(1)<module>()
-> 22
(Pdb) c
{'__dataclass_fields__': <class 'testmod.ClassVar'>}
The program finished and will be restarted
> /mnt/data/xxx/dev/playground/main.py(1)<module>()
-> import testmod
(Pdb) c
{'__dataclass_fields__': <class 'testmod.ClassVar'>}
The program finished and will be restarted
> /mnt/data/xxx/dev/playground/main.py(1)<module>()
-> import testmod
(Pdb)

> ../cpython/python -m dis testmod.py                                                                            (base) 
  --           MAKE_CELL                0 (__conditional_annotations__)

   0           RESUME                   0

# <--- Please let me know if this position is weird or correct, 
   8           LOAD_CONST               2 (<code object __annotate__ at 0x7d090a29cd50, file "testmod.py", line 8>)
               MAKE_FUNCTION
               STORE_NAME               2 (__annotate__)
               BUILD_SET                0
               STORE_NAME               0 (__conditional_annotations__) 

   1           NOP

   4           LOAD_BUILD_CLASS
               PUSH_NULL
               LOAD_CONST               0 (<code object ClassVar at 0x7d090a6b1940, file "testmod.py", line 4>)
               MAKE_FUNCTION
               LOAD_CONST               1 ('ClassVar')
               CALL                     2
               STORE_NAME               1 (ClassVar)

   8           LOAD_NAME                0 (__conditional_annotations__)
               LOAD_SMALL_INT           0
               SET_ADD                  1
               POP_TOP
               LOAD_CONST               3 (None)
               RETURN_VALUE

Disassembly of <code object ClassVar at 0x7d090a6b1940, file "testmod.py", line 4>:
  4           RESUME                   0
              LOAD_NAME                0 (__name__)
              STORE_NAME               1 (__module__)
              LOAD_CONST               0 ('ClassVar')
              STORE_NAME               2 (__qualname__)
              LOAD_SMALL_INT           4
              STORE_NAME               3 (__firstlineno__)

  5           LOAD_CONST               1 (())
              STORE_NAME               4 (__static_attributes__)
              LOAD_CONST               2 (None)
              RETURN_VALUE

Disassembly of <code object __annotate__ at 0x7d090a29cd50, file "testmod.py", line 8>:
  8           RESUME                   0
              LOAD_FAST_BORROW         0 (format)
              LOAD_SMALL_INT           2
              COMPARE_OP             132 (>)
              POP_JUMP_IF_FALSE        3 (to L1)
              NOT_TAKEN
              LOAD_COMMON_CONSTANT     1 (NotImplementedError)
              RAISE_VARARGS            1
      L1:     BUILD_MAP                0
              LOAD_SMALL_INT           0
              LOAD_GLOBAL              0 (__conditional_annotations__)
              CONTAINS_OP              0 (in)
              POP_JUMP_IF_FALSE       10 (to L2)
              NOT_TAKEN
              LOAD_GLOBAL              2 (ClassVar)
              COPY                     2
              LOAD_CONST               1 ('__dataclass_fields__')
              STORE_SUBSCR
      L2:     RETURN_VALUE