-
-
Notifications
You must be signed in to change notification settings - Fork 32.1k
tokenize._all_string_prefixes does not list t-string prefixes #134675
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I wrote a Python program that attempts all possible prefixes; it tries everything with
|
Also not mentioned in https://docs.python.org/3.15/reference/lexical_analysis.html#string-and-bytes-literals |
I guess the best way to test this is to use @larryhastings 's exhaustive test, and find all 2 (and maybe throw in 3 if it's not too slow) letter combinations, and see that it matches _all_string_prefixes. I don't see a way to test this using a public API: |
I'm thinking of using this: import string
import itertools
# Try all lengths until we find a length that has zero valid prefixes.
# This will miss the case where for example there are no valid 3
# character prefixes, but there are valid 4 character prefixes.
valid_prefixes = set()
for length in itertools.count():
num_at_this_length = 0
for prefix in (
"".join(l) for l in list(itertools.combinations(string.ascii_lowercase, length))
):
for t in itertools.permutations(prefix):
for u in itertools.product(*[(c, c.upper()) for c in t]):
p = ''.join(u)
if p == "not":
# 'not' can never be a string prefix, because it's a valid
# expression: not ""
continue
try:
eval(f'{p}""')
valid_prefixes.add(p)
num_at_this_length += 1
except SyntaxError:
pass
if num_at_this_length == 0:
break
print(valid_prefixes)
print(len(valid_prefixes)) Which in 3.9 produces:
And in 3.15 produces:
I'll produce a PR shortly. |
…s doc, and add a test to make sure we catch this error in the future. (#134734) * Add t-string prefixes to _all_string_prefixes, and add a test to make sure we catch this error in the future. * Update lexical analysis docs for t-string prefixes.
…nalysis doc, and add a test to make sure we catch this error in the future. (pythonGH-134734) * Add t-string prefixes to _all_string_prefixes, and add a test to make sure we catch this error in the future. * Update lexical analysis docs for t-string prefixes. (cherry picked from commit 08c78e0) Co-authored-by: Eric V. Smith <ericvsmith@users.noreply.github.com>
…analysis doc, and add a test to make sure we catch this error in the future. (GH-134734) (#134739) gh-134675: Add t-string prefixes to tokenizer module, lexical analysis doc, and add a test to make sure we catch this error in the future. (GH-134734) * Add t-string prefixes to _all_string_prefixes, and add a test to make sure we catch this error in the future. * Update lexical analysis docs for t-string prefixes. (cherry picked from commit 08c78e0) Co-authored-by: Eric V. Smith <ericvsmith@users.noreply.github.com>
Uh oh!
There was an error while loading. Please reload this page.
Bug report
Bug description:
tokenize._all_string_prefixes()
does not include t-string prefixes.This also affects
tokenize.endpats
andtokenize.StringPrefix
.CPython versions tested on:
3.15
Operating systems tested on:
Windows
Linked PRs
The text was updated successfully, but these errors were encountered: