Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-104016: Fixed off by 1 error in f string tokenizer #104047

Merged
merged 9 commits into from
May 1, 2023

Conversation

jx124
Copy link
Contributor

@jx124 jx124 commented May 1, 2023

@cpython-cla-bot
Copy link

cpython-cla-bot bot commented May 1, 2023

All commit authors signed the Contributor License Agreement.
CLA signed

@bedevere-bot
Copy link

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

blurb-it bot and others added 4 commits May 1, 2023 16:35
Co-Authored-By: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
Co-Authored-By: Ken Jin <kenjin@python.org>
@arhadthedev arhadthedev added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label May 1, 2023
@Fidget-Spinner Fidget-Spinner added the type-crash A hard crash of the interpreter, possibly with a core dump label May 1, 2023
@Fidget-Spinner
Copy link
Member

DIsclaimer: This is my friend who I'm helping guide to contribute to CPython. I will thus exclude myself from approving this to prevent any conflict of interest which may arise.

Lib/test/test_fstring.py Outdated Show resolved Hide resolved
@pablogsal
Copy link
Member

pablogsal commented May 1, 2023

I was actually preparing a fix for this problem. I think the correct approach here is to separate the two limits. I will update your PR with my approach and we can land it afterwards as we need this fix to build upon for other fixes. In other circumstances we could iterate together but beta freeze is too close 😅

@bedevere-bot
Copy link

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

@pablogsal
Copy link
Member

Great job @jx124 🚀

@pablogsal pablogsal merged commit 5078eed into python:main May 1, 2023
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-wasi 3.x has failed when building commit 5078eed.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1046/builds/1882) and take a look at the build logs.
  4. Check if the failure is related to this commit (5078eed) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1046/builds/1882

Failed tests:

  • test_fstring

Failed subtests:

  • test_fstring_nested_too_deeply - test.test_fstring.TestCase.test_fstring_nested_too_deeply

Summary of the results of the build (if available):

== Tests result: FAILURE ==

325 tests OK.

10 slowest tests:

  • test_math: 1 min 39 sec
  • test_tokenize: 1 min 20 sec
  • test_lib2to3: 53.0 sec
  • test_capi: 32.9 sec
  • test_unparse: 30.4 sec
  • test_hashlib: 24.5 sec
  • test_unicodedata: 16.5 sec
  • test_fstring: 15.3 sec
  • test_pickle: 13.4 sec
  • test_tarfile: 13.0 sec

1 test failed:
test_fstring

135 tests skipped:
test.test_asyncio.test_base_events
test.test_asyncio.test_buffered_proto
test.test_asyncio.test_context test.test_asyncio.test_events
test.test_asyncio.test_futures test.test_asyncio.test_futures2
test.test_asyncio.test_locks test.test_asyncio.test_pep492
test.test_asyncio.test_proactor_events
test.test_asyncio.test_protocols test.test_asyncio.test_queues
test.test_asyncio.test_runners
test.test_asyncio.test_selector_events
test.test_asyncio.test_sendfile test.test_asyncio.test_server
test.test_asyncio.test_sock_lowlevel test.test_asyncio.test_ssl
test.test_asyncio.test_sslproto test.test_asyncio.test_streams
test.test_asyncio.test_subprocess
test.test_asyncio.test_taskgroups test.test_asyncio.test_tasks
test.test_asyncio.test_threads test.test_asyncio.test_timeouts
test.test_asyncio.test_transports
test.test_asyncio.test_unix_events test.test_asyncio.test_waitfor
test.test_asyncio.test_windows_events
test.test_asyncio.test_windows_utils test__xxinterpchannels
test__xxsubinterpreters test_asyncgen test_bz2 test_clinic
test_cmd_line test_concurrent_futures test_contextlib_async
test_ctypes test_curses test_dbm_gnu test_dbm_ndbm test_devpoll
test_doctest test_docxmlrpc test_dtrace test_embed test_epoll
test_faulthandler test_fcntl test_file_eintr test_fork1
test_ftplib test_gdb test_grp test_gzip test_httplib
test_httpservers test_idle test_imaplib test_interpreters
test_ioctl test_kqueue test_launcher test_lzma test_mailbox
test_mmap test_msilib test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_nis test_openpty test_ossaudiodev
test_pdb test_peg_generator test_perf_profiler test_pipes
test_poll test_poplib test_pty test_pwd test_queue test_readline
test_regrtest test_repl test_resource test_select test_selectors
test_smtplib test_smtpnet test_socket test_socketserver test_spwd
test_sqlite3 test_ssl test_stable_abi_ctypes test_startfile
test_subprocess test_sys_settrace test_syslog test_tcl
test_telnetlib test_thread test_threadedtempfile test_threading
test_threading_local test_tix test_tkinter test_tools test_ttk
test_ttk_textonly test_turtle test_urllib test_urllib2
test_urllib2_localnet test_urllib2net test_urllib_response
test_urllibnet test_venv test_wait3 test_wait4 test_webbrowser
test_winconsoleio test_winreg test_winsound test_wmi test_wsgiref
test_xmlrpc test_xmlrpc_net test_xxlimited test_zipfile64
test_zipimport_support test_zlib test_zoneinfo
0:06:34 load avg: 3.31
0:06:34 load avg: 3.31 Re-running failed tests is not supported with --python host runner option.

Total duration: 6 min 34 sec

Click to see traceback logs
Traceback (most recent call last):
  File "/Lib/test/test_fstring.py", line 34, in assertAllRaise
    eval(str)
MemoryError

@brettcannon
Copy link
Member

@pablogsal ☝️ blowing out memory on WASI.

@pablogsal
Copy link
Member

@pablogsal point_up blowing out memory on WASI.

On it

carljm added a commit to carljm/cpython that referenced this pull request May 1, 2023
* main: (463 commits)
  pythongh-104057: Fix direct invocation of test_super (python#104064)
  pythongh-87092: Expose assembler to unit tests (python#103988)
  pythongh-97696: asyncio eager tasks factory (python#102853)
  pythongh-84436: Immortalize in _PyStructSequence_InitBuiltinWithFlags() (pythongh-104054)
  pythongh-104057: Fix direct invocation of test_module (pythonGH-104059)
  pythongh-100458: Clarify Enum.__format__() change of mixed-in types in the whatsnew/3.11.rst (pythonGH-100387)
  pythongh-104018: disallow "z" format specifier in %-format of byte strings (pythonGH-104033)
  pythongh-104016: Fixed off by 1 error in f string tokenizer (python#104047)
  pythonGH-103629: Update Unpack's repr in compliance with PEP 692 (python#104048)
  pythongh-102799: replace sys.exc_info by sys.exception in inspect and traceback modules (python#104032)
  Fix typo in "expected" word in few source files (python#104034)
  pythongh-103824: fix use-after-free error in Parser/tokenizer.c (python#103993)
  pythongh-104035: Do not ignore user-defined `__{get,set}state__` in slotted frozen dataclasses (python#104041)
  pythongh-104028: Reduce object creation while calling callback function from gc (pythongh-104030)
  pythongh-104036: Fix direct invocation of test_typing (python#104037)
  pythongh-102213: Optimize the performance of `__getattr__` (pythonGH-103761)
  pythongh-103895: Improve how invalid `Exception.__notes__` are displayed (python#103897)
  Adjust expression from `==` to `!=` in alignment with the meaning of the paragraph. (pythonGH-104021)
  pythongh-88496: Fix IDLE test hang on macOS (python#104025)
  Improve int test coverage (python#104024)
  ...
@arhadthedev
Copy link
Member

For the record: the WASM test got fixed gh-104071.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) skip news type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants