1**************************** 2 What's New In Python 3.9 3**************************** 4 5:Editor: Łukasz Langa 6 7.. Rules for maintenance: 8 9 * Anyone can add text to this document. Your text might get 10 rewritten to some degree. 11 12 * The maintainer will go through Misc/NEWS periodically and add 13 changes; it's therefore more important to add your changes to 14 Misc/NEWS than to this file. 15 16 * This is not a complete list of every single change; completeness 17 is the purpose of Misc/NEWS. Some changes will be too small 18 or esoteric to include. If such a change is added to the text, 19 it might get removed during final editing. 20 21 * If you want to draw your new text to the attention of the 22 maintainer, add 'XXX' to the beginning of the paragraph or 23 section. 24 25 * It's OK to just add a fragmentary note about a change. For 26 example: "XXX Describe the transmogrify() function added to the 27 socket module." The maintainer will research the change and 28 write the necessary text. 29 30 * You can comment out your additions if you like, but it's not 31 necessary (especially when a final release is some months away). 32 33 * Credit the author of a patch or bugfix. Just the name is 34 sufficient; the e-mail address isn't necessary. 35 36 * It's helpful to add the bug/patch number as a comment: 37 38 XXX Describe the transmogrify() function added to the socket 39 module. 40 (Contributed by P.Y. Developer in :issue:`12345`.) 41 42 This saves the maintainer the effort of going through the Mercurial log 43 when researching a change. 44 45This article explains the new features in Python 3.9, compared to 3.8. 46Python 3.9 was released on October 5, 2020. 47 48For full details, see the :ref:`changelog <changelog>`. 49 50.. seealso:: 51 52 :pep:`596` - Python 3.9 Release Schedule 53 54 55Summary -- Release highlights 56============================= 57 58.. This section singles out the most important changes in Python 3.9. 59 Brevity is key. 60 61New syntax features: 62 63* :pep:`584`, union operators added to ``dict``; 64* :pep:`585`, type hinting generics in standard collections; 65* :pep:`614`, relaxed grammar restrictions on decorators. 66 67New built-in features: 68 69* :pep:`616`, string methods to remove prefixes and suffixes. 70 71New features in the standard library: 72 73* :pep:`593`, flexible function and variable annotations; 74* :func:`os.pidfd_open` added that allows process management without races 75 and signals. 76 77Interpreter improvements: 78 79* :pep:`573`, fast access to module state from methods of C extension 80 types; 81* :pep:`617`, CPython now uses a new parser based on PEG; 82* a number of Python builtins (range, tuple, set, frozenset, list, dict) are 83 now sped up using :pep:`590` vectorcall; 84* garbage collection does not block on resurrected objects; 85* a number of Python modules (:mod:`_abc`, :mod:`audioop`, :mod:`_bz2`, 86 :mod:`_codecs`, :mod:`_contextvars`, :mod:`_crypt`, :mod:`_functools`, 87 :mod:`_json`, :mod:`_locale`, :mod:`math`, :mod:`operator`, :mod:`resource`, 88 :mod:`time`, :mod:`_weakref`) now use multiphase initialization as defined 89 by PEP 489; 90* a number of standard library modules (:mod:`audioop`, :mod:`ast`, :mod:`grp`, 91 :mod:`_hashlib`, :mod:`pwd`, :mod:`_posixsubprocess`, :mod:`random`, 92 :mod:`select`, :mod:`struct`, :mod:`termios`, :mod:`zlib`) are now using 93 the stable ABI defined by PEP 384. 94 95New library modules: 96 97* :pep:`615`, the IANA Time Zone Database is now present in the standard 98 library in the :mod:`zoneinfo` module; 99* an implementation of a topological sort of a graph is now provided in 100 the new :mod:`graphlib` module. 101 102Release process changes: 103 104* :pep:`602`, CPython adopts an annual release cycle. 105 106 107You should check for DeprecationWarning in your code 108==================================================== 109 110When Python 2.7 was still supported, a lot of functionality in Python 3 111was kept for backward compatibility with Python 2.7. With the end of Python 1122 support, these backward compatibility layers have been removed, or will 113be removed soon. Most of them emitted a :exc:`DeprecationWarning` warning for 114several years. For example, using ``collections.Mapping`` instead of 115``collections.abc.Mapping`` emits a :exc:`DeprecationWarning` since Python 1163.3, released in 2012. 117 118Test your application with the :option:`-W` ``default`` command-line option to see 119:exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning`, or even with 120:option:`-W` ``error`` to treat them as errors. :ref:`Warnings Filter 121<warning-filter>` can be used to ignore warnings from third-party code. 122 123Python 3.9 is the last version providing those Python 2 backward compatibility 124layers, to give more time to Python projects maintainers to organize the 125removal of the Python 2 support and add support for Python 3.9. 126 127Aliases to :ref:`Abstract Base Classes <collections-abstract-base-classes>` in 128the :mod:`collections` module, like ``collections.Mapping`` alias to 129:class:`collections.abc.Mapping`, are kept for one last release for backward 130compatibility. They will be removed from Python 3.10. 131 132More generally, try to run your tests in the :ref:`Python Development Mode 133<devmode>` which helps to prepare your code to make it compatible with the 134next Python version. 135 136Note: a number of pre-existing deprecations were removed in this version of 137Python as well. Consult the :ref:`removed-in-python-39` section. 138 139 140New Features 141============ 142 143Dictionary Merge & Update Operators 144----------------------------------- 145 146Merge (``|``) and update (``|=``) operators have been added to the built-in 147:class:`dict` class. Those complement the existing ``dict.update`` and 148``{**d1, **d2}`` methods of merging dictionaries. 149 150Example:: 151 152 >>> x = {"key1": "value1 from x", "key2": "value2 from x"} 153 >>> y = {"key2": "value2 from y", "key3": "value3 from y"} 154 >>> x | y 155 {'key1': 'value1 from x', 'key2': 'value2 from y', 'key3': 'value3 from y'} 156 >>> y | x 157 {'key2': 'value2 from x', 'key3': 'value3 from y', 'key1': 'value1 from x'} 158 159See :pep:`584` for a full description. 160(Contributed by Brandt Bucher in :issue:`36144`.) 161 162New String Methods to Remove Prefixes and Suffixes 163-------------------------------------------------- 164 165:meth:`str.removeprefix(prefix)<str.removeprefix>` and 166:meth:`str.removesuffix(suffix)<str.removesuffix>` have been added 167to easily remove an unneeded prefix or a suffix from a string. Corresponding 168``bytes``, ``bytearray``, and ``collections.UserString`` methods have also been 169added. See :pep:`616` for a full description. (Contributed by Dennis Sweeney in 170:issue:`39939`.) 171 172Type Hinting Generics in Standard Collections 173--------------------------------------------- 174 175In type annotations you can now use built-in collection types such as 176``list`` and ``dict`` as generic types instead of importing the 177corresponding capitalized types (e.g. ``List`` or ``Dict``) from 178``typing``. Some other types in the standard library are also now generic, 179for example ``queue.Queue``. 180 181Example: 182 183.. code-block:: python 184 185 def greet_all(names: list[str]) -> None: 186 for name in names: 187 print("Hello", name) 188 189See :pep:`585` for more details. (Contributed by Guido van Rossum, 190Ethan Smith, and Batuhan Taşkaya in :issue:`39481`.) 191 192New Parser 193---------- 194 195Python 3.9 uses a new parser, based on `PEG 196<https://en.wikipedia.org/wiki/Parsing_expression_grammar>`_ instead 197of `LL(1) <https://en.wikipedia.org/wiki/LL_parser>`_. The new 198parser's performance is roughly comparable to that of the old parser, 199but the PEG formalism is more flexible than LL(1) when it comes to 200designing new language features. We'll start using this flexibility 201in Python 3.10 and later. 202 203The :mod:`ast` module uses the new parser and produces the same AST as 204the old parser. 205 206In Python 3.10, the old parser will be deleted and so will all 207functionality that depends on it (primarily the :mod:`parser` module, 208which has long been deprecated). In Python 3.9 *only*, you can switch 209back to the LL(1) parser using a command line switch (``-X 210oldparser``) or an environment variable (``PYTHONOLDPARSER=1``). 211 212See :pep:`617` for more details. (Contributed by Guido van Rossum, 213Pablo Galindo and Lysandros Nikolaou in :issue:`40334`.) 214 215 216Other Language Changes 217====================== 218 219* :func:`__import__` now raises :exc:`ImportError` instead of 220 :exc:`ValueError`, which used to occur when a relative import went past 221 its top-level package. 222 (Contributed by Ngalim Siregar in :issue:`37444`.) 223 224* Python now gets the absolute path of the script filename specified on 225 the command line (ex: ``python3 script.py``): the ``__file__`` attribute of 226 the :mod:`__main__` module became an absolute path, rather than a relative 227 path. These paths now remain valid after the current directory is changed 228 by :func:`os.chdir`. As a side effect, the traceback also displays the 229 absolute path for :mod:`__main__` module frames in this case. 230 (Contributed by Victor Stinner in :issue:`20443`.) 231 232* In the :ref:`Python Development Mode <devmode>` and in :ref:`debug build <debug-build>`, the 233 *encoding* and *errors* arguments are now checked for string encoding and 234 decoding operations. Examples: :func:`open`, :meth:`str.encode` and 235 :meth:`bytes.decode`. 236 237 By default, for best performance, the *errors* argument is only checked at 238 the first encoding/decoding error and the *encoding* argument is sometimes 239 ignored for empty strings. 240 (Contributed by Victor Stinner in :issue:`37388`.) 241 242* ``"".replace("", s, n)`` now returns ``s`` instead of an empty string for 243 all non-zero ``n``. It is now consistent with ``"".replace("", s)``. 244 There are similar changes for :class:`bytes` and :class:`bytearray` objects. 245 (Contributed by Serhiy Storchaka in :issue:`28029`.) 246 247* Any valid expression can now be used as a :term:`decorator`. Previously, the 248 grammar was much more restrictive. See :pep:`614` for details. 249 (Contributed by Brandt Bucher in :issue:`39702`.) 250 251* Improved help for the :mod:`typing` module. Docstrings are now shown for 252 all special forms and special generic aliases (like ``Union`` and ``List``). 253 Using :func:`help` with generic alias like ``List[int]`` will show the help 254 for the correspondent concrete type (``list`` in this case). 255 (Contributed by Serhiy Storchaka in :issue:`40257`.) 256 257* Parallel running of :meth:`~agen.aclose` / :meth:`~agen.asend` / 258 :meth:`~agen.athrow` is now prohibited, and ``ag_running`` now reflects 259 the actual running status of the async generator. 260 (Contributed by Yury Selivanov in :issue:`30773`.) 261 262* Unexpected errors in calling the ``__iter__`` method are no longer masked by 263 ``TypeError`` in the :keyword:`in` operator and functions 264 :func:`~operator.contains`, :func:`~operator.indexOf` and 265 :func:`~operator.countOf` of the :mod:`operator` module. 266 (Contributed by Serhiy Storchaka in :issue:`40824`.) 267 268* Unparenthesized lambda expressions can no longer be the expression part in an 269 ``if`` clause in comprehensions and generator expressions. See :issue:`41848` 270 and :issue:`43755` for details. 271 272 273New Modules 274=========== 275 276zoneinfo 277-------- 278 279The :mod:`zoneinfo` module brings support for the IANA time zone database to 280the standard library. It adds :class:`zoneinfo.ZoneInfo`, a concrete 281:class:`datetime.tzinfo` implementation backed by the system's time zone data. 282 283Example:: 284 285 >>> from zoneinfo import ZoneInfo 286 >>> from datetime import datetime, timedelta 287 288 >>> # Daylight saving time 289 >>> dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles")) 290 >>> print(dt) 291 2020-10-31 12:00:00-07:00 292 >>> dt.tzname() 293 'PDT' 294 295 >>> # Standard time 296 >>> dt += timedelta(days=7) 297 >>> print(dt) 298 2020-11-07 12:00:00-08:00 299 >>> print(dt.tzname()) 300 PST 301 302 303As a fall-back source of data for platforms that don't ship the IANA database, 304the |tzdata|_ module was released as a first-party package -- distributed via 305PyPI and maintained by the CPython core team. 306 307.. |tzdata| replace:: ``tzdata`` 308.. _tzdata: https://pypi.org/project/tzdata/ 309 310.. seealso:: 311 312 :pep:`615` -- Support for the IANA Time Zone Database in the Standard Library 313 PEP written and implemented by Paul Ganssle 314 315 316graphlib 317--------- 318 319A new module, :mod:`graphlib`, was added that contains the 320:class:`graphlib.TopologicalSorter` class to offer functionality to perform 321topological sorting of graphs. (Contributed by Pablo Galindo, Tim Peters and 322Larry Hastings in :issue:`17005`.) 323 324 325Improved Modules 326================ 327 328ast 329--- 330 331Added the *indent* option to :func:`~ast.dump` which allows it to produce a 332multiline indented output. 333(Contributed by Serhiy Storchaka in :issue:`37995`.) 334 335Added :func:`ast.unparse` as a function in the :mod:`ast` module that can 336be used to unparse an :class:`ast.AST` object and produce a string with code 337that would produce an equivalent :class:`ast.AST` object when parsed. 338(Contributed by Pablo Galindo and Batuhan Taskaya in :issue:`38870`.) 339 340Added docstrings to AST nodes that contains the ASDL signature used to 341construct that node. (Contributed by Batuhan Taskaya in :issue:`39638`.) 342 343asyncio 344------- 345 346Due to significant security concerns, the *reuse_address* parameter of 347:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is 348because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more 349details, see the documentation for ``loop.create_datagram_endpoint()``. 350(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in 351:issue:`37228`.) 352 353Added a new :term:`coroutine` :meth:`~asyncio.loop.shutdown_default_executor` 354that schedules a shutdown for the default executor that waits on the 355:class:`~concurrent.futures.ThreadPoolExecutor` to finish closing. Also, 356:func:`asyncio.run` has been updated to use the new :term:`coroutine`. 357(Contributed by Kyle Stanley in :issue:`34037`.) 358 359Added :class:`asyncio.PidfdChildWatcher`, a Linux-specific child watcher 360implementation that polls process file descriptors. (:issue:`38692`) 361 362Added a new :term:`coroutine` :func:`asyncio.to_thread`. It is mainly used for 363running IO-bound functions in a separate thread to avoid blocking the event 364loop, and essentially works as a high-level version of 365:meth:`~asyncio.loop.run_in_executor` that can directly take keyword arguments. 366(Contributed by Kyle Stanley and Yury Selivanov in :issue:`32309`.) 367 368When cancelling the task due to a timeout, :meth:`asyncio.wait_for` will now 369wait until the cancellation is complete also in the case when *timeout* is 370<= 0, like it does with positive timeouts. 371(Contributed by Elvis Pranskevichus in :issue:`32751`.) 372 373:mod:`asyncio` now raises :exc:`TyperError` when calling incompatible 374methods with an :class:`ssl.SSLSocket` socket. 375(Contributed by Ido Michael in :issue:`37404`.) 376 377compileall 378---------- 379 380Added new possibility to use hardlinks for duplicated ``.pyc`` files: *hardlink_dupes* parameter and --hardlink-dupes command line option. 381(Contributed by Lumír 'Frenzy' Balhar in :issue:`40495`.) 382 383Added new options for path manipulation in resulting ``.pyc`` files: *stripdir*, *prependdir*, *limit_sl_dest* parameters and -s, -p, -e command line options. 384Added the possibility to specify the option for an optimization level multiple times. 385(Contributed by Lumír 'Frenzy' Balhar in :issue:`38112`.) 386 387concurrent.futures 388------------------ 389 390Added a new *cancel_futures* parameter to 391:meth:`concurrent.futures.Executor.shutdown` that cancels all pending futures 392which have not started running, instead of waiting for them to complete before 393shutting down the executor. 394(Contributed by Kyle Stanley in :issue:`39349`.) 395 396Removed daemon threads from :class:`~concurrent.futures.ThreadPoolExecutor` 397and :class:`~concurrent.futures.ProcessPoolExecutor`. This improves 398compatibility with subinterpreters and predictability in their shutdown 399processes. (Contributed by Kyle Stanley in :issue:`39812`.) 400 401Workers in :class:`~concurrent.futures.ProcessPoolExecutor` are now spawned on 402demand, only when there are no available idle workers to reuse. This optimizes 403startup overhead and reduces the amount of lost CPU time to idle workers. 404(Contributed by Kyle Stanley in :issue:`39207`.) 405 406curses 407------ 408 409Added :func:`curses.get_escdelay`, :func:`curses.set_escdelay`, 410:func:`curses.get_tabsize`, and :func:`curses.set_tabsize` functions. 411(Contributed by Anthony Sottile in :issue:`38312`.) 412 413datetime 414-------- 415The :meth:`~datetime.date.isocalendar()` of :class:`datetime.date` 416and :meth:`~datetime.datetime.isocalendar()` of :class:`datetime.datetime` 417methods now returns a :func:`~collections.namedtuple` instead of a :class:`tuple`. 418(Contributed by Dong-hee Na in :issue:`24416`.) 419 420distutils 421--------- 422 423The :command:`upload` command now creates SHA2-256 and Blake2b-256 hash 424digests. It skips MD5 on platforms that block MD5 digest. 425(Contributed by Christian Heimes in :issue:`40698`.) 426 427fcntl 428----- 429 430Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK` 431and :data:`~fcntl.F_OFD_SETLKW`. 432(Contributed by Dong-hee Na in :issue:`38602`.) 433 434ftplib 435------- 436 437:class:`~ftplib.FTP` and :class:`~ftplib.FTP_TLS` now raise a :class:`ValueError` 438if the given timeout for their constructor is zero to prevent the creation of 439a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) 440 441gc 442-- 443 444When the garbage collector makes a collection in which some objects resurrect 445(they are reachable from outside the isolated cycles after the finalizers have 446been executed), do not block the collection of all objects that are still 447unreachable. (Contributed by Pablo Galindo and Tim Peters in :issue:`38379`.) 448 449Added a new function :func:`gc.is_finalized` to check if an object has been 450finalized by the garbage collector. (Contributed by Pablo Galindo in 451:issue:`39322`.) 452 453hashlib 454------- 455 456The :mod:`hashlib` module can now use SHA3 hashes and SHAKE XOF from OpenSSL 457when available. 458(Contributed by Christian Heimes in :issue:`37630`.) 459 460Builtin hash modules can now be disabled with 461``./configure --without-builtin-hashlib-hashes`` or selectively enabled with 462e.g. ``./configure --with-builtin-hashlib-hashes=sha3,blake2`` to force use 463of OpenSSL based implementation. 464(Contributed by Christian Heimes in :issue:`40479`) 465 466 467http 468---- 469 470HTTP status codes ``103 EARLY_HINTS``, ``418 IM_A_TEAPOT`` and ``425 TOO_EARLY`` are added to 471:class:`http.HTTPStatus`. (Contributed by Dong-hee Na in :issue:`39509` and Ross Rhodes in :issue:`39507`.) 472 473IDLE and idlelib 474---------------- 475 476Added option to toggle cursor blink off. (Contributed by Zackery Spytz 477in :issue:`4603`.) 478 479Escape key now closes IDLE completion windows. (Contributed by Johnny 480Najera in :issue:`38944`.) 481 482Added keywords to module name completion list. (Contributed by Terry J. 483Reedy in :issue:`37765`.) 484 485New in 3.9 maintenance releases 486 487Make IDLE invoke :func:`sys.excepthook` (when started without '-n'). 488User hooks were previously ignored. (Contributed by Ken Hilton in 489:issue:`43008`.) 490 491The changes above have been backported to 3.8 maintenance releases. 492 493Rearrange the settings dialog. Split the General tab into Windows 494and Shell/Ed tabs. Move help sources, which extend the Help menu, to the 495Extensions tab. Make space for new options and shorten the dialog. The 496latter makes the dialog better fit small screens. (Contributed by Terry Jan 497Reedy in :issue:`40468`.) Move the indent space setting from the Font tab to 498the new Windows tab. (Contributed by Mark Roseman and Terry Jan Reedy in 499:issue:`33962`.) 500 501Apply syntax highlighting to ``.pyi`` files. (Contributed by Alex 502Waygood and Terry Jan Reedy in :issue:`45447`.) 503 504imaplib 505------- 506 507:class:`~imaplib.IMAP4` and :class:`~imaplib.IMAP4_SSL` now have 508an optional *timeout* parameter for their constructors. 509Also, the :meth:`~imaplib.IMAP4.open` method now has an optional *timeout* parameter 510with this change. The overridden methods of :class:`~imaplib.IMAP4_SSL` and 511:class:`~imaplib.IMAP4_stream` were applied to this change. 512(Contributed by Dong-hee Na in :issue:`38615`.) 513 514:meth:`imaplib.IMAP4.unselect` is added. 515:meth:`imaplib.IMAP4.unselect` frees server's resources associated with the 516selected mailbox and returns the server to the authenticated 517state. This command performs the same actions as :meth:`imaplib.IMAP4.close`, except 518that no messages are permanently removed from the currently 519selected mailbox. (Contributed by Dong-hee Na in :issue:`40375`.) 520 521importlib 522--------- 523 524To improve consistency with import statements, :func:`importlib.util.resolve_name` 525now raises :exc:`ImportError` instead of :exc:`ValueError` for invalid relative 526import attempts. 527(Contributed by Ngalim Siregar in :issue:`37444`.) 528 529Import loaders which publish immutable module objects can now publish 530immutable packages in addition to individual modules. 531(Contributed by Dino Viehland in :issue:`39336`.) 532 533Added :func:`importlib.resources.files` function with support for 534subdirectories in package data, matching backport in ``importlib_resources`` 535version 1.5. 536(Contributed by Jason R. Coombs in :issue:`39791`.) 537 538Refreshed ``importlib.metadata`` from ``importlib_metadata`` version 1.6.1. 539 540inspect 541------- 542 543:attr:`inspect.BoundArguments.arguments` is changed from ``OrderedDict`` to regular 544dict. (Contributed by Inada Naoki in :issue:`36350` and :issue:`39775`.) 545 546ipaddress 547--------- 548 549:mod:`ipaddress` now supports IPv6 Scoped Addresses (IPv6 address with suffix ``%<scope_id>``). 550 551Scoped IPv6 addresses can be parsed using :class:`ipaddress.IPv6Address`. 552If present, scope zone ID is available through the :attr:`~ipaddress.IPv6Address.scope_id` attribute. 553(Contributed by Oleksandr Pavliuk in :issue:`34788`.) 554 555Starting with Python 3.9.5 the :mod:`ipaddress` module no longer 556accepts any leading zeros in IPv4 address strings. 557(Contributed by Christian Heimes in :issue:`36384`). 558 559math 560---- 561 562Expanded the :func:`math.gcd` function to handle multiple arguments. 563Formerly, it only supported two arguments. 564(Contributed by Serhiy Storchaka in :issue:`39648`.) 565 566Added :func:`math.lcm`: return the least common multiple of specified arguments. 567(Contributed by Mark Dickinson, Ananthakrishnan and Serhiy Storchaka in 568:issue:`39479` and :issue:`39648`.) 569 570Added :func:`math.nextafter`: return the next floating-point value after *x* 571towards *y*. 572(Contributed by Victor Stinner in :issue:`39288`.) 573 574Added :func:`math.ulp`: return the value of the least significant bit 575of a float. 576(Contributed by Victor Stinner in :issue:`39310`.) 577 578multiprocessing 579--------------- 580 581The :class:`multiprocessing.SimpleQueue` class has a new 582:meth:`~multiprocessing.SimpleQueue.close` method to explicitly close the 583queue. 584(Contributed by Victor Stinner in :issue:`30966`.) 585 586nntplib 587------- 588 589:class:`~nntplib.NNTP` and :class:`~nntplib.NNTP_SSL` now raise a :class:`ValueError` 590if the given timeout for their constructor is zero to prevent the creation of 591a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) 592 593os 594-- 595 596Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`. 597(Contributed by Dong-hee Na in :issue:`38493`.) 598 599Exposed the Linux-specific :func:`os.pidfd_open` (:issue:`38692`) and 600:data:`os.P_PIDFD` (:issue:`38713`) for process management with file 601descriptors. 602 603The :func:`os.unsetenv` function is now also available on Windows. 604(Contributed by Victor Stinner in :issue:`39413`.) 605 606The :func:`os.putenv` and :func:`os.unsetenv` functions are now always 607available. 608(Contributed by Victor Stinner in :issue:`39395`.) 609 610Added :func:`os.waitstatus_to_exitcode` function: 611convert a wait status to an exit code. 612(Contributed by Victor Stinner in :issue:`40094`.) 613 614pathlib 615------- 616 617Added :meth:`pathlib.Path.readlink()` which acts similarly to 618:func:`os.readlink`. 619(Contributed by Girts Folkmanis in :issue:`30618`) 620 621pdb 622--- 623 624On Windows now :class:`~pdb.Pdb` supports ``~/.pdbrc``. 625(Contributed by Tim Hopper and Dan Lidral-Porter in :issue:`20523`.) 626 627poplib 628------ 629 630:class:`~poplib.POP3` and :class:`~poplib.POP3_SSL` now raise a :class:`ValueError` 631if the given timeout for their constructor is zero to prevent the creation of 632a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) 633 634pprint 635------ 636 637:mod:`pprint` can now pretty-print :class:`types.SimpleNamespace`. 638(Contributed by Carl Bordum Hansen in :issue:`37376`.) 639 640pydoc 641----- 642 643The documentation string is now shown not only for class, function, 644method etc, but for any object that has its own ``__doc__`` attribute. 645(Contributed by Serhiy Storchaka in :issue:`40257`.) 646 647random 648------ 649 650Added a new :attr:`random.Random.randbytes` method: generate random bytes. 651(Contributed by Victor Stinner in :issue:`40286`.) 652 653signal 654------ 655 656Exposed the Linux-specific :func:`signal.pidfd_send_signal` for sending to 657signals to a process using a file descriptor instead of a pid. (:issue:`38712`) 658 659smtplib 660------- 661 662:class:`~smtplib.SMTP` and :class:`~smtplib.SMTP_SSL` now raise a :class:`ValueError` 663if the given timeout for their constructor is zero to prevent the creation of 664a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) 665 666:class:`~smtplib.LMTP` constructor now has an optional *timeout* parameter. 667(Contributed by Dong-hee Na in :issue:`39329`.) 668 669socket 670------ 671 672The :mod:`socket` module now exports the :data:`~socket.CAN_RAW_JOIN_FILTERS` 673constant on Linux 4.1 and greater. 674(Contributed by Stefan Tatschner and Zackery Spytz in :issue:`25780`.) 675 676The socket module now supports the :data:`~socket.CAN_J1939` protocol on 677platforms that support it. (Contributed by Karl Ding in :issue:`40291`.) 678 679The socket module now has the :func:`socket.send_fds` and 680:func:`socket.recv_fds` functions. (Contributed by Joannah Nanjekye, Shinya 681Okano and Victor Stinner in :issue:`28724`.) 682 683 684time 685---- 686 687On AIX, :func:`~time.thread_time` is now implemented with ``thread_cputime()`` 688which has nanosecond resolution, rather than 689``clock_gettime(CLOCK_THREAD_CPUTIME_ID)`` which has a resolution of 10 milliseconds. 690(Contributed by Batuhan Taskaya in :issue:`40192`) 691 692sys 693--- 694 695Added a new :attr:`sys.platlibdir` attribute: name of the platform-specific 696library directory. It is used to build the path of standard library and the 697paths of installed extension modules. It is equal to ``"lib"`` on most 698platforms. On Fedora and SuSE, it is equal to ``"lib64"`` on 64-bit platforms. 699(Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner in :issue:`1294959`.) 700 701Previously, :attr:`sys.stderr` was block-buffered when non-interactive. Now 702``stderr`` defaults to always being line-buffered. 703(Contributed by Jendrik Seipp in :issue:`13601`.) 704 705tracemalloc 706----------- 707 708Added :func:`tracemalloc.reset_peak` to set the peak size of traced memory 709blocks to the current size, to measure the peak of specific pieces of code. 710(Contributed by Huon Wilson in :issue:`40630`.) 711 712typing 713------ 714 715:pep:`593` introduced an :data:`typing.Annotated` type to decorate existing 716types with context-specific metadata and new ``include_extras`` parameter to 717:func:`typing.get_type_hints` to access the metadata at runtime. (Contributed 718by Till Varoquaux and Konstantin Kashin.) 719 720unicodedata 721----------- 722 723The Unicode database has been updated to version 13.0.0. (:issue:`39926`). 724 725venv 726---- 727 728The activation scripts provided by :mod:`venv` now all specify their prompt 729customization consistently by always using the value specified by 730``__VENV_PROMPT__``. Previously some scripts unconditionally used 731``__VENV_PROMPT__``, others only if it happened to be set (which was the default 732case), and one used ``__VENV_NAME__`` instead. 733(Contributed by Brett Cannon in :issue:`37663`.) 734 735xml 736--- 737 738White space characters within attributes are now preserved when serializing 739:mod:`xml.etree.ElementTree` to XML file. EOLNs are no longer normalized 740to "\n". This is the result of discussion about how to interpret 741section 2.11 of XML spec. 742(Contributed by Mefistotelis in :issue:`39011`.) 743 744 745Optimizations 746============= 747 748* Optimized the idiom for assignment a temporary variable in comprehensions. 749 Now ``for y in [expr]`` in comprehensions is as fast as a simple assignment 750 ``y = expr``. For example: 751 752 sums = [s for s in [0] for x in data for s in [s + x]] 753 754 Unlike the ``:=`` operator this idiom does not leak a variable to the 755 outer scope. 756 757 (Contributed by Serhiy Storchaka in :issue:`32856`.) 758 759* Optimized signal handling in multithreaded applications. If a thread different 760 than the main thread gets a signal, the bytecode evaluation loop is no longer 761 interrupted at each bytecode instruction to check for pending signals which 762 cannot be handled. Only the main thread of the main interpreter can handle 763 signals. 764 765 Previously, the bytecode evaluation loop was interrupted at each instruction 766 until the main thread handles signals. 767 (Contributed by Victor Stinner in :issue:`40010`.) 768 769* Optimized the :mod:`subprocess` module on FreeBSD using ``closefrom()``. 770 (Contributed by Ed Maste, Conrad Meyer, Kyle Evans, Kubilay Kocak and Victor 771 Stinner in :issue:`38061`.) 772 773* :c:func:`PyLong_FromDouble` is now up to 1.87x faster for values that 774 fit into :c:expr:`long`. 775 (Contributed by Sergey Fedoseev in :issue:`37986`.) 776 777* A number of Python builtins (:class:`range`, :class:`tuple`, :class:`set`, 778 :class:`frozenset`, :class:`list`, :class:`dict`) are now sped up by using 779 :pep:`590` vectorcall protocol. 780 (Contributed by Dong-hee Na, Mark Shannon, Jeroen Demeyer and Petr Viktorin in :issue:`37207`.) 781 782* Optimized :func:`~set.difference_update` for the case when the other set 783 is much larger than the base set. 784 (Suggested by Evgeny Kapun with code contributed by Michele Orrù in :issue:`8425`.) 785 786* Python's small object allocator (``obmalloc.c``) now allows (no more than) 787 one empty arena to remain available for immediate reuse, without returning 788 it to the OS. This prevents thrashing in simple loops where an arena could 789 be created and destroyed anew on each iteration. 790 (Contributed by Tim Peters in :issue:`37257`.) 791 792* :term:`floor division` of float operation now has a better performance. Also 793 the message of :exc:`ZeroDivisionError` for this operation is updated. 794 (Contributed by Dong-hee Na in :issue:`39434`.) 795 796* Decoding short ASCII strings with UTF-8 and ascii codecs is now about 797 15% faster. (Contributed by Inada Naoki in :issue:`37348`.) 798 799Here's a summary of performance improvements from Python 3.4 through Python 3.9: 800 801.. code-block:: none 802 803 Python version 3.4 3.5 3.6 3.7 3.8 3.9 804 -------------- --- --- --- --- --- --- 805 806 Variable and attribute read access: 807 read_local 7.1 7.1 5.4 5.1 3.9 3.9 808 read_nonlocal 7.1 8.1 5.8 5.4 4.4 4.5 809 read_global 15.5 19.0 14.3 13.6 7.6 7.8 810 read_builtin 21.1 21.6 18.5 19.0 7.5 7.8 811 read_classvar_from_class 25.6 26.5 20.7 19.5 18.4 17.9 812 read_classvar_from_instance 22.8 23.5 18.8 17.1 16.4 16.9 813 read_instancevar 32.4 33.1 28.0 26.3 25.4 25.3 814 read_instancevar_slots 27.8 31.3 20.8 20.8 20.2 20.5 815 read_namedtuple 73.8 57.5 45.0 46.8 18.4 18.7 816 read_boundmethod 37.6 37.9 29.6 26.9 27.7 41.1 817 818 Variable and attribute write access: 819 write_local 8.7 9.3 5.5 5.3 4.3 4.3 820 write_nonlocal 10.5 11.1 5.6 5.5 4.7 4.8 821 write_global 19.7 21.2 18.0 18.0 15.8 16.7 822 write_classvar 92.9 96.0 104.6 102.1 39.2 39.8 823 write_instancevar 44.6 45.8 40.0 38.9 35.5 37.4 824 write_instancevar_slots 35.6 36.1 27.3 26.6 25.7 25.8 825 826 Data structure read access: 827 read_list 24.2 24.5 20.8 20.8 19.0 19.5 828 read_deque 24.7 25.5 20.2 20.6 19.8 20.2 829 read_dict 24.3 25.7 22.3 23.0 21.0 22.4 830 read_strdict 22.6 24.3 19.5 21.2 18.9 21.5 831 832 Data structure write access: 833 write_list 27.1 28.5 22.5 21.6 20.0 20.0 834 write_deque 28.7 30.1 22.7 21.8 23.5 21.7 835 write_dict 31.4 33.3 29.3 29.2 24.7 25.4 836 write_strdict 28.4 29.9 27.5 25.2 23.1 24.5 837 838 Stack (or queue) operations: 839 list_append_pop 93.4 112.7 75.4 74.2 50.8 50.6 840 deque_append_pop 43.5 57.0 49.4 49.2 42.5 44.2 841 deque_append_popleft 43.7 57.3 49.7 49.7 42.8 46.4 842 843 Timing loop: 844 loop_overhead 0.5 0.6 0.4 0.3 0.3 0.3 845 846These results were generated from the variable access benchmark script at: 847``Tools/scripts/var_access_benchmark.py``. The benchmark script displays timings 848in nanoseconds. The benchmarks were measured on an 849`Intel® Core™ i7-4960HQ processor 850<https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.html>`_ 851running the macOS 64-bit builds found at 852`python.org <https://www.python.org/downloads/macos/>`_. 853 854 855Deprecated 856========== 857 858* The distutils ``bdist_msi`` command is now deprecated, use 859 ``bdist_wheel`` (wheel packages) instead. 860 (Contributed by Hugo van Kemenade in :issue:`39586`.) 861 862* Currently :func:`math.factorial` accepts :class:`float` instances with 863 non-negative integer values (like ``5.0``). It raises a :exc:`ValueError` 864 for non-integral and negative floats. It is now deprecated. In future 865 Python versions it will raise a :exc:`TypeError` for all floats. 866 (Contributed by Serhiy Storchaka in :issue:`37315`.) 867 868* The :mod:`parser` and :mod:`symbol` modules are deprecated and will be 869 removed in future versions of Python. For the majority of use cases, 870 users can leverage the Abstract Syntax Tree (AST) generation and compilation 871 stage, using the :mod:`ast` module. 872 873* The Public C API functions :c:func:`PyParser_SimpleParseStringFlags`, 874 :c:func:`PyParser_SimpleParseStringFlagsFilename`, 875 :c:func:`PyParser_SimpleParseFileFlags` and :c:func:`PyNode_Compile` 876 are deprecated and will be removed in Python 3.10 together with the old parser. 877 878* Using :data:`NotImplemented` in a boolean context has been deprecated, 879 as it is almost exclusively the result of incorrect rich comparator 880 implementations. It will be made a :exc:`TypeError` in a future version 881 of Python. 882 (Contributed by Josh Rosenberg in :issue:`35712`.) 883 884* The :mod:`random` module currently accepts any hashable type as a 885 possible seed value. Unfortunately, some of those types are not 886 guaranteed to have a deterministic hash value. After Python 3.9, 887 the module will restrict its seeds to :const:`None`, :class:`int`, 888 :class:`float`, :class:`str`, :class:`bytes`, and :class:`bytearray`. 889 890* Opening the :class:`~gzip.GzipFile` file for writing without specifying 891 the *mode* argument is deprecated. In future Python versions it will always 892 be opened for reading by default. Specify the *mode* argument for opening 893 it for writing and silencing a warning. 894 (Contributed by Serhiy Storchaka in :issue:`28286`.) 895 896* Deprecated the ``split()`` method of :class:`_tkinter.TkappType` in 897 favour of the ``splitlist()`` method which has more consistent and 898 predicable behavior. 899 (Contributed by Serhiy Storchaka in :issue:`38371`.) 900 901* The explicit passing of coroutine objects to :func:`asyncio.wait` has been 902 deprecated and will be removed in version 3.11. 903 (Contributed by Yury Selivanov and Kyle Stanley in :issue:`34790`.) 904 905* binhex4 and hexbin4 standards are now deprecated. The :mod:`binhex` module 906 and the following :mod:`binascii` functions are now deprecated: 907 908 * :func:`~binascii.b2a_hqx`, :func:`~binascii.a2b_hqx` 909 * :func:`~binascii.rlecode_hqx`, :func:`~binascii.rledecode_hqx` 910 911 (Contributed by Victor Stinner in :issue:`39353`.) 912 913* :mod:`ast` classes ``slice``, ``Index`` and ``ExtSlice`` are considered deprecated 914 and will be removed in future Python versions. ``value`` itself should be 915 used instead of ``Index(value)``. ``Tuple(slices, Load())`` should be 916 used instead of ``ExtSlice(slices)``. 917 (Contributed by Serhiy Storchaka in :issue:`34822`.) 918 919* :mod:`ast` classes ``Suite``, ``Param``, ``AugLoad`` and ``AugStore`` 920 are considered deprecated and will be removed in future Python versions. 921 They were not generated by the parser and not accepted by the code 922 generator in Python 3. 923 (Contributed by Batuhan Taskaya in :issue:`39639` and :issue:`39969` 924 and Serhiy Storchaka in :issue:`39988`.) 925 926* The :c:func:`PyEval_InitThreads` and :c:func:`PyEval_ThreadsInitialized` 927 functions are now deprecated and will be removed in Python 3.11. Calling 928 :c:func:`PyEval_InitThreads` now does nothing. The :term:`GIL` is initialized 929 by :c:func:`Py_Initialize()` since Python 3.7. 930 (Contributed by Victor Stinner in :issue:`39877`.) 931 932* Passing ``None`` as the first argument to the :func:`shlex.split` function 933 has been deprecated. (Contributed by Zackery Spytz in :issue:`33262`.) 934 935* :func:`smtpd.MailmanProxy` is now deprecated as it is unusable without 936 an external module, ``mailman``. (Contributed by Samuel Colvin in :issue:`35800`.) 937 938* The :mod:`lib2to3` module now emits a :exc:`PendingDeprecationWarning`. 939 Python 3.9 switched to a PEG parser (see :pep:`617`), and Python 3.10 may 940 include new language syntax that is not parsable by lib2to3's LL(1) parser. 941 The ``lib2to3`` module may be removed from the standard library in a future 942 Python version. Consider third-party alternatives such as `LibCST`_ or 943 `parso`_. 944 (Contributed by Carl Meyer in :issue:`40360`.) 945 946* The *random* parameter of :func:`random.shuffle` has been deprecated. 947 (Contributed by Raymond Hettinger in :issue:`40465`) 948 949.. _LibCST: https://libcst.readthedocs.io/ 950.. _parso: https://parso.readthedocs.io/ 951 952.. _removed-in-python-39: 953 954Removed 955======= 956 957* The erroneous version at :data:`unittest.mock.__version__` has been removed. 958 959* :class:`nntplib.NNTP`: ``xpath()`` and ``xgtitle()`` methods have been removed. 960 These methods are deprecated since Python 3.3. Generally, these extensions 961 are not supported or not enabled by NNTP server administrators. 962 For ``xgtitle()``, please use :meth:`nntplib.NNTP.descriptions` or 963 :meth:`nntplib.NNTP.description` instead. 964 (Contributed by Dong-hee Na in :issue:`39366`.) 965 966* :class:`array.array`: ``tostring()`` and ``fromstring()`` methods have been 967 removed. They were aliases to ``tobytes()`` and ``frombytes()``, deprecated 968 since Python 3.2. 969 (Contributed by Victor Stinner in :issue:`38916`.) 970 971* The undocumented ``sys.callstats()`` function has been removed. Since Python 972 3.7, it was deprecated and always returned :const:`None`. It required a special 973 build option ``CALL_PROFILE`` which was already removed in Python 3.7. 974 (Contributed by Victor Stinner in :issue:`37414`.) 975 976* The ``sys.getcheckinterval()`` and ``sys.setcheckinterval()`` functions have 977 been removed. They were deprecated since Python 3.2. Use 978 :func:`sys.getswitchinterval` and :func:`sys.setswitchinterval` instead. 979 (Contributed by Victor Stinner in :issue:`37392`.) 980 981* The C function ``PyImport_Cleanup()`` has been removed. It was documented as: 982 "Empty the module table. For internal use only." 983 (Contributed by Victor Stinner in :issue:`36710`.) 984 985* ``_dummy_thread`` and ``dummy_threading`` modules have been removed. These 986 modules were deprecated since Python 3.7 which requires threading support. 987 (Contributed by Victor Stinner in :issue:`37312`.) 988 989* ``aifc.openfp()`` alias to ``aifc.open()``, ``sunau.openfp()`` alias to 990 ``sunau.open()``, and ``wave.openfp()`` alias to :func:`wave.open()` have been 991 removed. They were deprecated since Python 3.7. 992 (Contributed by Victor Stinner in :issue:`37320`.) 993 994* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread` 995 has been removed. It was deprecated since Python 3.8. 996 Use :meth:`~threading.Thread.is_alive()` instead. 997 (Contributed by Dong-hee Na in :issue:`37804`.) 998 999* Methods ``getchildren()`` and ``getiterator()`` of classes 1000 :class:`~xml.etree.ElementTree.ElementTree` and 1001 :class:`~xml.etree.ElementTree.Element` in the :mod:`~xml.etree.ElementTree` 1002 module have been removed. They were deprecated in Python 3.2. 1003 Use ``iter(x)`` or ``list(x)`` instead of ``x.getchildren()`` and 1004 ``x.iter()`` or ``list(x.iter())`` instead of ``x.getiterator()``. 1005 (Contributed by Serhiy Storchaka in :issue:`36543`.) 1006 1007* The old :mod:`plistlib` API has been removed, it was deprecated since Python 1008 3.4. Use the :func:`~plistlib.load`, :func:`~plistlib.loads`, :func:`~plistlib.dump`, and 1009 :func:`~plistlib.dumps` functions. Additionally, the *use_builtin_types* parameter was 1010 removed, standard :class:`bytes` objects are always used instead. 1011 (Contributed by Jon Janzen in :issue:`36409`.) 1012 1013* The C function ``PyGen_NeedsFinalizing`` has been removed. It was not 1014 documented, tested, or used anywhere within CPython after the implementation 1015 of :pep:`442`. Patch by Joannah Nanjekye. 1016 (Contributed by Joannah Nanjekye in :issue:`15088`) 1017 1018* ``base64.encodestring()`` and ``base64.decodestring()``, aliases deprecated 1019 since Python 3.1, have been removed: use :func:`base64.encodebytes` and 1020 :func:`base64.decodebytes` instead. 1021 (Contributed by Victor Stinner in :issue:`39351`.) 1022 1023* ``fractions.gcd()`` function has been removed, it was deprecated since Python 1024 3.5 (:issue:`22486`): use :func:`math.gcd` instead. 1025 (Contributed by Victor Stinner in :issue:`39350`.) 1026 1027* The *buffering* parameter of :class:`bz2.BZ2File` has been removed. Since 1028 Python 3.0, it was ignored and using it emitted a :exc:`DeprecationWarning`. 1029 Pass an open file object to control how the file is opened. 1030 (Contributed by Victor Stinner in :issue:`39357`.) 1031 1032* The *encoding* parameter of :func:`json.loads` has been removed. 1033 As of Python 3.1, it was deprecated and ignored; using it has emitted a 1034 :exc:`DeprecationWarning` since Python 3.8. 1035 (Contributed by Inada Naoki in :issue:`39377`) 1036 1037* ``with (await asyncio.lock):`` and ``with (yield from asyncio.lock):`` statements are 1038 not longer supported, use ``async with lock`` instead. The same is correct for 1039 ``asyncio.Condition`` and ``asyncio.Semaphore``. 1040 (Contributed by Andrew Svetlov in :issue:`34793`.) 1041 1042* The :func:`sys.getcounts` function, the ``-X showalloccount`` command line 1043 option and the ``show_alloc_count`` field of the C structure 1044 :c:type:`PyConfig` have been removed. They required a special Python build by 1045 defining ``COUNT_ALLOCS`` macro. 1046 (Contributed by Victor Stinner in :issue:`39489`.) 1047 1048* The ``_field_types`` attribute of the :class:`typing.NamedTuple` class 1049 has been removed. It was deprecated since Python 3.8. Use 1050 the ``__annotations__`` attribute instead. 1051 (Contributed by Serhiy Storchaka in :issue:`40182`.) 1052 1053* The :meth:`symtable.SymbolTable.has_exec` method has been removed. It was 1054 deprecated since 2006, and only returning ``False`` when it's called. 1055 (Contributed by Batuhan Taskaya in :issue:`40208`) 1056 1057* The :meth:`asyncio.Task.current_task` and :meth:`asyncio.Task.all_tasks` 1058 have been removed. They were deprecated since Python 3.7 and you can use 1059 :func:`asyncio.current_task` and :func:`asyncio.all_tasks` instead. 1060 (Contributed by Rémi Lapeyre in :issue:`40967`) 1061 1062* The ``unescape()`` method in the :class:`html.parser.HTMLParser` class 1063 has been removed (it was deprecated since Python 3.4). :func:`html.unescape` 1064 should be used for converting character references to the corresponding 1065 unicode characters. 1066 1067 1068Porting to Python 3.9 1069===================== 1070 1071This section lists previously described changes and other bugfixes 1072that may require changes to your code. 1073 1074 1075Changes in the Python API 1076------------------------- 1077 1078* :func:`__import__` and :func:`importlib.util.resolve_name` now raise 1079 :exc:`ImportError` where it previously raised :exc:`ValueError`. Callers 1080 catching the specific exception type and supporting both Python 3.9 and 1081 earlier versions will need to catch both using ``except (ImportError, ValueError):``. 1082 1083* The :mod:`venv` activation scripts no longer special-case when 1084 ``__VENV_PROMPT__`` is set to ``""``. 1085 1086* The :meth:`select.epoll.unregister` method no longer ignores the 1087 :data:`~errno.EBADF` error. 1088 (Contributed by Victor Stinner in :issue:`39239`.) 1089 1090* The *compresslevel* parameter of :class:`bz2.BZ2File` became keyword-only, 1091 since the *buffering* parameter has been removed. 1092 (Contributed by Victor Stinner in :issue:`39357`.) 1093 1094* Simplified AST for subscription. Simple indices will be represented by 1095 their value, extended slices will be represented as tuples. 1096 ``Index(value)`` will return a ``value`` itself, ``ExtSlice(slices)`` 1097 will return ``Tuple(slices, Load())``. 1098 (Contributed by Serhiy Storchaka in :issue:`34822`.) 1099 1100* The :mod:`importlib` module now ignores the :envvar:`PYTHONCASEOK` 1101 environment variable when the :option:`-E` or :option:`-I` command line 1102 options are being used. 1103 1104* The *encoding* parameter has been added to the classes :class:`ftplib.FTP` and 1105 :class:`ftplib.FTP_TLS` as a keyword-only parameter, and the default encoding 1106 is changed from Latin-1 to UTF-8 to follow :rfc:`2640`. 1107 1108* :meth:`asyncio.loop.shutdown_default_executor` has been added to 1109 :class:`~asyncio.AbstractEventLoop`, meaning alternative event loops that 1110 inherit from it should have this method defined. 1111 (Contributed by Kyle Stanley in :issue:`34037`.) 1112 1113* The constant values of future flags in the :mod:`__future__` module 1114 is updated in order to prevent collision with compiler flags. Previously 1115 ``PyCF_ALLOW_TOP_LEVEL_AWAIT`` was clashing with ``CO_FUTURE_DIVISION``. 1116 (Contributed by Batuhan Taskaya in :issue:`39562`) 1117 1118* ``array('u')`` now uses ``wchar_t`` as C type instead of ``Py_UNICODE``. 1119 This change doesn't affect to its behavior because ``Py_UNICODE`` is alias 1120 of ``wchar_t`` since Python 3.3. 1121 (Contributed by Inada Naoki in :issue:`34538`.) 1122 1123* The :func:`logging.getLogger` API now returns the root logger when passed 1124 the name ``'root'``, whereas previously it returned a non-root logger named 1125 ``'root'``. This could affect cases where user code explicitly wants a 1126 non-root logger named ``'root'``, or instantiates a logger using 1127 ``logging.getLogger(__name__)`` in some top-level module called ``'root.py'``. 1128 (Contributed by Vinay Sajip in :issue:`37742`.) 1129 1130* Division handling of :class:`~pathlib.PurePath` now returns ``NotImplemented`` 1131 instead of raising a :exc:`TypeError` when passed something other than an 1132 instance of ``str`` or :class:`~pathlib.PurePath`. This allows creating 1133 compatible classes that don't inherit from those mentioned types. 1134 (Contributed by Roger Aiudi in :issue:`34775`). 1135 1136* Starting with Python 3.9.5 the :mod:`ipaddress` module no longer 1137 accepts any leading zeros in IPv4 address strings. Leading zeros are 1138 ambiguous and interpreted as octal notation by some libraries. For example 1139 the legacy function :func:`socket.inet_aton` treats leading zeros as octal 1140 notatation. glibc implementation of modern :func:`~socket.inet_pton` does 1141 not accept any leading zeros. 1142 (Contributed by Christian Heimes in :issue:`36384`). 1143 1144* :func:`codecs.lookup` now normalizes the encoding name the same way as 1145 :func:`encodings.normalize_encoding`, except that :func:`codecs.lookup` also 1146 converts the name to lower case. For example, ``"latex+latin1"`` encoding 1147 name is now normalized to ``"latex_latin1"``. 1148 (Contributed by Jordon Xu in :issue:`37751`.) 1149 1150 1151Changes in the C API 1152-------------------- 1153 1154* Instances of :ref:`heap-allocated types <heap-types>` (such as those created with 1155 :c:func:`PyType_FromSpec` and similar APIs) hold a reference to their type 1156 object since Python 3.8. As indicated in the "Changes in the C API" of Python 1157 3.8, for the vast majority of cases, there should be no side effect but for 1158 types that have a custom :c:member:`~PyTypeObject.tp_traverse` function, 1159 ensure that all custom ``tp_traverse`` functions of heap-allocated types 1160 visit the object's type. 1161 1162 Example: 1163 1164 .. code-block:: c 1165 1166 int 1167 foo_traverse(foo_struct *self, visitproc visit, void *arg) { 1168 // Rest of the traverse function 1169 #if PY_VERSION_HEX >= 0x03090000 1170 // This was not needed before Python 3.9 (Python issue 35810 and 40217) 1171 Py_VISIT(Py_TYPE(self)); 1172 #endif 1173 } 1174 1175 If your traverse function delegates to ``tp_traverse`` of its base class 1176 (or another type), ensure that ``Py_TYPE(self)`` is visited only once. 1177 Note that only :ref:`heap type <heap-types>` are expected to visit the type 1178 in ``tp_traverse``. 1179 1180 For example, if your ``tp_traverse`` function includes: 1181 1182 .. code-block:: c 1183 1184 base->tp_traverse(self, visit, arg) 1185 1186 then add: 1187 1188 .. code-block:: c 1189 1190 #if PY_VERSION_HEX >= 0x03090000 1191 // This was not needed before Python 3.9 (bpo-35810 and bpo-40217) 1192 if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) { 1193 // a heap type's tp_traverse already visited Py_TYPE(self) 1194 } else { 1195 Py_VISIT(Py_TYPE(self)); 1196 } 1197 #else 1198 1199 (See :issue:`35810` and :issue:`40217` for more information.) 1200 1201* The functions ``PyEval_CallObject``, ``PyEval_CallFunction``, 1202 ``PyEval_CallMethod`` and ``PyEval_CallObjectWithKeywords`` are deprecated. 1203 Use :c:func:`PyObject_Call` and its variants instead. 1204 (See more details in :issue:`29548`.) 1205 1206CPython bytecode changes 1207------------------------ 1208 1209* The :opcode:`LOAD_ASSERTION_ERROR` opcode was added for handling the 1210 :keyword:`assert` statement. Previously, the assert statement would not work 1211 correctly if the :exc:`AssertionError` exception was being shadowed. 1212 (Contributed by Zackery Spytz in :issue:`34880`.) 1213 1214* The :opcode:`COMPARE_OP` opcode was split into four distinct instructions: 1215 1216 * ``COMPARE_OP`` for rich comparisons 1217 * ``IS_OP`` for 'is' and 'is not' tests 1218 * ``CONTAINS_OP`` for 'in' and 'not in' tests 1219 * ``JUMP_IF_NOT_EXC_MATCH`` for checking exceptions in 'try-except' 1220 statements. 1221 1222 (Contributed by Mark Shannon in :issue:`39156`.) 1223 1224 1225Build Changes 1226============= 1227 1228* Added ``--with-platlibdir`` option to the ``configure`` script: name of the 1229 platform-specific library directory, stored in the new :attr:`sys.platlibdir` 1230 attribute. See :attr:`sys.platlibdir` attribute for more information. 1231 (Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis 1232 and Victor Stinner in :issue:`1294959`.) 1233 1234* The ``COUNT_ALLOCS`` special build macro has been removed. 1235 (Contributed by Victor Stinner in :issue:`39489`.) 1236 1237* On non-Windows platforms, the :c:func:`setenv` and :c:func:`unsetenv` 1238 functions are now required to build Python. 1239 (Contributed by Victor Stinner in :issue:`39395`.) 1240 1241* On non-Windows platforms, creating ``bdist_wininst`` installers is now 1242 officially unsupported. (See :issue:`10945` for more details.) 1243 1244* When building Python on macOS from source, ``_tkinter`` now links with 1245 non-system Tcl and Tk frameworks if they are installed in 1246 ``/Library/Frameworks``, as had been the case on older releases 1247 of macOS. If a macOS SDK is explicitly configured, by using 1248 :option:`--enable-universalsdk` or ``-isysroot``, only the SDK itself is 1249 searched. The default behavior can still be overridden with 1250 ``--with-tcltk-includes`` and ``--with-tcltk-libs``. 1251 (Contributed by Ned Deily in :issue:`34956`.) 1252 1253* Python can now be built for Windows 10 ARM64. 1254 (Contributed by Steve Dower in :issue:`33125`.) 1255 1256* Some individual tests are now skipped when ``--pgo`` is used. The tests 1257 in question increased the PGO task time significantly and likely 1258 didn't help improve optimization of the final executable. This 1259 speeds up the task by a factor of about 15x. Running the full unit test 1260 suite is slow. This change may result in a slightly less optimized build 1261 since not as many code branches will be executed. If you are willing to 1262 wait for the much slower build, the old behavior can be restored using 1263 ``./configure [..] PROFILE_TASK="-m test --pgo-extended"``. We make no 1264 guarantees as to which PGO task set produces a faster build. Users who care 1265 should run their own relevant benchmarks as results can depend on the 1266 environment, workload, and compiler tool chain. 1267 (See :issue:`36044` and :issue:`37707` for more details.) 1268 1269 1270C API Changes 1271============= 1272 1273New Features 1274------------ 1275 1276* :pep:`573`: Added :c:func:`PyType_FromModuleAndSpec` to associate 1277 a module with a class; :c:func:`PyType_GetModule` and 1278 :c:func:`PyType_GetModuleState` to retrieve the module and its state; and 1279 :c:data:`PyCMethod` and :c:data:`METH_METHOD` to allow a method to 1280 access the class it was defined in. 1281 (Contributed by Marcel Plch and Petr Viktorin in :issue:`38787`.) 1282 1283* Added :c:func:`PyFrame_GetCode` function: get a frame code. 1284 Added :c:func:`PyFrame_GetBack` function: get the frame next outer frame. 1285 (Contributed by Victor Stinner in :issue:`40421`.) 1286 1287* Added :c:func:`PyFrame_GetLineNumber` to the limited C API. 1288 (Contributed by Victor Stinner in :issue:`40421`.) 1289 1290* Added :c:func:`PyThreadState_GetInterpreter` and 1291 :c:func:`PyInterpreterState_Get` functions to get the interpreter. 1292 Added :c:func:`PyThreadState_GetFrame` function to get the current frame of a 1293 Python thread state. 1294 Added :c:func:`PyThreadState_GetID` function: get the unique identifier of a 1295 Python thread state. 1296 (Contributed by Victor Stinner in :issue:`39947`.) 1297 1298* Added a new public :c:func:`PyObject_CallNoArgs` function to the C API, which 1299 calls a callable Python object without any arguments. It is the most efficient 1300 way to call a callable Python object without any argument. 1301 (Contributed by Victor Stinner in :issue:`37194`.) 1302 1303* Changes in the limited C API (if ``Py_LIMITED_API`` macro is defined): 1304 1305 * Provide :c:func:`Py_EnterRecursiveCall` and :c:func:`Py_LeaveRecursiveCall` 1306 as regular functions for the limited API. Previously, there were defined as 1307 macros, but these macros didn't compile with the limited C API which cannot 1308 access ``PyThreadState.recursion_depth`` field (the structure is opaque in 1309 the limited C API). 1310 1311 * ``PyObject_INIT()`` and ``PyObject_INIT_VAR()`` become regular "opaque" 1312 function to hide implementation details. 1313 1314 (Contributed by Victor Stinner in :issue:`38644` and :issue:`39542`.) 1315 1316* The :c:func:`PyModule_AddType` function is added to help adding a type 1317 to a module. 1318 (Contributed by Dong-hee Na in :issue:`40024`.) 1319 1320* Added the functions :c:func:`PyObject_GC_IsTracked` and 1321 :c:func:`PyObject_GC_IsFinalized` to the public API to allow to query if 1322 Python objects are being currently tracked or have been already finalized by 1323 the garbage collector respectively. 1324 (Contributed by Pablo Galindo Salgado in :issue:`40241`.) 1325 1326* Added :c:func:`_PyObject_FunctionStr` to get a user-friendly string 1327 representation of a function-like object. 1328 (Patch by Jeroen Demeyer in :issue:`37645`.) 1329 1330* Added :c:func:`PyObject_CallOneArg` for calling an object with one 1331 positional argument 1332 (Patch by Jeroen Demeyer in :issue:`37483`.) 1333 1334 1335Porting to Python 3.9 1336--------------------- 1337 1338* ``PyInterpreterState.eval_frame`` (:pep:`523`) now requires a new mandatory 1339 *tstate* parameter (``PyThreadState*``). 1340 (Contributed by Victor Stinner in :issue:`38500`.) 1341 1342* Extension modules: :c:member:`~PyModuleDef.m_traverse`, 1343 :c:member:`~PyModuleDef.m_clear` and :c:member:`~PyModuleDef.m_free` 1344 functions of :c:type:`PyModuleDef` are no longer called if the module state 1345 was requested but is not allocated yet. This is the case immediately after 1346 the module is created and before the module is executed 1347 (:c:data:`Py_mod_exec` function). More precisely, these functions are not called 1348 if :c:member:`~PyModuleDef.m_size` is greater than 0 and the module state (as 1349 returned by :c:func:`PyModule_GetState`) is ``NULL``. 1350 1351 Extension modules without module state (``m_size <= 0``) are not affected. 1352 1353* If :c:func:`Py_AddPendingCall` is called in a subinterpreter, the function is 1354 now scheduled to be called from the subinterpreter, rather than being called 1355 from the main interpreter. Each subinterpreter now has its own list of 1356 scheduled calls. 1357 (Contributed by Victor Stinner in :issue:`39984`.) 1358 1359* The Windows registry is no longer used to initialize :data:`sys.path` when 1360 the ``-E`` option is used (if :c:member:`PyConfig.use_environment` is set to 1361 ``0``). This is significant when embedding Python on Windows. 1362 (Contributed by Zackery Spytz in :issue:`8901`.) 1363 1364* The global variable :c:data:`PyStructSequence_UnnamedField` is now a constant 1365 and refers to a constant string. 1366 (Contributed by Serhiy Storchaka in :issue:`38650`.) 1367 1368* The :c:type:`PyGC_Head` structure is now opaque. It is only defined in the 1369 internal C API (``pycore_gc.h``). 1370 (Contributed by Victor Stinner in :issue:`40241`.) 1371 1372* The ``Py_UNICODE_COPY``, ``Py_UNICODE_FILL``, ``PyUnicode_WSTR_LENGTH``, 1373 :c:func:`PyUnicode_FromUnicode`, :c:func:`PyUnicode_AsUnicode`, 1374 ``_PyUnicode_AsUnicode``, and :c:func:`PyUnicode_AsUnicodeAndSize` are 1375 marked as deprecated in C. They have been deprecated by :pep:`393` since 1376 Python 3.3. 1377 (Contributed by Inada Naoki in :issue:`36346`.) 1378 1379* The :c:func:`Py_FatalError` function is replaced with a macro which logs 1380 automatically the name of the current function, unless the 1381 ``Py_LIMITED_API`` macro is defined. 1382 (Contributed by Victor Stinner in :issue:`39882`.) 1383 1384* The vectorcall protocol now requires that the caller passes only strings as 1385 keyword names. (See :issue:`37540` for more information.) 1386 1387* Implementation details of a number of macros and functions are now hidden: 1388 1389 * :c:func:`PyObject_IS_GC` macro was converted to a function. 1390 1391 * The :c:func:`PyObject_NEW` macro becomes an alias to the 1392 :c:func:`PyObject_New` macro, and the :c:func:`PyObject_NEW_VAR` macro 1393 becomes an alias to the :c:func:`PyObject_NewVar` macro. They no longer 1394 access directly the :c:member:`PyTypeObject.tp_basicsize` member. 1395 1396 * :c:func:`PyObject_GET_WEAKREFS_LISTPTR` macro was converted to a function: 1397 the macro accessed directly the :c:member:`PyTypeObject.tp_weaklistoffset` 1398 member. 1399 1400 * :c:func:`PyObject_CheckBuffer` macro was converted to a function: the macro 1401 accessed directly the :c:member:`PyTypeObject.tp_as_buffer` member. 1402 1403 * :c:func:`PyIndex_Check` is now always declared as an opaque function to hide 1404 implementation details: removed the ``PyIndex_Check()`` macro. The macro accessed 1405 directly the :c:member:`PyTypeObject.tp_as_number` member. 1406 1407 (See :issue:`40170` for more details.) 1408 1409Removed 1410------- 1411 1412* Excluded ``PyFPE_START_PROTECT()`` and ``PyFPE_END_PROTECT()`` macros of 1413 ``pyfpe.h`` from the limited C API. 1414 (Contributed by Victor Stinner in :issue:`38835`.) 1415 1416* The ``tp_print`` slot of :ref:`PyTypeObject <type-structs>` has been removed. 1417 It was used for printing objects to files in Python 2.7 and before. Since 1418 Python 3.0, it has been ignored and unused. 1419 (Contributed by Jeroen Demeyer in :issue:`36974`.) 1420 1421* Changes in the limited C API (if ``Py_LIMITED_API`` macro is defined): 1422 1423 * Excluded the following functions from the limited C API: 1424 1425 * ``PyThreadState_DeleteCurrent()`` 1426 (Contributed by Joannah Nanjekye in :issue:`37878`.) 1427 * ``_Py_CheckRecursionLimit`` 1428 * ``_Py_NewReference()`` 1429 * ``_Py_ForgetReference()`` 1430 * ``_PyTraceMalloc_NewReference()`` 1431 * ``_Py_GetRefTotal()`` 1432 * The trashcan mechanism which never worked in the limited C API. 1433 * ``PyTrash_UNWIND_LEVEL`` 1434 * ``Py_TRASHCAN_BEGIN_CONDITION`` 1435 * ``Py_TRASHCAN_BEGIN`` 1436 * ``Py_TRASHCAN_END`` 1437 * ``Py_TRASHCAN_SAFE_BEGIN`` 1438 * ``Py_TRASHCAN_SAFE_END`` 1439 1440 * Moved following functions and definitions to the internal C API: 1441 1442 * ``_PyDebug_PrintTotalRefs()`` 1443 * ``_Py_PrintReferences()`` 1444 * ``_Py_PrintReferenceAddresses()`` 1445 * ``_Py_tracemalloc_config`` 1446 * ``_Py_AddToAllObjects()`` (specific to ``Py_TRACE_REFS`` build) 1447 1448 (Contributed by Victor Stinner in :issue:`38644` and :issue:`39542`.) 1449 1450* Removed ``_PyRuntime.getframe`` hook and removed ``_PyThreadState_GetFrame`` 1451 macro which was an alias to ``_PyRuntime.getframe``. They were only exposed 1452 by the internal C API. Removed also ``PyThreadFrameGetter`` type. 1453 (Contributed by Victor Stinner in :issue:`39946`.) 1454 1455* Removed the following functions from the C API. Call :c:func:`PyGC_Collect` 1456 explicitly to clear all free lists. 1457 (Contributed by Inada Naoki and Victor Stinner in :issue:`37340`, 1458 :issue:`38896` and :issue:`40428`.) 1459 1460 * ``PyAsyncGen_ClearFreeLists()`` 1461 * ``PyContext_ClearFreeList()`` 1462 * ``PyDict_ClearFreeList()`` 1463 * ``PyFloat_ClearFreeList()`` 1464 * ``PyFrame_ClearFreeList()`` 1465 * ``PyList_ClearFreeList()`` 1466 * ``PyMethod_ClearFreeList()`` and ``PyCFunction_ClearFreeList()``: 1467 the free lists of bound method objects have been removed. 1468 * ``PySet_ClearFreeList()``: the set free list has been removed 1469 in Python 3.4. 1470 * ``PyTuple_ClearFreeList()`` 1471 * ``PyUnicode_ClearFreeList()``: the Unicode free list has been removed in 1472 Python 3.3. 1473 1474* Removed ``_PyUnicode_ClearStaticStrings()`` function. 1475 (Contributed by Victor Stinner in :issue:`39465`.) 1476 1477* Removed ``Py_UNICODE_MATCH``. It has been deprecated by :pep:`393`, and 1478 broken since Python 3.3. The :c:func:`PyUnicode_Tailmatch` function can be 1479 used instead. 1480 (Contributed by Inada Naoki in :issue:`36346`.) 1481 1482* Cleaned header files of interfaces defined but with no implementation. 1483 The public API symbols being removed are: 1484 ``_PyBytes_InsertThousandsGroupingLocale``, 1485 ``_PyBytes_InsertThousandsGrouping``, ``_Py_InitializeFromArgs``, 1486 ``_Py_InitializeFromWideArgs``, ``_PyFloat_Repr``, ``_PyFloat_Digits``, 1487 ``_PyFloat_DigitsInit``, ``PyFrame_ExtendStack``, ``_PyAIterWrapper_Type``, 1488 ``PyNullImporter_Type``, ``PyCmpWrapper_Type``, ``PySortWrapper_Type``, 1489 ``PyNoArgsFunction``. 1490 (Contributed by Pablo Galindo Salgado in :issue:`39372`.) 1491 1492Notable changes in Python 3.9.1 1493=============================== 1494 1495typing 1496------ 1497 1498The behavior of :class:`typing.Literal` was changed to conform with :pep:`586` 1499and to match the behavior of static type checkers specified in the PEP. 1500 15011. ``Literal`` now de-duplicates parameters. 15022. Equality comparisons between ``Literal`` objects are now order independent. 15033. ``Literal`` comparisons now respect types. For example, 1504 ``Literal[0] == Literal[False]`` previously evaluated to ``True``. It is 1505 now ``False``. To support this change, the internally used type cache now 1506 supports differentiating types. 15074. ``Literal`` objects will now raise a :exc:`TypeError` exception during 1508 equality comparisons if any of their parameters are not :term:`hashable`. 1509 Note that declaring ``Literal`` with mutable parameters will not throw 1510 an error:: 1511 1512 >>> from typing import Literal 1513 >>> Literal[{0}] 1514 >>> Literal[{0}] == Literal[{False}] 1515 Traceback (most recent call last): 1516 File "<stdin>", line 1, in <module> 1517 TypeError: unhashable type: 'set' 1518 1519(Contributed by Yurii Karabas in :issue:`42345`.) 1520 1521macOS 11.0 (Big Sur) and Apple Silicon Mac support 1522-------------------------------------------------- 1523 1524As of 3.9.1, Python now fully supports building and running on macOS 11.0 1525(Big Sur) and on Apple Silicon Macs (based on the ``ARM64`` architecture). 1526A new universal build variant, ``universal2``, is now available to natively 1527support both ``ARM64`` and ``Intel 64`` in one set of executables. Binaries 1528can also now be built on current versions of macOS to be deployed on a range 1529of older macOS versions (tested to 10.9) while making some newer OS 1530functions and options conditionally available based on the operating system 1531version in use at runtime ("weaklinking"). 1532 1533(Contributed by Ronald Oussoren and Lawrence D'Anna in :issue:`41100`.) 1534 1535Notable changes in Python 3.9.2 1536=============================== 1537 1538collections.abc 1539--------------- 1540 1541:class:`collections.abc.Callable` generic now flattens type parameters, similar 1542to what :data:`typing.Callable` currently does. This means that 1543``collections.abc.Callable[[int, str], str]`` will have ``__args__`` of 1544``(int, str, str)``; previously this was ``([int, str], str)``. To allow this 1545change, :class:`types.GenericAlias` can now be subclassed, and a subclass will 1546be returned when subscripting the :class:`collections.abc.Callable` type. 1547Code which accesses the arguments via :func:`typing.get_args` or ``__args__`` 1548need to account for this change. A :exc:`DeprecationWarning` may be emitted for 1549invalid forms of parameterizing :class:`collections.abc.Callable` which may have 1550passed silently in Python 3.9.1. This :exc:`DeprecationWarning` will 1551become a :exc:`TypeError` in Python 3.10. 1552(Contributed by Ken Jin in :issue:`42195`.) 1553 1554urllib.parse 1555------------ 1556 1557Earlier Python versions allowed using both ``;`` and ``&`` as 1558query parameter separators in :func:`urllib.parse.parse_qs` and 1559:func:`urllib.parse.parse_qsl`. Due to security concerns, and to conform with 1560newer W3C recommendations, this has been changed to allow only a single 1561separator key, with ``&`` as the default. This change also affects 1562:func:`cgi.parse` and :func:`cgi.parse_multipart` as they use the affected 1563functions internally. For more details, please see their respective 1564documentation. 1565(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in :issue:`42967`.) 1566