1**************************** 2 What's New In Python 3.2 3**************************** 4 5:Author: Raymond Hettinger 6 7.. $Id$ 8 Rules for maintenance: 9 10 * Anyone can add text to this document. Do not spend very much time 11 on the wording of your changes, because your text will probably 12 get rewritten. (Note, during release candidate phase or just before 13 a beta release, please use the tracker instead -- this helps avoid 14 merge conflicts. If you must add a suggested entry directly, 15 please put it in an XXX comment and the maintainer will take notice). 16 17 * The maintainer will go through Misc/NEWS periodically and add 18 changes; it's therefore more important to add your changes to 19 Misc/NEWS than to this file. 20 21 * This is not a complete list of every single change; completeness 22 is the purpose of Misc/NEWS. Some changes I consider too small 23 or esoteric to include. If such a change is added to the text, 24 I'll just remove it. (This is another reason you shouldn't spend 25 too much time on writing your addition.) 26 27 * If you want to draw your new text to the attention of the 28 maintainer, add 'XXX' to the beginning of the paragraph or 29 section. 30 31 * It's OK to just add a fragmentary note about a change. For 32 example: "XXX Describe the transmogrify() function added to the 33 socket module." The maintainer will research the change and 34 write the necessary text. 35 36 * You can comment out your additions if you like, but it's not 37 necessary (especially when a final release is some months away). 38 39 * Credit the author of a patch or bugfix. Just the name is 40 sufficient; the e-mail address isn't necessary. It's helpful to 41 add the issue number: 42 43 XXX Describe the transmogrify() function added to the socket 44 module. 45 46 (Contributed by P.Y. Developer; :issue:`12345`.) 47 48 This saves the maintainer the effort of going through the SVN log 49 when researching a change. 50 51This article explains the new features in Python 3.2 as compared to 3.1. 52Python 3.2 was released on February 20, 2011. It 53focuses on a few highlights and gives a few examples. For full details, see the 54`Misc/NEWS 55<https://github.com/python/cpython/blob/076ca6c3c8df3030307e548d9be792ce3c1c6eea/Misc/NEWS>`__ 56file. 57 58.. seealso:: 59 60 :pep:`392` - Python 3.2 Release Schedule 61 62 63PEP 384: Defining a Stable ABI 64============================== 65 66In the past, extension modules built for one Python version were often 67not usable with other Python versions. Particularly on Windows, every 68feature release of Python required rebuilding all extension modules that 69one wanted to use. This requirement was the result of the free access to 70Python interpreter internals that extension modules could use. 71 72With Python 3.2, an alternative approach becomes available: extension 73modules which restrict themselves to a limited API (by defining 74Py_LIMITED_API) cannot use many of the internals, but are constrained 75to a set of API functions that are promised to be stable for several 76releases. As a consequence, extension modules built for 3.2 in that 77mode will also work with 3.3, 3.4, and so on. Extension modules that 78make use of details of memory structures can still be built, but will 79need to be recompiled for every feature release. 80 81.. seealso:: 82 83 :pep:`384` - Defining a Stable ABI 84 PEP written by Martin von Löwis. 85 86 87PEP 389: Argparse Command Line Parsing Module 88============================================= 89 90A new module for command line parsing, :mod:`argparse`, was introduced to 91overcome the limitations of :mod:`optparse` which did not provide support for 92positional arguments (not just options), subcommands, required options and other 93common patterns of specifying and validating options. 94 95This module has already had widespread success in the community as a 96third-party module. Being more fully featured than its predecessor, the 97:mod:`argparse` module is now the preferred module for command-line processing. 98The older module is still being kept available because of the substantial amount 99of legacy code that depends on it. 100 101Here's an annotated example parser showing features like limiting results to a 102set of choices, specifying a *metavar* in the help screen, validating that one 103or more positional arguments is present, and making a required option:: 104 105 import argparse 106 parser = argparse.ArgumentParser( 107 description = 'Manage servers', # main description for help 108 epilog = 'Tested on Solaris and Linux') # displayed after help 109 parser.add_argument('action', # argument name 110 choices = ['deploy', 'start', 'stop'], # three allowed values 111 help = 'action on each target') # help msg 112 parser.add_argument('targets', 113 metavar = 'HOSTNAME', # var name used in help msg 114 nargs = '+', # require one or more targets 115 help = 'url for target machines') # help msg explanation 116 parser.add_argument('-u', '--user', # -u or --user option 117 required = True, # make it a required argument 118 help = 'login as user') 119 120Example of calling the parser on a command string:: 121 122 >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain' 123 >>> result = parser.parse_args(cmd.split()) 124 >>> result.action 125 'deploy' 126 >>> result.targets 127 ['sneezy.example.com', 'sleepy.example.com'] 128 >>> result.user 129 'skycaptain' 130 131Example of the parser's automatically generated help:: 132 133 >>> parser.parse_args('-h'.split()) 134 135 usage: manage_cloud.py [-h] -u USER 136 {deploy,start,stop} HOSTNAME [HOSTNAME ...] 137 138 Manage servers 139 140 positional arguments: 141 {deploy,start,stop} action on each target 142 HOSTNAME url for target machines 143 144 optional arguments: 145 -h, --help show this help message and exit 146 -u USER, --user USER login as user 147 148 Tested on Solaris and Linux 149 150An especially nice :mod:`argparse` feature is the ability to define subparsers, 151each with their own argument patterns and help displays:: 152 153 import argparse 154 parser = argparse.ArgumentParser(prog='HELM') 155 subparsers = parser.add_subparsers() 156 157 parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup 158 parser_l.add_argument('-m', '--missiles', action='store_true') 159 parser_l.add_argument('-t', '--torpedos', action='store_true') 160 161 parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup 162 aliases=('steer', 'turn')) # equivalent names 163 parser_m.add_argument('-c', '--course', type=int, required=True) 164 parser_m.add_argument('-s', '--speed', type=int, default=0) 165 166.. code-block:: shell-session 167 168 $ ./helm.py --help # top level help (launch and move) 169 $ ./helm.py launch --help # help for launch options 170 $ ./helm.py launch --missiles # set missiles=True and torpedos=False 171 $ ./helm.py steer --course 180 --speed 5 # set movement parameters 172 173.. seealso:: 174 175 :pep:`389` - New Command Line Parsing Module 176 PEP written by Steven Bethard. 177 178 :ref:`upgrading-optparse-code` for details on the differences from :mod:`optparse`. 179 180 181PEP 391: Dictionary Based Configuration for Logging 182==================================================== 183 184The :mod:`logging` module provided two kinds of configuration, one style with 185function calls for each option or another style driven by an external file saved 186in a :mod:`ConfigParser` format. Those options did not provide the flexibility 187to create configurations from JSON or YAML files, nor did they support 188incremental configuration, which is needed for specifying logger options from a 189command line. 190 191To support a more flexible style, the module now offers 192:func:`logging.config.dictConfig` for specifying logging configuration with 193plain Python dictionaries. The configuration options include formatters, 194handlers, filters, and loggers. Here's a working example of a configuration 195dictionary:: 196 197 {"version": 1, 198 "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"}, 199 "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"} 200 }, 201 "handlers": {"console": { 202 "class": "logging.StreamHandler", 203 "formatter": "brief", 204 "level": "INFO", 205 "stream": "ext://sys.stdout"}, 206 "console_priority": { 207 "class": "logging.StreamHandler", 208 "formatter": "full", 209 "level": "ERROR", 210 "stream": "ext://sys.stderr"} 211 }, 212 "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}} 213 214 215If that dictionary is stored in a file called :file:`conf.json`, it can be 216loaded and called with code like this:: 217 218 >>> import json, logging.config 219 >>> with open('conf.json') as f: 220 ... conf = json.load(f) 221 ... 222 >>> logging.config.dictConfig(conf) 223 >>> logging.info("Transaction completed normally") 224 INFO : root : Transaction completed normally 225 >>> logging.critical("Abnormal termination") 226 2011-02-17 11:14:36,694 root CRITICAL Abnormal termination 227 228.. seealso:: 229 230 :pep:`391` - Dictionary Based Configuration for Logging 231 PEP written by Vinay Sajip. 232 233 234PEP 3148: The ``concurrent.futures`` module 235============================================ 236 237Code for creating and managing concurrency is being collected in a new top-level 238namespace, *concurrent*. Its first member is a *futures* package which provides 239a uniform high-level interface for managing threads and processes. 240 241The design for :mod:`concurrent.futures` was inspired by the 242*java.util.concurrent* package. In that model, a running call and its result 243are represented by a :class:`~concurrent.futures.Future` object that abstracts 244features common to threads, processes, and remote procedure calls. That object 245supports status checks (running or done), timeouts, cancellations, adding 246callbacks, and access to results or exceptions. 247 248The primary offering of the new module is a pair of executor classes for 249launching and managing calls. The goal of the executors is to make it easier to 250use existing tools for making parallel calls. They save the effort needed to 251setup a pool of resources, launch the calls, create a results queue, add 252time-out handling, and limit the total number of threads, processes, or remote 253procedure calls. 254 255Ideally, each application should share a single executor across multiple 256components so that process and thread limits can be centrally managed. This 257solves the design challenge that arises when each component has its own 258competing strategy for resource management. 259 260Both classes share a common interface with three methods: 261:meth:`~concurrent.futures.Executor.submit` for scheduling a callable and 262returning a :class:`~concurrent.futures.Future` object; 263:meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls 264at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing 265resources. The class is a :term:`context manager` and can be used in a 266:keyword:`with` statement to assure that resources are automatically released 267when currently pending futures are done executing. 268 269A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a 270launch of four parallel threads for copying files:: 271 272 import concurrent.futures, shutil 273 with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e: 274 e.submit(shutil.copy, 'src1.txt', 'dest1.txt') 275 e.submit(shutil.copy, 'src2.txt', 'dest2.txt') 276 e.submit(shutil.copy, 'src3.txt', 'dest3.txt') 277 e.submit(shutil.copy, 'src3.txt', 'dest4.txt') 278 279.. seealso:: 280 281 :pep:`3148` - Futures -- Execute Computations Asynchronously 282 PEP written by Brian Quinlan. 283 284 :ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an 285 example using threads to fetch multiple web pages in parallel. 286 287 :ref:`Code for computing prime numbers in 288 parallel<processpoolexecutor-example>`, an example demonstrating 289 :class:`~concurrent.futures.ProcessPoolExecutor`. 290 291 292PEP 3147: PYC Repository Directories 293===================================== 294 295Python's scheme for caching bytecode in *.pyc* files did not work well in 296environments with multiple Python interpreters. If one interpreter encountered 297a cached file created by another interpreter, it would recompile the source and 298overwrite the cached file, thus losing the benefits of caching. 299 300The issue of "pyc fights" has become more pronounced as it has become 301commonplace for Linux distributions to ship with multiple versions of Python. 302These conflicts also arise with CPython alternatives such as Unladen Swallow. 303 304To solve this problem, Python's import machinery has been extended to use 305distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and 306Unladen Swallow each competing for a file called "mymodule.pyc", they will now 307look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and 308"mymodule.unladen10.pyc". And to prevent all of these new files from 309cluttering source directories, the *pyc* files are now collected in a 310"__pycache__" directory stored under the package directory. 311 312Aside from the filenames and target directories, the new scheme has a few 313aspects that are visible to the programmer: 314 315* Imported modules now have a :attr:`__cached__` attribute which stores the name 316 of the actual file that was imported: 317 318 >>> import collections 319 >>> collections.__cached__ # doctest: +SKIP 320 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' 321 322* The tag that is unique to each interpreter is accessible from the :mod:`imp` 323 module: 324 325 >>> import imp 326 >>> imp.get_tag() # doctest: +SKIP 327 'cpython-32' 328 329* Scripts that try to deduce source filename from the imported file now need to 330 be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc" 331 filename. Instead, use the new functions in the :mod:`imp` module: 332 333 >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') 334 'c:/py32/lib/collections.py' 335 >>> imp.cache_from_source('c:/py32/lib/collections.py') # doctest: +SKIP 336 'c:/py32/lib/__pycache__/collections.cpython-32.pyc' 337 338* The :mod:`py_compile` and :mod:`compileall` modules have been updated to 339 reflect the new naming convention and target directory. The command-line 340 invocation of *compileall* has new options: ``-i`` for 341 specifying a list of files and directories to compile and ``-b`` which causes 342 bytecode files to be written to their legacy location rather than 343 *__pycache__*. 344 345* The :mod:`importlib.abc` module has been updated with new :term:`abstract base 346 classes <abstract base class>` for loading bytecode files. The obsolete 347 ABCs, :class:`~importlib.abc.PyLoader` and 348 :class:`~importlib.abc.PyPycLoader`, have been deprecated (instructions on how 349 to stay Python 3.1 compatible are included with the documentation). 350 351.. seealso:: 352 353 :pep:`3147` - PYC Repository Directories 354 PEP written by Barry Warsaw. 355 356 357PEP 3149: ABI Version Tagged .so Files 358====================================== 359 360The PYC repository directory allows multiple bytecode cache files to be 361co-located. This PEP implements a similar mechanism for shared object files by 362giving them a common directory and distinct names for each version. 363 364The common directory is "pyshared" and the file names are made distinct by 365identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the 366major and minor version numbers, and optional build flags (such as "d" for 367debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo", 368you may see these files when the distribution package is installed:: 369 370 /usr/share/pyshared/foo.cpython-32m.so 371 /usr/share/pyshared/foo.cpython-33md.so 372 373In Python itself, the tags are accessible from functions in the :mod:`sysconfig` 374module:: 375 376 >>> import sysconfig 377 >>> sysconfig.get_config_var('SOABI') # find the version tag 378 'cpython-32mu' 379 >>> sysconfig.get_config_var('EXT_SUFFIX') # find the full filename extension 380 '.cpython-32mu.so' 381 382.. seealso:: 383 384 :pep:`3149` - ABI Version Tagged .so Files 385 PEP written by Barry Warsaw. 386 387 388PEP 3333: Python Web Server Gateway Interface v1.0.1 389===================================================== 390 391This informational PEP clarifies how bytes/text issues are to be handled by the 392WSGI protocol. The challenge is that string handling in Python 3 is most 393conveniently handled with the :class:`str` type even though the HTTP protocol 394is itself bytes oriented. 395 396The PEP differentiates so-called *native strings* that are used for 397request/response headers and metadata versus *byte strings* which are used for 398the bodies of requests and responses. 399 400The *native strings* are always of type :class:`str` but are restricted to code 401points between *U+0000* through *U+00FF* which are translatable to bytes using 402*Latin-1* encoding. These strings are used for the keys and values in the 403environment dictionary and for response headers and statuses in the 404:func:`start_response` function. They must follow :rfc:`2616` with respect to 405encoding. That is, they must either be *ISO-8859-1* characters or use 406:rfc:`2047` MIME encoding. 407 408For developers porting WSGI applications from Python 2, here are the salient 409points: 410 411* If the app already used strings for headers in Python 2, no change is needed. 412 413* If instead, the app encoded output headers or decoded input headers, then the 414 headers will need to be re-encoded to Latin-1. For example, an output header 415 encoded in utf-8 was using ``h.encode('utf-8')`` now needs to convert from 416 bytes to native strings using ``h.encode('utf-8').decode('latin-1')``. 417 418* Values yielded by an application or sent using the :meth:`write` method 419 must be byte strings. The :func:`start_response` function and environ 420 must use native strings. The two cannot be mixed. 421 422For server implementers writing CGI-to-WSGI pathways or other CGI-style 423protocols, the users must to be able access the environment using native strings 424even though the underlying platform may have a different convention. To bridge 425this gap, the :mod:`wsgiref` module has a new function, 426:func:`wsgiref.handlers.read_environ` for transcoding CGI variables from 427:attr:`os.environ` into native strings and returning a new dictionary. 428 429.. seealso:: 430 431 :pep:`3333` - Python Web Server Gateway Interface v1.0.1 432 PEP written by Phillip Eby. 433 434 435Other Language Changes 436====================== 437 438Some smaller changes made to the core Python language are: 439 440* String formatting for :func:`format` and :meth:`str.format` gained new 441 capabilities for the format character **#**. Previously, for integers in 442 binary, octal, or hexadecimal, it caused the output to be prefixed with '0b', 443 '0o', or '0x' respectively. Now it can also handle floats, complex, and 444 Decimal, causing the output to always have a decimal point even when no digits 445 follow it. 446 447 >>> format(20, '#o') 448 '0o24' 449 >>> format(12.34, '#5.0f') 450 ' 12.' 451 452 (Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.) 453 454* There is also a new :meth:`str.format_map` method that extends the 455 capabilities of the existing :meth:`str.format` method by accepting arbitrary 456 :term:`mapping` objects. This new method makes it possible to use string 457 formatting with any of Python's many dictionary-like objects such as 458 :class:`~collections.defaultdict`, :class:`~shelve.Shelf`, 459 :class:`~configparser.ConfigParser`, or :mod:`dbm`. It is also useful with 460 custom :class:`dict` subclasses that normalize keys before look-up or that 461 supply a :meth:`__missing__` method for unknown keys:: 462 463 >>> import shelve 464 >>> d = shelve.open('tmp.shl') 465 >>> 'The {project_name} status is {status} as of {date}'.format_map(d) 466 'The testing project status is green as of February 15, 2011' 467 468 >>> class LowerCasedDict(dict): 469 ... def __getitem__(self, key): 470 ... return dict.__getitem__(self, key.lower()) 471 >>> lcd = LowerCasedDict(part='widgets', quantity=10) 472 >>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd) 473 'There are 10 widgets in stock' 474 475 >>> class PlaceholderDict(dict): 476 ... def __missing__(self, key): 477 ... return '<{}>'.format(key) 478 >>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict()) 479 'Hello <name>, welcome to <location>' 480 481 (Suggested by Raymond Hettinger and implemented by Eric Smith in 482 :issue:`6081`.) 483 484* The interpreter can now be started with a quiet option, ``-q``, to prevent 485 the copyright and version information from being displayed in the interactive 486 mode. The option can be introspected using the :attr:`sys.flags` attribute: 487 488 .. code-block:: shell-session 489 490 $ python -q 491 >>> sys.flags 492 sys.flags(debug=0, division_warning=0, inspect=0, interactive=0, 493 optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, 494 ignore_environment=0, verbose=0, bytes_warning=0, quiet=1) 495 496 (Contributed by Marcin Wojdyr in :issue:`1772833`). 497 498* The :func:`hasattr` function works by calling :func:`getattr` and detecting 499 whether an exception is raised. This technique allows it to detect methods 500 created dynamically by :meth:`__getattr__` or :meth:`__getattribute__` which 501 would otherwise be absent from the class dictionary. Formerly, *hasattr* 502 would catch any exception, possibly masking genuine errors. Now, *hasattr* 503 has been tightened to only catch :exc:`AttributeError` and let other 504 exceptions pass through:: 505 506 >>> class A: 507 ... @property 508 ... def f(self): 509 ... return 1 // 0 510 ... 511 >>> a = A() 512 >>> hasattr(a, 'f') 513 Traceback (most recent call last): 514 ... 515 ZeroDivisionError: integer division or modulo by zero 516 517 (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.) 518 519* The :func:`str` of a float or complex number is now the same as its 520 :func:`repr`. Previously, the :func:`str` form was shorter but that just 521 caused confusion and is no longer needed now that the shortest possible 522 :func:`repr` is displayed by default: 523 524 >>> import math 525 >>> repr(math.pi) 526 '3.141592653589793' 527 >>> str(math.pi) 528 '3.141592653589793' 529 530 (Proposed and implemented by Mark Dickinson; :issue:`9337`.) 531 532* :class:`memoryview` objects now have a :meth:`~memoryview.release()` method 533 and they also now support the context management protocol. This allows timely 534 release of any resources that were acquired when requesting a buffer from the 535 original object. 536 537 >>> with memoryview(b'abcdefgh') as v: 538 ... print(v.tolist()) 539 [97, 98, 99, 100, 101, 102, 103, 104] 540 541 (Added by Antoine Pitrou; :issue:`9757`.) 542 543* Previously it was illegal to delete a name from the local namespace if it 544 occurs as a free variable in a nested block:: 545 546 def outer(x): 547 def inner(): 548 return x 549 inner() 550 del x 551 552 This is now allowed. Remember that the target of an :keyword:`except` clause 553 is cleared, so this code which used to work with Python 2.6, raised a 554 :exc:`SyntaxError` with Python 3.1 and now works again:: 555 556 def f(): 557 def print_error(): 558 print(e) 559 try: 560 something 561 except Exception as e: 562 print_error() 563 # implicit "del e" here 564 565 (See :issue:`4617`.) 566 567* The internal :c:type:`structsequence` tool now creates subclasses of tuple. 568 This means that C structures like those returned by :func:`os.stat`, 569 :func:`time.gmtime`, and :attr:`sys.version_info` now work like a 570 :term:`named tuple` and now work with functions and methods that 571 expect a tuple as an argument. This is a big step forward in making the C 572 structures as flexible as their pure Python counterparts: 573 574 >>> import sys 575 >>> isinstance(sys.version_info, tuple) 576 True 577 >>> 'Version %d.%d.%d %s(%d)' % sys.version_info # doctest: +SKIP 578 'Version 3.2.0 final(0)' 579 580 (Suggested by Arfrever Frehtes Taifersar Arahesis and implemented 581 by Benjamin Peterson in :issue:`8413`.) 582 583* Warnings are now easier to control using the :envvar:`PYTHONWARNINGS` 584 environment variable as an alternative to using ``-W`` at the command line: 585 586 .. code-block:: shell-session 587 588 $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::' 589 590 (Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.) 591 592* A new warning category, :exc:`ResourceWarning`, has been added. It is 593 emitted when potential issues with resource consumption or cleanup 594 are detected. It is silenced by default in normal release builds but 595 can be enabled through the means provided by the :mod:`warnings` 596 module, or on the command line. 597 598 A :exc:`ResourceWarning` is issued at interpreter shutdown if the 599 :data:`gc.garbage` list isn't empty, and if :attr:`gc.DEBUG_UNCOLLECTABLE` is 600 set, all uncollectable objects are printed. This is meant to make the 601 programmer aware that their code contains object finalization issues. 602 603 A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed 604 without having been explicitly closed. While the deallocator for such 605 object ensures it closes the underlying operating system resource 606 (usually, a file descriptor), the delay in deallocating the object could 607 produce various issues, especially under Windows. Here is an example 608 of enabling the warning from the command line: 609 610 .. code-block:: shell-session 611 612 $ python -q -Wdefault 613 >>> f = open("foo", "wb") 614 >>> del f 615 __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'> 616 617 (Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.) 618 619* :class:`range` objects now support *index* and *count* methods. This is part 620 of an effort to make more objects fully implement the 621 :class:`collections.Sequence` :term:`abstract base class`. As a result, the 622 language will have a more uniform API. In addition, :class:`range` objects 623 now support slicing and negative indices, even with values larger than 624 :attr:`sys.maxsize`. This makes *range* more interoperable with lists:: 625 626 >>> range(0, 100, 2).count(10) 627 1 628 >>> range(0, 100, 2).index(10) 629 5 630 >>> range(0, 100, 2)[5] 631 10 632 >>> range(0, 100, 2)[0:5] 633 range(0, 10, 2) 634 635 (Contributed by Daniel Stutzbach in :issue:`9213`, by Alexander Belopolsky 636 in :issue:`2690`, and by Nick Coghlan in :issue:`10889`.) 637 638* The :func:`callable` builtin function from Py2.x was resurrected. It provides 639 a concise, readable alternative to using an :term:`abstract base class` in an 640 expression like ``isinstance(x, collections.Callable)``: 641 642 >>> callable(max) 643 True 644 >>> callable(20) 645 False 646 647 (See :issue:`10518`.) 648 649* Python's import mechanism can now load modules installed in directories with 650 non-ASCII characters in the path name. This solved an aggravating problem 651 with home directories for users with non-ASCII characters in their usernames. 652 653 (Required extensive work by Victor Stinner in :issue:`9425`.) 654 655 656New, Improved, and Deprecated Modules 657===================================== 658 659Python's standard library has undergone significant maintenance efforts and 660quality improvements. 661 662The biggest news for Python 3.2 is that the :mod:`email` package, :mod:`mailbox` 663module, and :mod:`nntplib` modules now work correctly with the bytes/text model 664in Python 3. For the first time, there is correct handling of messages with 665mixed encodings. 666 667Throughout the standard library, there has been more careful attention to 668encodings and text versus bytes issues. In particular, interactions with the 669operating system are now better able to exchange non-ASCII data using the 670Windows MBCS encoding, locale-aware encodings, or UTF-8. 671 672Another significant win is the addition of substantially better support for 673*SSL* connections and security certificates. 674 675In addition, more classes now implement a :term:`context manager` to support 676convenient and reliable resource clean-up using a :keyword:`with` statement. 677 678email 679----- 680 681The usability of the :mod:`email` package in Python 3 has been mostly fixed by 682the extensive efforts of R. David Murray. The problem was that emails are 683typically read and stored in the form of :class:`bytes` rather than :class:`str` 684text, and they may contain multiple encodings within a single email. So, the 685email package had to be extended to parse and generate email messages in bytes 686format. 687 688* New functions :func:`~email.message_from_bytes` and 689 :func:`~email.message_from_binary_file`, and new classes 690 :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser` 691 allow binary message data to be parsed into model objects. 692 693* Given bytes input to the model, :meth:`~email.message.Message.get_payload` 694 will by default decode a message body that has a 695 :mailheader:`Content-Transfer-Encoding` of *8bit* using the charset 696 specified in the MIME headers and return the resulting string. 697 698* Given bytes input to the model, :class:`~email.generator.Generator` will 699 convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of 700 *8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`. 701 702 Headers with unencoded non-ASCII bytes are deemed to be :rfc:`2047`\ -encoded 703 using the *unknown-8bit* character set. 704 705* A new class :class:`~email.generator.BytesGenerator` produces bytes as output, 706 preserving any unchanged non-ASCII data that was present in the input used to 707 build the model, including message bodies with a 708 :mailheader:`Content-Transfer-Encoding` of *8bit*. 709 710* The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string 711 for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method, 712 and a new method, :meth:`~smtplib.SMTP.send_message` accepts a 713 :class:`~email.message.Message` object and can optionally obtain the 714 *from_addr* and *to_addrs* addresses directly from the object. 715 716(Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.) 717 718elementtree 719----------- 720 721The :mod:`xml.etree.ElementTree` package and its :mod:`xml.etree.cElementTree` 722counterpart have been updated to version 1.3. 723 724Several new and useful functions and methods have been added: 725 726* :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document 727 from a sequence of fragments 728* :func:`xml.etree.ElementTree.register_namespace` for registering a global 729 namespace prefix 730* :func:`xml.etree.ElementTree.tostringlist` for string representation 731 including all sublists 732* :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero 733 or more elements 734* :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and 735 subelements 736* :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over 737 an element and its subelements 738* :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element 739* :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype 740 declaration 741 742Two methods have been deprecated: 743 744* :meth:`xml.etree.ElementTree.getchildren` use ``list(elem)`` instead. 745* :meth:`xml.etree.ElementTree.getiterator` use ``Element.iter`` instead. 746 747For details of the update, see `Introducing ElementTree 748<https://web.archive.org/web/20200703234532/http://effbot.org/zone/elementtree-13-intro.htm>`_ 749on Fredrik Lundh's website. 750 751(Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.) 752 753functools 754--------- 755 756* The :mod:`functools` module includes a new decorator for caching function 757 calls. :func:`functools.lru_cache` can save repeated queries to an external 758 resource whenever the results are expected to be the same. 759 760 For example, adding a caching decorator to a database query function can save 761 database accesses for popular searches: 762 763 >>> import functools 764 >>> @functools.lru_cache(maxsize=300) 765 ... def get_phone_number(name): 766 ... c = conn.cursor() 767 ... c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,)) 768 ... return c.fetchone()[0] 769 770 >>> for name in user_requests: # doctest: +SKIP 771 ... get_phone_number(name) # cached lookup 772 773 To help with choosing an effective cache size, the wrapped function is 774 instrumented for tracking cache statistics: 775 776 >>> get_phone_number.cache_info() # doctest: +SKIP 777 CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300) 778 779 If the phonelist table gets updated, the outdated contents of the cache can be 780 cleared with: 781 782 >>> get_phone_number.cache_clear() 783 784 (Contributed by Raymond Hettinger and incorporating design ideas from Jim 785 Baker, Miki Tebeka, and Nick Coghlan; see `recipe 498245 786 <https://code.activestate.com/recipes/498245/>`_\, `recipe 577479 787 <https://code.activestate.com/recipes/577479/>`_\, :issue:`10586`, and 788 :issue:`10593`.) 789 790* The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute 791 pointing to the original callable function. This allows wrapped functions to 792 be introspected. It also copies :attr:`__annotations__` if defined. And now 793 it also gracefully skips over missing attributes such as :attr:`__doc__` which 794 might not be defined for the wrapped callable. 795 796 In the above example, the cache can be removed by recovering the original 797 function: 798 799 >>> get_phone_number = get_phone_number.__wrapped__ # uncached function 800 801 (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and 802 :issue:`8814`.) 803 804* To help write classes with rich comparison methods, a new decorator 805 :func:`functools.total_ordering` will use existing equality and inequality 806 methods to fill in the remaining methods. 807 808 For example, supplying *__eq__* and *__lt__* will enable 809 :func:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*:: 810 811 @total_ordering 812 class Student: 813 def __eq__(self, other): 814 return ((self.lastname.lower(), self.firstname.lower()) == 815 (other.lastname.lower(), other.firstname.lower())) 816 817 def __lt__(self, other): 818 return ((self.lastname.lower(), self.firstname.lower()) < 819 (other.lastname.lower(), other.firstname.lower())) 820 821 With the *total_ordering* decorator, the remaining comparison methods 822 are filled in automatically. 823 824 (Contributed by Raymond Hettinger.) 825 826* To aid in porting programs from Python 2, the :func:`functools.cmp_to_key` 827 function converts an old-style comparison function to 828 modern :term:`key function`: 829 830 >>> # locale-aware sort order 831 >>> sorted(iterable, key=cmp_to_key(locale.strcoll)) # doctest: +SKIP 832 833 For sorting examples and a brief sorting tutorial, see the `Sorting HowTo 834 <https://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial. 835 836 (Contributed by Raymond Hettinger.) 837 838itertools 839--------- 840 841* The :mod:`itertools` module has a new :func:`~itertools.accumulate` function 842 modeled on APL's *scan* operator and Numpy's *accumulate* function: 843 844 >>> from itertools import accumulate 845 >>> list(accumulate([8, 2, 50])) 846 [8, 10, 60] 847 848 >>> prob_dist = [0.1, 0.4, 0.2, 0.3] 849 >>> list(accumulate(prob_dist)) # cumulative probability distribution 850 [0.1, 0.5, 0.7, 1.0] 851 852 For an example using :func:`~itertools.accumulate`, see the :ref:`examples for 853 the random module <random-examples>`. 854 855 (Contributed by Raymond Hettinger and incorporating design suggestions 856 from Mark Dickinson.) 857 858collections 859----------- 860 861* The :class:`collections.Counter` class now has two forms of in-place 862 subtraction, the existing *-=* operator for `saturating subtraction 863 <https://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new 864 :meth:`~collections.Counter.subtract` method for regular subtraction. The 865 former is suitable for `multisets <https://en.wikipedia.org/wiki/Multiset>`_ 866 which only have positive counts, and the latter is more suitable for use cases 867 that allow negative counts: 868 869 >>> from collections import Counter 870 >>> tally = Counter(dogs=5, cats=3) 871 >>> tally -= Counter(dogs=2, cats=8) # saturating subtraction 872 >>> tally 873 Counter({'dogs': 3}) 874 875 >>> tally = Counter(dogs=5, cats=3) 876 >>> tally.subtract(dogs=2, cats=8) # regular subtraction 877 >>> tally 878 Counter({'dogs': 3, 'cats': -5}) 879 880 (Contributed by Raymond Hettinger.) 881 882* The :class:`collections.OrderedDict` class has a new method 883 :meth:`~collections.OrderedDict.move_to_end` which takes an existing key and 884 moves it to either the first or last position in the ordered sequence. 885 886 The default is to move an item to the last position. This is equivalent of 887 renewing an entry with ``od[k] = od.pop(k)``. 888 889 A fast move-to-end operation is useful for resequencing entries. For example, 890 an ordered dictionary can be used to track order of access by aging entries 891 from the oldest to the most recently accessed. 892 893 >>> from collections import OrderedDict 894 >>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e']) 895 >>> list(d) 896 ['a', 'b', 'X', 'd', 'e'] 897 >>> d.move_to_end('X') 898 >>> list(d) 899 ['a', 'b', 'd', 'e', 'X'] 900 901 (Contributed by Raymond Hettinger.) 902 903* The :class:`collections.deque` class grew two new methods 904 :meth:`~collections.deque.count` and :meth:`~collections.deque.reverse` that 905 make them more substitutable for :class:`list` objects: 906 907 >>> from collections import deque 908 >>> d = deque('simsalabim') 909 >>> d.count('s') 910 2 911 >>> d.reverse() 912 >>> d 913 deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's']) 914 915 (Contributed by Raymond Hettinger.) 916 917threading 918--------- 919 920The :mod:`threading` module has a new :class:`~threading.Barrier` 921synchronization class for making multiple threads wait until all of them have 922reached a common barrier point. Barriers are useful for making sure that a task 923with multiple preconditions does not run until all of the predecessor tasks are 924complete. 925 926Barriers can work with an arbitrary number of threads. This is a generalization 927of a `Rendezvous <https://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which 928is defined for only two threads. 929 930Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects 931are suitable for use in loops. The separate *filling* and *draining* phases 932assure that all threads get released (drained) before any one of them can loop 933back and re-enter the barrier. The barrier fully resets after each cycle. 934 935Example of using barriers:: 936 937 from threading import Barrier, Thread 938 939 def get_votes(site): 940 ballots = conduct_election(site) 941 all_polls_closed.wait() # do not count until all polls are closed 942 totals = summarize(ballots) 943 publish(site, totals) 944 945 all_polls_closed = Barrier(len(sites)) 946 for site in sites: 947 Thread(target=get_votes, args=(site,)).start() 948 949In this example, the barrier enforces a rule that votes cannot be counted at any 950polling site until all polls are closed. Notice how a solution with a barrier 951is similar to one with :meth:`threading.Thread.join`, but the threads stay alive 952and continue to do work (summarizing ballots) after the barrier point is 953crossed. 954 955If any of the predecessor tasks can hang or be delayed, a barrier can be created 956with an optional *timeout* parameter. Then if the timeout period elapses before 957all the predecessor tasks reach the barrier point, all waiting threads are 958released and a :exc:`~threading.BrokenBarrierError` exception is raised:: 959 960 def get_votes(site): 961 ballots = conduct_election(site) 962 try: 963 all_polls_closed.wait(timeout=midnight - time.now()) 964 except BrokenBarrierError: 965 lockbox = seal_ballots(ballots) 966 queue.put(lockbox) 967 else: 968 totals = summarize(ballots) 969 publish(site, totals) 970 971In this example, the barrier enforces a more robust rule. If some election 972sites do not finish before midnight, the barrier times-out and the ballots are 973sealed and deposited in a queue for later handling. 974 975See `Barrier Synchronization Patterns 976<https://osl.cs.illinois.edu/media/papers/karmani-2009-barrier_synchronization_pattern.pdf>`_ 977for more examples of how barriers can be used in parallel computing. Also, there is 978a simple but thorough explanation of barriers in `The Little Book of Semaphores 979<https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf>`_, *section 3.6*. 980 981(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in 982:issue:`8777`.) 983 984datetime and time 985----------------- 986 987* The :mod:`datetime` module has a new type :class:`~datetime.timezone` that 988 implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC 989 offset and timezone name. This makes it easier to create timezone-aware 990 datetime objects:: 991 992 >>> from datetime import datetime, timezone 993 994 >>> datetime.now(timezone.utc) 995 datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc) 996 997 >>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z") 998 datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc) 999 1000* Also, :class:`~datetime.timedelta` objects can now be multiplied by 1001 :class:`float` and divided by :class:`float` and :class:`int` objects. 1002 And :class:`~datetime.timedelta` objects can now divide one another. 1003 1004* The :meth:`datetime.date.strftime` method is no longer restricted to years 1005 after 1900. The new supported year range is from 1000 to 9999 inclusive. 1006 1007* Whenever a two-digit year is used in a time tuple, the interpretation has been 1008 governed by :attr:`time.accept2dyear`. The default is ``True`` which means that 1009 for a two-digit year, the century is guessed according to the POSIX rules 1010 governing the ``%y`` strptime format. 1011 1012 Starting with Py3.2, use of the century guessing heuristic will emit a 1013 :exc:`DeprecationWarning`. Instead, it is recommended that 1014 :attr:`time.accept2dyear` be set to ``False`` so that large date ranges 1015 can be used without guesswork:: 1016 1017 >>> import time, warnings 1018 >>> warnings.resetwarnings() # remove the default warning filters 1019 1020 >>> time.accept2dyear = True # guess whether 11 means 11 or 2011 1021 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) 1022 Warning (from warnings module): 1023 ... 1024 DeprecationWarning: Century info guessed for a 2-digit year. 1025 'Fri Jan 1 12:34:56 2011' 1026 1027 >>> time.accept2dyear = False # use the full range of allowable dates 1028 >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0)) 1029 'Fri Jan 1 12:34:56 11' 1030 1031 Several functions now have significantly expanded date ranges. When 1032 :attr:`time.accept2dyear` is false, the :func:`time.asctime` function will 1033 accept any year that fits in a C int, while the :func:`time.mktime` and 1034 :func:`time.strftime` functions will accept the full range supported by the 1035 corresponding operating system functions. 1036 1037(Contributed by Alexander Belopolsky and Victor Stinner in :issue:`1289118`, 1038:issue:`5094`, :issue:`6641`, :issue:`2706`, :issue:`1777412`, :issue:`8013`, 1039and :issue:`10827`.) 1040 1041.. XXX https://bugs.python.org/issue?%40search_text=datetime&%40sort=-activity 1042 1043math 1044---- 1045 1046The :mod:`math` module has been updated with six new functions inspired by the 1047C99 standard. 1048 1049The :func:`~math.isfinite` function provides a reliable and fast way to detect 1050special values. It returns ``True`` for regular numbers and ``False`` for *Nan* or 1051*Infinity*: 1052 1053>>> from math import isfinite 1054>>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))] 1055[True, True, False, False] 1056 1057The :func:`~math.expm1` function computes ``e**x-1`` for small values of *x* 1058without incurring the loss of precision that usually accompanies the subtraction 1059of nearly equal quantities: 1060 1061>>> from math import expm1 1062>>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x 10630.013765762467652909 1064 1065The :func:`~math.erf` function computes a probability integral or `Gaussian 1066error function <https://en.wikipedia.org/wiki/Error_function>`_. The 1067complementary error function, :func:`~math.erfc`, is ``1 - erf(x)``: 1068 1069.. doctest:: 1070 :options: +SKIP 1071 1072 >>> from math import erf, erfc, sqrt 1073 >>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation 1074 0.682689492137086 1075 >>> erfc(1.0/sqrt(2.0)) # portion of normal distribution outside 1 standard deviation 1076 0.31731050786291404 1077 >>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0)) 1078 1.0 1079 1080The :func:`~math.gamma` function is a continuous extension of the factorial 1081function. See https://en.wikipedia.org/wiki/Gamma_function for details. Because 1082the function is related to factorials, it grows large even for small values of 1083*x*, so there is also a :func:`~math.lgamma` function for computing the natural 1084logarithm of the gamma function: 1085 1086>>> from math import gamma, lgamma 1087>>> gamma(7.0) # six factorial 1088720.0 1089>>> lgamma(801.0) # log(800 factorial) 10904551.950730698041 1091 1092(Contributed by Mark Dickinson.) 1093 1094abc 1095--- 1096 1097The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and 1098:func:`~abc.abstractstaticmethod`. 1099 1100These tools make it possible to define an :term:`abstract base class` that 1101requires a particular :func:`classmethod` or :func:`staticmethod` to be 1102implemented:: 1103 1104 class Temperature(metaclass=abc.ABCMeta): 1105 @abc.abstractclassmethod 1106 def from_fahrenheit(cls, t): 1107 ... 1108 @abc.abstractclassmethod 1109 def from_celsius(cls, t): 1110 ... 1111 1112(Patch submitted by Daniel Urban; :issue:`5867`.) 1113 1114io 1115-- 1116 1117The :class:`io.BytesIO` has a new method, :meth:`~io.BytesIO.getbuffer`, which 1118provides functionality similar to :func:`memoryview`. It creates an editable 1119view of the data without making a copy. The buffer's random access and support 1120for slice notation are well-suited to in-place editing:: 1121 1122 >>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11 1123 1124 >>> def change_location(buffer, record_number, location): 1125 ... start = record_number * REC_LEN + LOC_START 1126 ... buffer[start: start+LOC_LEN] = location 1127 1128 >>> import io 1129 1130 >>> byte_stream = io.BytesIO( 1131 ... b'G3805 storeroom Main chassis ' 1132 ... b'X7899 shipping Reserve cog ' 1133 ... b'L6988 receiving Primary sprocket' 1134 ... ) 1135 >>> buffer = byte_stream.getbuffer() 1136 >>> change_location(buffer, 1, b'warehouse ') 1137 >>> change_location(buffer, 0, b'showroom ') 1138 >>> print(byte_stream.getvalue()) 1139 b'G3805 showroom Main chassis ' 1140 b'X7899 warehouse Reserve cog ' 1141 b'L6988 receiving Primary sprocket' 1142 1143(Contributed by Antoine Pitrou in :issue:`5506`.) 1144 1145reprlib 1146------- 1147 1148When writing a :meth:`__repr__` method for a custom container, it is easy to 1149forget to handle the case where a member refers back to the container itself. 1150Python's builtin objects such as :class:`list` and :class:`set` handle 1151self-reference by displaying "..." in the recursive part of the representation 1152string. 1153 1154To help write such :meth:`__repr__` methods, the :mod:`reprlib` module has a new 1155decorator, :func:`~reprlib.recursive_repr`, for detecting recursive calls to 1156:meth:`__repr__` and substituting a placeholder string instead:: 1157 1158 >>> class MyList(list): 1159 ... @recursive_repr() 1160 ... def __repr__(self): 1161 ... return '<' + '|'.join(map(repr, self)) + '>' 1162 ... 1163 >>> m = MyList('abc') 1164 >>> m.append(m) 1165 >>> m.append('x') 1166 >>> print(m) 1167 <'a'|'b'|'c'|...|'x'> 1168 1169(Contributed by Raymond Hettinger in :issue:`9826` and :issue:`9840`.) 1170 1171logging 1172------- 1173 1174In addition to dictionary-based configuration described above, the 1175:mod:`logging` package has many other improvements. 1176 1177The logging documentation has been augmented by a :ref:`basic tutorial 1178<logging-basic-tutorial>`\, an :ref:`advanced tutorial 1179<logging-advanced-tutorial>`\, and a :ref:`cookbook <logging-cookbook>` of 1180logging recipes. These documents are the fastest way to learn about logging. 1181 1182The :func:`logging.basicConfig` set-up function gained a *style* argument to 1183support three different types of string formatting. It defaults to "%" for 1184traditional %-formatting, can be set to "{" for the new :meth:`str.format` style, or 1185can be set to "$" for the shell-style formatting provided by 1186:class:`string.Template`. The following three configurations are equivalent:: 1187 1188 >>> from logging import basicConfig 1189 >>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s") 1190 >>> basicConfig(style='{', format="{name} -> {levelname} {message}") 1191 >>> basicConfig(style='$', format="$name -> $levelname: $message") 1192 1193If no configuration is set-up before a logging event occurs, there is now a 1194default configuration using a :class:`~logging.StreamHandler` directed to 1195:attr:`sys.stderr` for events of ``WARNING`` level or higher. Formerly, an 1196event occurring before a configuration was set-up would either raise an 1197exception or silently drop the event depending on the value of 1198:attr:`logging.raiseExceptions`. The new default handler is stored in 1199:attr:`logging.lastResort`. 1200 1201The use of filters has been simplified. Instead of creating a 1202:class:`~logging.Filter` object, the predicate can be any Python callable that 1203returns ``True`` or ``False``. 1204 1205There were a number of other improvements that add flexibility and simplify 1206configuration. See the module documentation for a full listing of changes in 1207Python 3.2. 1208 1209csv 1210--- 1211 1212The :mod:`csv` module now supports a new dialect, :class:`~csv.unix_dialect`, 1213which applies quoting for all fields and a traditional Unix style with ``'\n'`` as 1214the line terminator. The registered dialect name is ``unix``. 1215 1216The :class:`csv.DictWriter` has a new method, 1217:meth:`~csv.DictWriter.writeheader` for writing-out an initial row to document 1218the field names:: 1219 1220 >>> import csv, sys 1221 >>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix') 1222 >>> w.writeheader() 1223 "name","dept" 1224 >>> w.writerows([ 1225 ... {'name': 'tom', 'dept': 'accounting'}, 1226 ... {'name': 'susan', 'dept': 'Salesl'}]) 1227 "tom","accounting" 1228 "susan","sales" 1229 1230(New dialect suggested by Jay Talbot in :issue:`5975`, and the new method 1231suggested by Ed Abraham in :issue:`1537721`.) 1232 1233contextlib 1234---------- 1235 1236There is a new and slightly mind-blowing tool 1237:class:`~contextlib.ContextDecorator` that is helpful for creating a 1238:term:`context manager` that does double duty as a function decorator. 1239 1240As a convenience, this new functionality is used by 1241:func:`~contextlib.contextmanager` so that no extra effort is needed to support 1242both roles. 1243 1244The basic idea is that both context managers and function decorators can be used 1245for pre-action and post-action wrappers. Context managers wrap a group of 1246statements using a :keyword:`with` statement, and function decorators wrap a 1247group of statements enclosed in a function. So, occasionally there is a need to 1248write a pre-action or post-action wrapper that can be used in either role. 1249 1250For example, it is sometimes useful to wrap functions or groups of statements 1251with a logger that can track the time of entry and time of exit. Rather than 1252writing both a function decorator and a context manager for the task, the 1253:func:`~contextlib.contextmanager` provides both capabilities in a single 1254definition:: 1255 1256 from contextlib import contextmanager 1257 import logging 1258 1259 logging.basicConfig(level=logging.INFO) 1260 1261 @contextmanager 1262 def track_entry_and_exit(name): 1263 logging.info('Entering: %s', name) 1264 yield 1265 logging.info('Exiting: %s', name) 1266 1267Formerly, this would have only been usable as a context manager:: 1268 1269 with track_entry_and_exit('widget loader'): 1270 print('Some time consuming activity goes here') 1271 load_widget() 1272 1273Now, it can be used as a decorator as well:: 1274 1275 @track_entry_and_exit('widget loader') 1276 def activity(): 1277 print('Some time consuming activity goes here') 1278 load_widget() 1279 1280Trying to fulfill two roles at once places some limitations on the technique. 1281Context managers normally have the flexibility to return an argument usable by 1282a :keyword:`with` statement, but there is no parallel for function decorators. 1283 1284In the above example, there is not a clean way for the *track_entry_and_exit* 1285context manager to return a logging instance for use in the body of enclosed 1286statements. 1287 1288(Contributed by Michael Foord in :issue:`9110`.) 1289 1290decimal and fractions 1291--------------------- 1292 1293Mark Dickinson crafted an elegant and efficient scheme for assuring that 1294different numeric datatypes will have the same hash value whenever their actual 1295values are equal (:issue:`8188`):: 1296 1297 assert hash(Fraction(3, 2)) == hash(1.5) == \ 1298 hash(Decimal("1.5")) == hash(complex(1.5, 0)) 1299 1300Some of the hashing details are exposed through a new attribute, 1301:attr:`sys.hash_info`, which describes the bit width of the hash value, the 1302prime modulus, the hash values for *infinity* and *nan*, and the multiplier 1303used for the imaginary part of a number: 1304 1305>>> sys.hash_info # doctest: +SKIP 1306sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003) 1307 1308An early decision to limit the inter-operability of various numeric types has 1309been relaxed. It is still unsupported (and ill-advised) to have implicit 1310mixing in arithmetic expressions such as ``Decimal('1.1') + float('1.1')`` 1311because the latter loses information in the process of constructing the binary 1312float. However, since existing floating point value can be converted losslessly 1313to either a decimal or rational representation, it makes sense to add them to 1314the constructor and to support mixed-type comparisons. 1315 1316* The :class:`decimal.Decimal` constructor now accepts :class:`float` objects 1317 directly so there in no longer a need to use the :meth:`~decimal.Decimal.from_float` 1318 method (:issue:`8257`). 1319 1320* Mixed type comparisons are now fully supported so that 1321 :class:`~decimal.Decimal` objects can be directly compared with :class:`float` 1322 and :class:`fractions.Fraction` (:issue:`2531` and :issue:`8188`). 1323 1324Similar changes were made to :class:`fractions.Fraction` so that the 1325:meth:`~fractions.Fraction.from_float()` and :meth:`~fractions.Fraction.from_decimal` 1326methods are no longer needed (:issue:`8294`): 1327 1328>>> from decimal import Decimal 1329>>> from fractions import Fraction 1330>>> Decimal(1.1) 1331Decimal('1.100000000000000088817841970012523233890533447265625') 1332>>> Fraction(1.1) 1333Fraction(2476979795053773, 2251799813685248) 1334 1335Another useful change for the :mod:`decimal` module is that the 1336:attr:`Context.clamp` attribute is now public. This is useful in creating 1337contexts that correspond to the decimal interchange formats specified in IEEE 1338754 (see :issue:`8540`). 1339 1340(Contributed by Mark Dickinson and Raymond Hettinger.) 1341 1342ftp 1343--- 1344 1345The :class:`ftplib.FTP` class now supports the context management protocol to 1346unconditionally consume :exc:`socket.error` exceptions and to close the FTP 1347connection when done:: 1348 1349 >>> from ftplib import FTP 1350 >>> with FTP("ftp1.at.proftpd.org") as ftp: 1351 ftp.login() 1352 ftp.dir() 1353 1354 '230 Anonymous login ok, restrictions apply.' 1355 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 . 1356 dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .. 1357 dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS 1358 dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora 1359 1360Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input` 1361also grew auto-closing context managers:: 1362 1363 with fileinput.input(files=('log1.txt', 'log2.txt')) as f: 1364 for line in f: 1365 process(line) 1366 1367(Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and 1368by Georg Brandl in :issue:`8046` and :issue:`1286`.) 1369 1370The :class:`~ftplib.FTP_TLS` class now accepts a *context* parameter, which is a 1371:class:`ssl.SSLContext` object allowing bundling SSL configuration options, 1372certificates and private keys into a single (potentially long-lived) structure. 1373 1374(Contributed by Giampaolo Rodolà; :issue:`8806`.) 1375 1376popen 1377----- 1378 1379The :func:`os.popen` and :func:`subprocess.Popen` functions now support 1380:keyword:`with` statements for auto-closing of the file descriptors. 1381 1382(Contributed by Antoine Pitrou and Brian Curtin in :issue:`7461` and 1383:issue:`10554`.) 1384 1385select 1386------ 1387 1388The :mod:`select` module now exposes a new, constant attribute, 1389:attr:`~select.PIPE_BUF`, which gives the minimum number of bytes which are 1390guaranteed not to block when :func:`select.select` says a pipe is ready 1391for writing. 1392 1393>>> import select 1394>>> select.PIPE_BUF # doctest: +SKIP 1395512 1396 1397(Available on Unix systems. Patch by Sébastien Sablé in :issue:`9862`) 1398 1399gzip and zipfile 1400---------------- 1401 1402:class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase` 1403:term:`abstract base class` (except for ``truncate()``). It also has a 1404:meth:`~gzip.GzipFile.peek` method and supports unseekable as well as 1405zero-padded file objects. 1406 1407The :mod:`gzip` module also gains the :func:`~gzip.compress` and 1408:func:`~gzip.decompress` functions for easier in-memory compression and 1409decompression. Keep in mind that text needs to be encoded as :class:`bytes` 1410before compressing and decompressing: 1411 1412>>> import gzip 1413>>> s = 'Three shall be the number thou shalt count, ' 1414>>> s += 'and the number of the counting shall be three' 1415>>> b = s.encode() # convert to utf-8 1416>>> len(b) 141789 1418>>> c = gzip.compress(b) 1419>>> len(c) 142077 1421>>> gzip.decompress(c).decode()[:42] # decompress and convert to text 1422'Three shall be the number thou shalt count' 1423 1424(Contributed by Anand B. Pillai in :issue:`3488`; and by Antoine Pitrou, Nir 1425Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and 1426:issue:`2846`.) 1427 1428Also, the :class:`zipfile.ZipExtFile` class was reworked internally to represent 1429files stored inside an archive. The new implementation is significantly faster 1430and can be wrapped in an :class:`io.BufferedReader` object for more speedups. It 1431also solves an issue where interleaved calls to *read* and *readline* gave the 1432wrong results. 1433 1434(Patch submitted by Nir Aides in :issue:`7610`.) 1435 1436tarfile 1437------- 1438 1439The :class:`~tarfile.TarFile` class can now be used as a context manager. In 1440addition, its :meth:`~tarfile.TarFile.add` method has a new option, *filter*, 1441that controls which files are added to the archive and allows the file metadata 1442to be edited. 1443 1444The new *filter* option replaces the older, less flexible *exclude* parameter 1445which is now deprecated. If specified, the optional *filter* parameter needs to 1446be a :term:`keyword argument`. The user-supplied filter function accepts a 1447:class:`~tarfile.TarInfo` object and returns an updated 1448:class:`~tarfile.TarInfo` object, or if it wants the file to be excluded, the 1449function can return ``None``:: 1450 1451 >>> import tarfile, glob 1452 1453 >>> def myfilter(tarinfo): 1454 ... if tarinfo.isfile(): # only save real files 1455 ... tarinfo.uname = 'monty' # redact the user name 1456 ... return tarinfo 1457 1458 >>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf: 1459 ... for filename in glob.glob('*.txt'): 1460 ... tf.add(filename, filter=myfilter) 1461 ... tf.list() 1462 -rw-r--r-- monty/501 902 2011-01-26 17:59:11 annotations.txt 1463 -rw-r--r-- monty/501 123 2011-01-26 17:59:11 general_questions.txt 1464 -rw-r--r-- monty/501 3514 2011-01-26 17:59:11 prion.txt 1465 -rw-r--r-- monty/501 124 2011-01-26 17:59:11 py_todo.txt 1466 -rw-r--r-- monty/501 1399 2011-01-26 17:59:11 semaphore_notes.txt 1467 1468(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in :issue:`6856`.) 1469 1470hashlib 1471------- 1472 1473The :mod:`hashlib` module has two new constant attributes listing the hashing 1474algorithms guaranteed to be present in all implementations and those available 1475on the current implementation:: 1476 1477 >>> import hashlib 1478 1479 >>> hashlib.algorithms_guaranteed 1480 {'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'} 1481 1482 >>> hashlib.algorithms_available 1483 {'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256', 1484 'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2', 1485 'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224', 1486 'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'} 1487 1488(Suggested by Carl Chenet in :issue:`7418`.) 1489 1490ast 1491--- 1492 1493The :mod:`ast` module has a wonderful a general-purpose tool for safely 1494evaluating expression strings using the Python literal 1495syntax. The :func:`ast.literal_eval` function serves as a secure alternative to 1496the builtin :func:`eval` function which is easily abused. Python 3.2 adds 1497:class:`bytes` and :class:`set` literals to the list of supported types: 1498strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and ``None``. 1499 1500:: 1501 1502 >>> from ast import literal_eval 1503 1504 >>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}" 1505 >>> literal_eval(request) 1506 {'args': (2, 0.5), 'req': 3, 'func': 'pow'} 1507 1508 >>> request = "os.system('do something harmful')" 1509 >>> literal_eval(request) 1510 Traceback (most recent call last): 1511 ... 1512 ValueError: malformed node or string: <_ast.Call object at 0x101739a10> 1513 1514(Implemented by Benjamin Peterson and Georg Brandl.) 1515 1516os 1517-- 1518 1519Different operating systems use various encodings for filenames and environment 1520variables. The :mod:`os` module provides two new functions, 1521:func:`~os.fsencode` and :func:`~os.fsdecode`, for encoding and decoding 1522filenames: 1523 1524>>> import os 1525>>> filename = 'Sehenswürdigkeiten' 1526>>> os.fsencode(filename) 1527b'Sehensw\xc3\xbcrdigkeiten' 1528 1529Some operating systems allow direct access to encoded bytes in the 1530environment. If so, the :attr:`os.supports_bytes_environ` constant will be 1531true. 1532 1533For direct access to encoded environment variables (if available), 1534use the new :func:`os.getenvb` function or use :data:`os.environb` 1535which is a bytes version of :data:`os.environ`. 1536 1537(Contributed by Victor Stinner.) 1538 1539shutil 1540------ 1541 1542The :func:`shutil.copytree` function has two new options: 1543 1544* *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function 1545 copies a file pointed to by a symlink, not the symlink itself. This option 1546 will silence the error raised if the file doesn't exist. 1547 1548* *copy_function*: is a callable that will be used to copy files. 1549 :func:`shutil.copy2` is used by default. 1550 1551(Contributed by Tarek Ziadé.) 1552 1553In addition, the :mod:`shutil` module now supports :ref:`archiving operations 1554<archiving-operations>` for zipfiles, uncompressed tarfiles, gzipped tarfiles, 1555and bzipped tarfiles. And there are functions for registering additional 1556archiving file formats (such as xz compressed tarfiles or custom formats). 1557 1558The principal functions are :func:`~shutil.make_archive` and 1559:func:`~shutil.unpack_archive`. By default, both operate on the current 1560directory (which can be set by :func:`os.chdir`) and on any sub-directories. 1561The archive filename needs to be specified with a full pathname. The archiving 1562step is non-destructive (the original files are left unchanged). 1563 1564:: 1565 1566 >>> import shutil, pprint 1567 1568 >>> os.chdir('mydata') # change to the source directory 1569 >>> f = shutil.make_archive('/var/backup/mydata', 1570 ... 'zip') # archive the current directory 1571 >>> f # show the name of archive 1572 '/var/backup/mydata.zip' 1573 >>> os.chdir('tmp') # change to an unpacking 1574 >>> shutil.unpack_archive('/var/backup/mydata.zip') # recover the data 1575 1576 >>> pprint.pprint(shutil.get_archive_formats()) # display known formats 1577 [('bztar', "bzip2'ed tar-file"), 1578 ('gztar', "gzip'ed tar-file"), 1579 ('tar', 'uncompressed tar file'), 1580 ('zip', 'ZIP file')] 1581 1582 >>> shutil.register_archive_format( # register a new archive format 1583 ... name='xz', 1584 ... function=xz.compress, # callable archiving function 1585 ... extra_args=[('level', 8)], # arguments to the function 1586 ... description='xz compression' 1587 ... ) 1588 1589(Contributed by Tarek Ziadé.) 1590 1591sqlite3 1592------- 1593 1594The :mod:`sqlite3` module was updated to pysqlite version 2.6.0. It has two new capabilities. 1595 1596* The :attr:`sqlite3.Connection.in_transit` attribute is true if there is an 1597 active transaction for uncommitted changes. 1598 1599* The :meth:`sqlite3.Connection.enable_load_extension` and 1600 :meth:`sqlite3.Connection.load_extension` methods allows you to load SQLite 1601 extensions from ".so" files. One well-known extension is the fulltext-search 1602 extension distributed with SQLite. 1603 1604(Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.) 1605 1606html 1607---- 1608 1609A new :mod:`html` module was introduced with only a single function, 1610:func:`~html.escape`, which is used for escaping reserved characters from HTML 1611markup: 1612 1613>>> import html 1614>>> html.escape('x > 2 && x < 7') 1615'x > 2 && x < 7' 1616 1617socket 1618------ 1619 1620The :mod:`socket` module has two new improvements. 1621 1622* Socket objects now have a :meth:`~socket.socket.detach()` method which puts 1623 the socket into closed state without actually closing the underlying file 1624 descriptor. The latter can then be reused for other purposes. 1625 (Added by Antoine Pitrou; :issue:`8524`.) 1626 1627* :func:`socket.create_connection` now supports the context management protocol 1628 to unconditionally consume :exc:`socket.error` exceptions and to close the 1629 socket when done. 1630 (Contributed by Giampaolo Rodolà; :issue:`9794`.) 1631 1632ssl 1633--- 1634 1635The :mod:`ssl` module added a number of features to satisfy common requirements 1636for secure (encrypted, authenticated) internet connections: 1637 1638* A new class, :class:`~ssl.SSLContext`, serves as a container for persistent 1639 SSL data, such as protocol settings, certificates, private keys, and various 1640 other options. It includes a :meth:`~ssl.SSLContext.wrap_socket` for creating 1641 an SSL socket from an SSL context. 1642 1643* A new function, :func:`ssl.match_hostname`, supports server identity 1644 verification for higher-level protocols by implementing the rules of HTTPS 1645 (from :rfc:`2818`) which are also suitable for other protocols. 1646 1647* The :func:`ssl.wrap_socket` constructor function now takes a *ciphers* 1648 argument. The *ciphers* string lists the allowed encryption algorithms using 1649 the format described in the `OpenSSL documentation 1650 <https://www.openssl.org/docs/man1.0.2/man1/ciphers.html#CIPHER-LIST-FORMAT>`__. 1651 1652* When linked against recent versions of OpenSSL, the :mod:`ssl` module now 1653 supports the Server Name Indication extension to the TLS protocol, allowing 1654 multiple "virtual hosts" using different certificates on a single IP port. 1655 This extension is only supported in client mode, and is activated by passing 1656 the *server_hostname* argument to :meth:`ssl.SSLContext.wrap_socket`. 1657 1658* Various options have been added to the :mod:`ssl` module, such as 1659 :data:`~ssl.OP_NO_SSLv2` which disables the insecure and obsolete SSLv2 1660 protocol. 1661 1662* The extension now loads all the OpenSSL ciphers and digest algorithms. If 1663 some SSL certificates cannot be verified, they are reported as an "unknown 1664 algorithm" error. 1665 1666* The version of OpenSSL being used is now accessible using the module 1667 attributes :data:`ssl.OPENSSL_VERSION` (a string), 1668 :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and 1669 :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). 1670 1671(Contributed by Antoine Pitrou in :issue:`8850`, :issue:`1589`, :issue:`8322`, 1672:issue:`5639`, :issue:`4870`, :issue:`8484`, and :issue:`8321`.) 1673 1674nntp 1675---- 1676 1677The :mod:`nntplib` module has a revamped implementation with better bytes and 1678text semantics as well as more practical APIs. These improvements break 1679compatibility with the nntplib version in Python 3.1, which was partly 1680dysfunctional in itself. 1681 1682Support for secure connections through both implicit (using 1683:class:`nntplib.NNTP_SSL`) and explicit (using :meth:`nntplib.NNTP.starttls`) 1684TLS has also been added. 1685 1686(Contributed by Antoine Pitrou in :issue:`9360` and Andrew Vant in :issue:`1926`.) 1687 1688certificates 1689------------ 1690 1691:class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler` 1692and :func:`urllib.request.urlopen` now take optional arguments to allow for 1693server certificate checking against a set of Certificate Authorities, 1694as recommended in public uses of HTTPS. 1695 1696(Added by Antoine Pitrou, :issue:`9003`.) 1697 1698imaplib 1699------- 1700 1701Support for explicit TLS on standard IMAP4 connections has been added through 1702the new :mod:`imaplib.IMAP4.starttls` method. 1703 1704(Contributed by Lorenzo M. Catucci and Antoine Pitrou, :issue:`4471`.) 1705 1706http.client 1707----------- 1708 1709There were a number of small API improvements in the :mod:`http.client` module. 1710The old-style HTTP 0.9 simple responses are no longer supported and the *strict* 1711parameter is deprecated in all classes. 1712 1713The :class:`~http.client.HTTPConnection` and 1714:class:`~http.client.HTTPSConnection` classes now have a *source_address* 1715parameter for a (host, port) tuple indicating where the HTTP connection is made 1716from. 1717 1718Support for certificate checking and HTTPS virtual hosts were added to 1719:class:`~http.client.HTTPSConnection`. 1720 1721The :meth:`~http.client.HTTPConnection.request` method on connection objects 1722allowed an optional *body* argument so that a :term:`file object` could be used 1723to supply the content of the request. Conveniently, the *body* argument now 1724also accepts an :term:`iterable` object so long as it includes an explicit 1725``Content-Length`` header. This extended interface is much more flexible than 1726before. 1727 1728To establish an HTTPS connection through a proxy server, there is a new 1729:meth:`~http.client.HTTPConnection.set_tunnel` method that sets the host and 1730port for HTTP Connect tunneling. 1731 1732To match the behavior of :mod:`http.server`, the HTTP client library now also 1733encodes headers with ISO-8859-1 (Latin-1) encoding. It was already doing that 1734for incoming headers, so now the behavior is consistent for both incoming and 1735outgoing traffic. (See work by Armin Ronacher in :issue:`10980`.) 1736 1737unittest 1738-------- 1739 1740The unittest module has a number of improvements supporting test discovery for 1741packages, easier experimentation at the interactive prompt, new testcase 1742methods, improved diagnostic messages for test failures, and better method 1743names. 1744 1745* The command-line call ``python -m unittest`` can now accept file paths 1746 instead of module names for running specific tests (:issue:`10620`). The new 1747 test discovery can find tests within packages, locating any test importable 1748 from the top-level directory. The top-level directory can be specified with 1749 the ``-t`` option, a pattern for matching files with ``-p``, and a directory to 1750 start discovery with ``-s``: 1751 1752 .. code-block:: shell-session 1753 1754 $ python -m unittest discover -s my_proj_dir -p _test.py 1755 1756 (Contributed by Michael Foord.) 1757 1758* Experimentation at the interactive prompt is now easier because the 1759 :class:`unittest.case.TestCase` class can now be instantiated without 1760 arguments: 1761 1762 >>> from unittest import TestCase 1763 >>> TestCase().assertEqual(pow(2, 3), 8) 1764 1765 (Contributed by Michael Foord.) 1766 1767* The :mod:`unittest` module has two new methods, 1768 :meth:`~unittest.TestCase.assertWarns` and 1769 :meth:`~unittest.TestCase.assertWarnsRegex` to verify that a given warning type 1770 is triggered by the code under test:: 1771 1772 with self.assertWarns(DeprecationWarning): 1773 legacy_function('XYZ') 1774 1775 (Contributed by Antoine Pitrou, :issue:`9754`.) 1776 1777 Another new method, :meth:`~unittest.TestCase.assertCountEqual` is used to 1778 compare two iterables to determine if their element counts are equal (whether 1779 the same elements are present with the same number of occurrences regardless 1780 of order):: 1781 1782 def test_anagram(self): 1783 self.assertCountEqual('algorithm', 'logarithm') 1784 1785 (Contributed by Raymond Hettinger.) 1786 1787* A principal feature of the unittest module is an effort to produce meaningful 1788 diagnostics when a test fails. When possible, the failure is recorded along 1789 with a diff of the output. This is especially helpful for analyzing log files 1790 of failed test runs. However, since diffs can sometime be voluminous, there is 1791 a new :attr:`~unittest.TestCase.maxDiff` attribute that sets maximum length of 1792 diffs displayed. 1793 1794* In addition, the method names in the module have undergone a number of clean-ups. 1795 1796 For example, :meth:`~unittest.TestCase.assertRegex` is the new name for 1797 :meth:`~unittest.TestCase.assertRegexpMatches` which was misnamed because the 1798 test uses :func:`re.search`, not :func:`re.match`. Other methods using 1799 regular expressions are now named using short form "Regex" in preference to 1800 "Regexp" -- this matches the names used in other unittest implementations, 1801 matches Python's old name for the :mod:`re` module, and it has unambiguous 1802 camel-casing. 1803 1804 (Contributed by Raymond Hettinger and implemented by Ezio Melotti.) 1805 1806* To improve consistency, some long-standing method aliases are being 1807 deprecated in favor of the preferred names: 1808 1809 =============================== ============================== 1810 Old Name Preferred Name 1811 =============================== ============================== 1812 :meth:`assert_` :meth:`.assertTrue` 1813 :meth:`assertEquals` :meth:`.assertEqual` 1814 :meth:`assertNotEquals` :meth:`.assertNotEqual` 1815 :meth:`assertAlmostEquals` :meth:`.assertAlmostEqual` 1816 :meth:`assertNotAlmostEquals` :meth:`.assertNotAlmostEqual` 1817 =============================== ============================== 1818 1819 Likewise, the ``TestCase.fail*`` methods deprecated in Python 3.1 are expected 1820 to be removed in Python 3.3. Also see the :ref:`deprecated-aliases` section in 1821 the :mod:`unittest` documentation. 1822 1823 (Contributed by Ezio Melotti; :issue:`9424`.) 1824 1825* The :meth:`~unittest.TestCase.assertDictContainsSubset` method was deprecated 1826 because it was misimplemented with the arguments in the wrong order. This 1827 created hard-to-debug optical illusions where tests like 1828 ``TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1})`` would fail. 1829 1830 (Contributed by Raymond Hettinger.) 1831 1832random 1833------ 1834 1835The integer methods in the :mod:`random` module now do a better job of producing 1836uniform distributions. Previously, they computed selections with 1837``int(n*random())`` which had a slight bias whenever *n* was not a power of two. 1838Now, multiple selections are made from a range up to the next power of two and a 1839selection is kept only when it falls within the range ``0 <= x < n``. The 1840functions and methods affected are :func:`~random.randrange`, 1841:func:`~random.randint`, :func:`~random.choice`, :func:`~random.shuffle` and 1842:func:`~random.sample`. 1843 1844(Contributed by Raymond Hettinger; :issue:`9025`.) 1845 1846poplib 1847------ 1848 1849:class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a 1850:class:`ssl.SSLContext` object allowing bundling SSL configuration options, 1851certificates and private keys into a single (potentially long-lived) 1852structure. 1853 1854(Contributed by Giampaolo Rodolà; :issue:`8807`.) 1855 1856asyncore 1857-------- 1858 1859:class:`asyncore.dispatcher` now provides a 1860:meth:`~asyncore.dispatcher.handle_accepted()` method 1861returning a ``(sock, addr)`` pair which is called when a connection has actually 1862been established with a new remote endpoint. This is supposed to be used as a 1863replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids 1864the user to call :meth:`~asyncore.dispatcher.accept()` directly. 1865 1866(Contributed by Giampaolo Rodolà; :issue:`6706`.) 1867 1868tempfile 1869-------- 1870 1871The :mod:`tempfile` module has a new context manager, 1872:class:`~tempfile.TemporaryDirectory` which provides easy deterministic 1873cleanup of temporary directories:: 1874 1875 with tempfile.TemporaryDirectory() as tmpdirname: 1876 print('created temporary dir:', tmpdirname) 1877 1878(Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.) 1879 1880inspect 1881------- 1882 1883* The :mod:`inspect` module has a new function 1884 :func:`~inspect.getgeneratorstate` to easily identify the current state of a 1885 generator-iterator:: 1886 1887 >>> from inspect import getgeneratorstate 1888 >>> def gen(): 1889 ... yield 'demo' 1890 >>> g = gen() 1891 >>> getgeneratorstate(g) 1892 'GEN_CREATED' 1893 >>> next(g) 1894 'demo' 1895 >>> getgeneratorstate(g) 1896 'GEN_SUSPENDED' 1897 >>> next(g, None) 1898 >>> getgeneratorstate(g) 1899 'GEN_CLOSED' 1900 1901 (Contributed by Rodolpho Eckhardt and Nick Coghlan, :issue:`10220`.) 1902 1903* To support lookups without the possibility of activating a dynamic attribute, 1904 the :mod:`inspect` module has a new function, :func:`~inspect.getattr_static`. 1905 Unlike :func:`hasattr`, this is a true read-only search, guaranteed not to 1906 change state while it is searching:: 1907 1908 >>> class A: 1909 ... @property 1910 ... def f(self): 1911 ... print('Running') 1912 ... return 10 1913 ... 1914 >>> a = A() 1915 >>> getattr(a, 'f') 1916 Running 1917 10 1918 >>> inspect.getattr_static(a, 'f') 1919 <property object at 0x1022bd788> 1920 1921 (Contributed by Michael Foord.) 1922 1923pydoc 1924----- 1925 1926The :mod:`pydoc` module now provides a much-improved web server interface, as 1927well as a new command-line option ``-b`` to automatically open a browser window 1928to display that server: 1929 1930.. code-block:: shell-session 1931 1932 $ pydoc3.2 -b 1933 1934(Contributed by Ron Adam; :issue:`2001`.) 1935 1936dis 1937--- 1938 1939The :mod:`dis` module gained two new functions for inspecting code, 1940:func:`~dis.code_info` and :func:`~dis.show_code`. Both provide detailed code 1941object information for the supplied function, method, source code string or code 1942object. The former returns a string and the latter prints it:: 1943 1944 >>> import dis, random 1945 >>> dis.show_code(random.choice) 1946 Name: choice 1947 Filename: /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py 1948 Argument count: 2 1949 Kw-only arguments: 0 1950 Number of locals: 3 1951 Stack size: 11 1952 Flags: OPTIMIZED, NEWLOCALS, NOFREE 1953 Constants: 1954 0: 'Choose a random element from a non-empty sequence.' 1955 1: 'Cannot choose from an empty sequence' 1956 Names: 1957 0: _randbelow 1958 1: len 1959 2: ValueError 1960 3: IndexError 1961 Variable names: 1962 0: self 1963 1: seq 1964 2: i 1965 1966In addition, the :func:`~dis.dis` function now accepts string arguments 1967so that the common idiom ``dis(compile(s, '', 'eval'))`` can be shortened 1968to ``dis(s)``:: 1969 1970 >>> dis('3*x+1 if x%2==1 else x//2') 1971 1 0 LOAD_NAME 0 (x) 1972 3 LOAD_CONST 0 (2) 1973 6 BINARY_MODULO 1974 7 LOAD_CONST 1 (1) 1975 10 COMPARE_OP 2 (==) 1976 13 POP_JUMP_IF_FALSE 28 1977 16 LOAD_CONST 2 (3) 1978 19 LOAD_NAME 0 (x) 1979 22 BINARY_MULTIPLY 1980 23 LOAD_CONST 1 (1) 1981 26 BINARY_ADD 1982 27 RETURN_VALUE 1983 >> 28 LOAD_NAME 0 (x) 1984 31 LOAD_CONST 0 (2) 1985 34 BINARY_FLOOR_DIVIDE 1986 35 RETURN_VALUE 1987 1988Taken together, these improvements make it easier to explore how CPython is 1989implemented and to see for yourself what the language syntax does 1990under-the-hood. 1991 1992(Contributed by Nick Coghlan in :issue:`9147`.) 1993 1994dbm 1995--- 1996 1997All database modules now support the :meth:`get` and :meth:`setdefault` methods. 1998 1999(Suggested by Ray Allen in :issue:`9523`.) 2000 2001ctypes 2002------ 2003 2004A new type, :class:`ctypes.c_ssize_t` represents the C :c:type:`ssize_t` datatype. 2005 2006site 2007---- 2008 2009The :mod:`site` module has three new functions useful for reporting on the 2010details of a given Python installation. 2011 2012* :func:`~site.getsitepackages` lists all global site-packages directories. 2013 2014* :func:`~site.getuserbase` reports on the user's base directory where data can 2015 be stored. 2016 2017* :func:`~site.getusersitepackages` reveals the user-specific site-packages 2018 directory path. 2019 2020:: 2021 2022 >>> import site 2023 >>> site.getsitepackages() 2024 ['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages', 2025 '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python', 2026 '/Library/Python/3.2/site-packages'] 2027 >>> site.getuserbase() 2028 '/Users/raymondhettinger/Library/Python/3.2' 2029 >>> site.getusersitepackages() 2030 '/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages' 2031 2032Conveniently, some of site's functionality is accessible directly from the 2033command-line: 2034 2035.. code-block:: shell-session 2036 2037 $ python -m site --user-base 2038 /Users/raymondhettinger/.local 2039 $ python -m site --user-site 2040 /Users/raymondhettinger/.local/lib/python3.2/site-packages 2041 2042(Contributed by Tarek Ziadé in :issue:`6693`.) 2043 2044sysconfig 2045--------- 2046 2047The new :mod:`sysconfig` module makes it straightforward to discover 2048installation paths and configuration variables that vary across platforms and 2049installations. 2050 2051The module offers access simple access functions for platform and version 2052information: 2053 2054* :func:`~sysconfig.get_platform` returning values like *linux-i586* or 2055 *macosx-10.6-ppc*. 2056* :func:`~sysconfig.get_python_version` returns a Python version string 2057 such as "3.2". 2058 2059It also provides access to the paths and variables corresponding to one of 2060seven named schemes used by :mod:`distutils`. Those include *posix_prefix*, 2061*posix_home*, *posix_user*, *nt*, *nt_user*, *os2*, *os2_home*: 2062 2063* :func:`~sysconfig.get_paths` makes a dictionary containing installation paths 2064 for the current installation scheme. 2065* :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific 2066 variables. 2067 2068There is also a convenient command-line interface: 2069 2070.. code-block:: doscon 2071 2072 C:\Python32>python -m sysconfig 2073 Platform: "win32" 2074 Python version: "3.2" 2075 Current installation scheme: "nt" 2076 2077 Paths: 2078 data = "C:\Python32" 2079 include = "C:\Python32\Include" 2080 platinclude = "C:\Python32\Include" 2081 platlib = "C:\Python32\Lib\site-packages" 2082 platstdlib = "C:\Python32\Lib" 2083 purelib = "C:\Python32\Lib\site-packages" 2084 scripts = "C:\Python32\Scripts" 2085 stdlib = "C:\Python32\Lib" 2086 2087 Variables: 2088 BINDIR = "C:\Python32" 2089 BINLIBDEST = "C:\Python32\Lib" 2090 EXE = ".exe" 2091 INCLUDEPY = "C:\Python32\Include" 2092 LIBDEST = "C:\Python32\Lib" 2093 SO = ".pyd" 2094 VERSION = "32" 2095 abiflags = "" 2096 base = "C:\Python32" 2097 exec_prefix = "C:\Python32" 2098 platbase = "C:\Python32" 2099 prefix = "C:\Python32" 2100 projectbase = "C:\Python32" 2101 py_version = "3.2" 2102 py_version_nodot = "32" 2103 py_version_short = "3.2" 2104 srcdir = "C:\Python32" 2105 userbase = "C:\Documents and Settings\Raymond\Application Data\Python" 2106 2107(Moved out of Distutils by Tarek Ziadé.) 2108 2109pdb 2110--- 2111 2112The :mod:`pdb` debugger module gained a number of usability improvements: 2113 2114* :file:`pdb.py` now has a ``-c`` option that executes commands as given in a 2115 :file:`.pdbrc` script file. 2116* A :file:`.pdbrc` script file can contain ``continue`` and ``next`` commands 2117 that continue debugging. 2118* The :class:`Pdb` class constructor now accepts a *nosigint* argument. 2119* New commands: ``l(list)``, ``ll(long list)`` and ``source`` for 2120 listing source code. 2121* New commands: ``display`` and ``undisplay`` for showing or hiding 2122 the value of an expression if it has changed. 2123* New command: ``interact`` for starting an interactive interpreter containing 2124 the global and local names found in the current scope. 2125* Breakpoints can be cleared by breakpoint number. 2126 2127(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.) 2128 2129configparser 2130------------ 2131 2132The :mod:`configparser` module was modified to improve usability and 2133predictability of the default parser and its supported INI syntax. The old 2134:class:`ConfigParser` class was removed in favor of :class:`SafeConfigParser` 2135which has in turn been renamed to :class:`~configparser.ConfigParser`. Support 2136for inline comments is now turned off by default and section or option 2137duplicates are not allowed in a single configuration source. 2138 2139Config parsers gained a new API based on the mapping protocol:: 2140 2141 >>> parser = ConfigParser() 2142 >>> parser.read_string(""" 2143 ... [DEFAULT] 2144 ... location = upper left 2145 ... visible = yes 2146 ... editable = no 2147 ... color = blue 2148 ... 2149 ... [main] 2150 ... title = Main Menu 2151 ... color = green 2152 ... 2153 ... [options] 2154 ... title = Options 2155 ... """) 2156 >>> parser['main']['color'] 2157 'green' 2158 >>> parser['main']['editable'] 2159 'no' 2160 >>> section = parser['options'] 2161 >>> section['title'] 2162 'Options' 2163 >>> section['title'] = 'Options (editable: %(editable)s)' 2164 >>> section['title'] 2165 'Options (editable: no)' 2166 2167The new API is implemented on top of the classical API, so custom parser 2168subclasses should be able to use it without modifications. 2169 2170The INI file structure accepted by config parsers can now be customized. Users 2171can specify alternative option/value delimiters and comment prefixes, change the 2172name of the *DEFAULT* section or switch the interpolation syntax. 2173 2174There is support for pluggable interpolation including an additional interpolation 2175handler :class:`~configparser.ExtendedInterpolation`:: 2176 2177 >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) 2178 >>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'}, 2179 ... 'custom': {'prefix': '/usr/local'}}) 2180 >>> parser.read_string(""" 2181 ... [buildout] 2182 ... parts = 2183 ... zope9 2184 ... instance 2185 ... find-links = 2186 ... ${buildout:directory}/downloads/dist 2187 ... 2188 ... [zope9] 2189 ... recipe = plone.recipe.zope9install 2190 ... location = /opt/zope 2191 ... 2192 ... [instance] 2193 ... recipe = plone.recipe.zope9instance 2194 ... zope9-location = ${zope9:location} 2195 ... zope-conf = ${custom:prefix}/etc/zope.conf 2196 ... """) 2197 >>> parser['buildout']['find-links'] 2198 '\n/home/ambv/zope9/downloads/dist' 2199 >>> parser['instance']['zope-conf'] 2200 '/usr/local/etc/zope.conf' 2201 >>> instance = parser['instance'] 2202 >>> instance['zope-conf'] 2203 '/usr/local/etc/zope.conf' 2204 >>> instance['zope9-location'] 2205 '/opt/zope' 2206 2207A number of smaller features were also introduced, like support for specifying 2208encoding in read operations, specifying fallback values for get-functions, or 2209reading directly from dictionaries and strings. 2210 2211(All changes contributed by Łukasz Langa.) 2212 2213.. XXX consider showing a difflib example 2214 2215urllib.parse 2216------------ 2217 2218A number of usability improvements were made for the :mod:`urllib.parse` module. 2219 2220The :func:`~urllib.parse.urlparse` function now supports `IPv6 2221<https://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`: 2222 2223 >>> import urllib.parse 2224 >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') # doctest: +NORMALIZE_WHITESPACE 2225 ParseResult(scheme='http', 2226 netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]', 2227 path='/foo/', 2228 params='', 2229 query='', 2230 fragment='') 2231 2232The :func:`~urllib.parse.urldefrag` function now returns a :term:`named tuple`:: 2233 2234 >>> r = urllib.parse.urldefrag('http://python.org/about/#target') 2235 >>> r 2236 DefragResult(url='http://python.org/about/', fragment='target') 2237 >>> r[0] 2238 'http://python.org/about/' 2239 >>> r.fragment 2240 'target' 2241 2242And, the :func:`~urllib.parse.urlencode` function is now much more flexible, 2243accepting either a string or bytes type for the *query* argument. If it is a 2244string, then the *safe*, *encoding*, and *error* parameters are sent to 2245:func:`~urllib.parse.quote_plus` for encoding:: 2246 2247 >>> urllib.parse.urlencode([ 2248 ... ('type', 'telenovela'), 2249 ... ('name', '¿Dónde Está Elisa?')], 2250 ... encoding='latin-1') 2251 'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F' 2252 2253As detailed in :ref:`parsing-ascii-encoded-bytes`, all the :mod:`urllib.parse` 2254functions now accept ASCII-encoded byte strings as input, so long as they are 2255not mixed with regular strings. If ASCII-encoded byte strings are given as 2256parameters, the return types will also be an ASCII-encoded byte strings: 2257 2258 >>> urllib.parse.urlparse(b'http://www.python.org:80/about/') # doctest: +NORMALIZE_WHITESPACE 2259 ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80', 2260 path=b'/about/', params=b'', query=b'', fragment=b'') 2261 2262(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in :issue:`2987`, 2263:issue:`5468`, and :issue:`9873`.) 2264 2265mailbox 2266------- 2267 2268Thanks to a concerted effort by R. David Murray, the :mod:`mailbox` module has 2269been fixed for Python 3.2. The challenge was that mailbox had been originally 2270designed with a text interface, but email messages are best represented with 2271:class:`bytes` because various parts of a message may have different encodings. 2272 2273The solution harnessed the :mod:`email` package's binary support for parsing 2274arbitrary email messages. In addition, the solution required a number of API 2275changes. 2276 2277As expected, the :meth:`~mailbox.Mailbox.add` method for 2278:class:`mailbox.Mailbox` objects now accepts binary input. 2279 2280:class:`~io.StringIO` and text file input are deprecated. Also, string input 2281will fail early if non-ASCII characters are used. Previously it would fail when 2282the email was processed in a later step. 2283 2284There is also support for binary output. The :meth:`~mailbox.Mailbox.get_file` 2285method now returns a file in the binary mode (where it used to incorrectly set 2286the file to text-mode). There is also a new :meth:`~mailbox.Mailbox.get_bytes` 2287method that returns a :class:`bytes` representation of a message corresponding 2288to a given *key*. 2289 2290It is still possible to get non-binary output using the old API's 2291:meth:`~mailbox.Mailbox.get_string` method, but that approach 2292is not very useful. Instead, it is best to extract messages from 2293a :class:`~mailbox.Message` object or to load them from binary input. 2294 2295(Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an 2296initial patch by Victor Stinner in :issue:`9124`.) 2297 2298turtledemo 2299---------- 2300 2301The demonstration code for the :mod:`turtle` module was moved from the *Demo* 2302directory to main library. It includes over a dozen sample scripts with 2303lively displays. Being on :attr:`sys.path`, it can now be run directly 2304from the command-line: 2305 2306.. code-block:: shell-session 2307 2308 $ python -m turtledemo 2309 2310(Moved from the Demo directory by Alexander Belopolsky in :issue:`10199`.) 2311 2312Multi-threading 2313=============== 2314 2315* The mechanism for serializing execution of concurrently running Python threads 2316 (generally known as the :term:`GIL` or Global Interpreter Lock) has 2317 been rewritten. Among the objectives were more predictable switching 2318 intervals and reduced overhead due to lock contention and the number of 2319 ensuing system calls. The notion of a "check interval" to allow thread 2320 switches has been abandoned and replaced by an absolute duration expressed in 2321 seconds. This parameter is tunable through :func:`sys.setswitchinterval()`. 2322 It currently defaults to 5 milliseconds. 2323 2324 Additional details about the implementation can be read from a `python-dev 2325 mailing-list message 2326 <https://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_ 2327 (however, "priority requests" as exposed in this message have not been kept 2328 for inclusion). 2329 2330 (Contributed by Antoine Pitrou.) 2331 2332* Regular and recursive locks now accept an optional *timeout* argument to their 2333 :meth:`~threading.Lock.acquire` method. (Contributed by Antoine Pitrou; 2334 :issue:`7316`.) 2335 2336* Similarly, :meth:`threading.Semaphore.acquire` also gained a *timeout* 2337 argument. (Contributed by Torsten Landschoff; :issue:`850728`.) 2338 2339* Regular and recursive lock acquisitions can now be interrupted by signals on 2340 platforms using Pthreads. This means that Python programs that deadlock while 2341 acquiring locks can be successfully killed by repeatedly sending SIGINT to the 2342 process (by pressing :kbd:`Ctrl+C` in most shells). 2343 (Contributed by Reid Kleckner; :issue:`8844`.) 2344 2345 2346Optimizations 2347============= 2348 2349A number of small performance enhancements have been added: 2350 2351* Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as 2352 being a test for membership in a set of constants. The optimizer recasts the 2353 :class:`set` as a :class:`frozenset` and stores the pre-built constant. 2354 2355 Now that the speed penalty is gone, it is practical to start writing 2356 membership tests using set-notation. This style is both semantically clear 2357 and operationally fast:: 2358 2359 extension = name.rpartition('.')[2] 2360 if extension in {'xml', 'html', 'xhtml', 'css'}: 2361 handle(name) 2362 2363 (Patch and additional tests contributed by Dave Malcolm; :issue:`6690`). 2364 2365* Serializing and unserializing data using the :mod:`pickle` module is now 2366 several times faster. 2367 2368 (Contributed by Alexandre Vassalotti, Antoine Pitrou 2369 and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.) 2370 2371* The `Timsort algorithm <https://en.wikipedia.org/wiki/Timsort>`_ used in 2372 :meth:`list.sort` and :func:`sorted` now runs faster and uses less memory 2373 when called with a :term:`key function`. Previously, every element of 2374 a list was wrapped with a temporary object that remembered the key value 2375 associated with each element. Now, two arrays of keys and values are 2376 sorted in parallel. This saves the memory consumed by the sort wrappers, 2377 and it saves time lost to delegating comparisons. 2378 2379 (Patch by Daniel Stutzbach in :issue:`9915`.) 2380 2381* JSON decoding performance is improved and memory consumption is reduced 2382 whenever the same string is repeated for multiple keys. Also, JSON encoding 2383 now uses the C speedups when the ``sort_keys`` argument is true. 2384 2385 (Contributed by Antoine Pitrou in :issue:`7451` and by Raymond Hettinger and 2386 Antoine Pitrou in :issue:`10314`.) 2387 2388* Recursive locks (created with the :func:`threading.RLock` API) now benefit 2389 from a C implementation which makes them as fast as regular locks, and between 2390 10x and 15x faster than their previous pure Python implementation. 2391 2392 (Contributed by Antoine Pitrou; :issue:`3001`.) 2393 2394* The fast-search algorithm in stringlib is now used by the :meth:`split`, 2395 :meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on 2396 :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the 2397 algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and 2398 :meth:`rpartition`. 2399 2400 (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.) 2401 2402 2403* Integer to string conversions now work two "digits" at a time, reducing the 2404 number of division and modulo operations. 2405 2406 (:issue:`6713` by Gawain Bolton, Mark Dickinson, and Victor Stinner.) 2407 2408There were several other minor optimizations. Set differencing now runs faster 2409when one operand is much larger than the other (patch by Andress Bennetts in 2410:issue:`8685`). The :meth:`array.repeat` method has a faster implementation 2411(:issue:`1569291` by Alexander Belopolsky). The :class:`BaseHTTPRequestHandler` 2412has more efficient buffering (:issue:`3709` by Andrew Schaaf). The 2413:func:`operator.attrgetter` function has been sped-up (:issue:`10160` by 2414Christos Georgiou). And :class:`ConfigParser` loads multi-line arguments a bit 2415faster (:issue:`7113` by Łukasz Langa). 2416 2417 2418Unicode 2419======= 2420 2421Python has been updated to `Unicode 6.0.0 2422<https://unicode.org/versions/Unicode6.0.0/>`_. The update to the standard adds 2423over 2,000 new characters including `emoji <https://en.wikipedia.org/wiki/Emoji>`_ 2424symbols which are important for mobile phones. 2425 2426In addition, the updated standard has altered the character properties for two 2427Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character 2428(U+19DA), making the former eligible for use in identifiers while disqualifying 2429the latter. For more information, see `Unicode Character Database Changes 2430<https://www.unicode.org/versions/Unicode6.0.0/#Database_Changes>`_. 2431 2432 2433Codecs 2434====== 2435 2436Support was added for *cp720* Arabic DOS encoding (:issue:`1616979`). 2437 2438MBCS encoding no longer ignores the error handler argument. In the default 2439strict mode, it raises an :exc:`UnicodeDecodeError` when it encounters an 2440undecodable byte sequence and an :exc:`UnicodeEncodeError` for an unencodable 2441character. 2442 2443The MBCS codec supports ``'strict'`` and ``'ignore'`` error handlers for 2444decoding, and ``'strict'`` and ``'replace'`` for encoding. 2445 2446To emulate Python3.1 MBCS encoding, select the ``'ignore'`` handler for decoding 2447and the ``'replace'`` handler for encoding. 2448 2449On Mac OS X, Python decodes command line arguments with ``'utf-8'`` rather than 2450the locale encoding. 2451 2452By default, :mod:`tarfile` uses ``'utf-8'`` encoding on Windows (instead of 2453``'mbcs'``) and the ``'surrogateescape'`` error handler on all operating 2454systems. 2455 2456 2457Documentation 2458============= 2459 2460The documentation continues to be improved. 2461 2462* A table of quick links has been added to the top of lengthy sections such as 2463 :ref:`built-in-funcs`. In the case of :mod:`itertools`, the links are 2464 accompanied by tables of cheatsheet-style summaries to provide an overview and 2465 memory jog without having to read all of the docs. 2466 2467* In some cases, the pure Python source code can be a helpful adjunct to the 2468 documentation, so now many modules now feature quick links to the latest 2469 version of the source code. For example, the :mod:`functools` module 2470 documentation has a quick link at the top labeled: 2471 2472 **Source code** :source:`Lib/functools.py`. 2473 2474 (Contributed by Raymond Hettinger; see 2475 `rationale <https://rhettinger.wordpress.com/2011/01/28/open-your-source-more/>`_.) 2476 2477* The docs now contain more examples and recipes. In particular, :mod:`re` 2478 module has an extensive section, :ref:`re-examples`. Likewise, the 2479 :mod:`itertools` module continues to be updated with new 2480 :ref:`itertools-recipes`. 2481 2482* The :mod:`datetime` module now has an auxiliary implementation in pure Python. 2483 No functionality was changed. This just provides an easier-to-read alternate 2484 implementation. 2485 2486 (Contributed by Alexander Belopolsky in :issue:`9528`.) 2487 2488* The unmaintained :file:`Demo` directory has been removed. Some demos were 2489 integrated into the documentation, some were moved to the :file:`Tools/demo` 2490 directory, and others were removed altogether. 2491 2492 (Contributed by Georg Brandl in :issue:`7962`.) 2493 2494 2495IDLE 2496==== 2497 2498* The format menu now has an option to clean source files by stripping 2499 trailing whitespace. 2500 2501 (Contributed by Raymond Hettinger; :issue:`5150`.) 2502 2503* IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk. 2504 2505 (Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; :issue:`6075`.) 2506 2507Code Repository 2508=============== 2509 2510In addition to the existing Subversion code repository at https://svn.python.org 2511there is now a `Mercurial <https://www.mercurial-scm.org/>`_ repository at 2512https://hg.python.org/\ . 2513 2514After the 3.2 release, there are plans to switch to Mercurial as the primary 2515repository. This distributed version control system should make it easier for 2516members of the community to create and share external changesets. See 2517:pep:`385` for details. 2518 2519To learn to use the new version control system, see the `Quick Start 2520<https://www.mercurial-scm.org/wiki/QuickStart>`_ or the `Guide to 2521Mercurial Workflows <https://www.mercurial-scm.org/guide>`_. 2522 2523 2524Build and C API Changes 2525======================= 2526 2527Changes to Python's build process and to the C API include: 2528 2529* The *idle*, *pydoc* and *2to3* scripts are now installed with a 2530 version-specific suffix on ``make altinstall`` (:issue:`10679`). 2531 2532* The C functions that access the Unicode Database now accept and return 2533 characters from the full Unicode range, even on narrow unicode builds 2534 (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference 2535 in Python is that :func:`unicodedata.numeric` now returns the correct value 2536 for large code points, and :func:`repr` may consider more characters as 2537 printable. 2538 2539 (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.) 2540 2541* Computed gotos are now enabled by default on supported compilers (which are 2542 detected by the configure script). They can still be disabled selectively by 2543 specifying ``--without-computed-gotos``. 2544 2545 (Contributed by Antoine Pitrou; :issue:`9203`.) 2546 2547* The option ``--with-wctype-functions`` was removed. The built-in unicode 2548 database is now used for all functions. 2549 2550 (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.) 2551 2552* Hash values are now values of a new type, :c:type:`Py_hash_t`, which is 2553 defined to be the same size as a pointer. Previously they were of type long, 2554 which on some 64-bit operating systems is still only 32 bits long. As a 2555 result of this fix, :class:`set` and :class:`dict` can now hold more than 2556 ``2**32`` entries on builds with 64-bit pointers (previously, they could grow 2557 to that size but their performance degraded catastrophically). 2558 2559 (Suggested by Raymond Hettinger and implemented by Benjamin Peterson; 2560 :issue:`9778`.) 2561 2562* A new macro :c:macro:`Py_VA_COPY` copies the state of the variable argument 2563 list. It is equivalent to C99 *va_copy* but available on all Python platforms 2564 (:issue:`2443`). 2565 2566* A new C API function :c:func:`PySys_SetArgvEx` allows an embedded interpreter 2567 to set :attr:`sys.argv` without also modifying :attr:`sys.path` 2568 (:issue:`5753`). 2569 2570* :c:macro:`PyEval_CallObject` is now only available in macro form. The 2571 function declaration, which was kept for backwards compatibility reasons, is 2572 now removed -- the macro was introduced in 1997 (:issue:`8276`). 2573 2574* There is a new function :c:func:`PyLong_AsLongLongAndOverflow` which 2575 is analogous to :c:func:`PyLong_AsLongAndOverflow`. They both serve to 2576 convert Python :class:`int` into a native fixed-width type while providing 2577 detection of cases where the conversion won't fit (:issue:`7767`). 2578 2579* The :c:func:`PyUnicode_CompareWithASCIIString` function now returns *not 2580 equal* if the Python string is *NUL* terminated. 2581 2582* There is a new function :c:func:`PyErr_NewExceptionWithDoc` that is 2583 like :c:func:`PyErr_NewException` but allows a docstring to be specified. 2584 This lets C exceptions have the same self-documenting capabilities as 2585 their pure Python counterparts (:issue:`7033`). 2586 2587* When compiled with the ``--with-valgrind`` option, the pymalloc 2588 allocator will be automatically disabled when running under Valgrind. This 2589 gives improved memory leak detection when running under Valgrind, while taking 2590 advantage of pymalloc at other times (:issue:`2422`). 2591 2592* Removed the ``O?`` format from the *PyArg_Parse* functions. The format is no 2593 longer used and it had never been documented (:issue:`8837`). 2594 2595There were a number of other small changes to the C-API. See the 2596`Misc/NEWS <https://github.com/python/cpython/blob/v3.2.6/Misc/NEWS>`__ 2597file for a complete list. 2598 2599Also, there were a number of updates to the Mac OS X build, see 2600`Mac/BuildScript/README.txt <https://github.com/python/cpython/blob/v3.2.6/Mac/BuildScript/README.txt>`_ 2601for details. For users running a 32/64-bit 2602build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6. 2603Accordingly, we recommend installing an updated alternative such as 2604`ActiveState Tcl/Tk 8.5.9 <https://web.archive.org/web/20101208191259/https://www.activestate.com/activetcl/downloads>`_\. 2605See https://www.python.org/download/mac/tcltk/ for additional details. 2606 2607Porting to Python 3.2 2608===================== 2609 2610This section lists previously described changes and other bugfixes that may 2611require changes to your code: 2612 2613* The :mod:`configparser` module has a number of clean-ups. The major change is 2614 to replace the old :class:`ConfigParser` class with long-standing preferred 2615 alternative :class:`SafeConfigParser`. In addition there are a number of 2616 smaller incompatibilities: 2617 2618 * The interpolation syntax is now validated on 2619 :meth:`~configparser.ConfigParser.get` and 2620 :meth:`~configparser.ConfigParser.set` operations. In the default 2621 interpolation scheme, only two tokens with percent signs are valid: ``%(name)s`` 2622 and ``%%``, the latter being an escaped percent sign. 2623 2624 * The :meth:`~configparser.ConfigParser.set` and 2625 :meth:`~configparser.ConfigParser.add_section` methods now verify that 2626 values are actual strings. Formerly, unsupported types could be introduced 2627 unintentionally. 2628 2629 * Duplicate sections or options from a single source now raise either 2630 :exc:`~configparser.DuplicateSectionError` or 2631 :exc:`~configparser.DuplicateOptionError`. Formerly, duplicates would 2632 silently overwrite a previous entry. 2633 2634 * Inline comments are now disabled by default so now the **;** character 2635 can be safely used in values. 2636 2637 * Comments now can be indented. Consequently, for **;** or **#** to appear at 2638 the start of a line in multiline values, it has to be interpolated. This 2639 keeps comment prefix characters in values from being mistaken as comments. 2640 2641 * ``""`` is now a valid value and is no longer automatically converted to an 2642 empty string. For empty strings, use ``"option ="`` in a line. 2643 2644* The :mod:`nntplib` module was reworked extensively, meaning that its APIs 2645 are often incompatible with the 3.1 APIs. 2646 2647* :class:`bytearray` objects can no longer be used as filenames; instead, 2648 they should be converted to :class:`bytes`. 2649 2650* The :meth:`array.tostring` and :meth:`array.fromstring` have been renamed to 2651 :meth:`array.tobytes` and :meth:`array.frombytes` for clarity. The old names 2652 have been deprecated. (See :issue:`8990`.) 2653 2654* ``PyArg_Parse*()`` functions: 2655 2656 * "t#" format has been removed: use "s#" or "s*" instead 2657 * "w" and "w#" formats has been removed: use "w*" instead 2658 2659* The :c:type:`PyCObject` type, deprecated in 3.1, has been removed. To wrap 2660 opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used 2661 instead; the new type has a well-defined interface for passing typing safety 2662 information and a less complicated signature for calling a destructor. 2663 2664* The :func:`sys.setfilesystemencoding` function was removed because 2665 it had a flawed design. 2666 2667* The :func:`random.seed` function and method now salt string seeds with an 2668 sha512 hash function. To access the previous version of *seed* in order to 2669 reproduce Python 3.1 sequences, set the *version* argument to *1*, 2670 ``random.seed(s, version=1)``. 2671 2672* The previously deprecated :func:`string.maketrans` function has been removed 2673 in favor of the static methods :meth:`bytes.maketrans` and 2674 :meth:`bytearray.maketrans`. This change solves the confusion around which 2675 types were supported by the :mod:`string` module. Now, :class:`str`, 2676 :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and 2677 **translate** methods with intermediate translation tables of the appropriate 2678 type. 2679 2680 (Contributed by Georg Brandl; :issue:`5675`.) 2681 2682* The previously deprecated :func:`contextlib.nested` function has been removed 2683 in favor of a plain :keyword:`with` statement which can accept multiple 2684 context managers. The latter technique is faster (because it is built-in), 2685 and it does a better job finalizing multiple context managers when one of them 2686 raises an exception:: 2687 2688 with open('mylog.txt') as infile, open('a.out', 'w') as outfile: 2689 for line in infile: 2690 if '<critical>' in line: 2691 outfile.write(line) 2692 2693 (Contributed by Georg Brandl and Mattias Brändström; 2694 `appspot issue 53094 <https://codereview.appspot.com/53094>`_.) 2695 2696* :func:`struct.pack` now only allows bytes for the ``s`` string pack code. 2697 Formerly, it would accept text arguments and implicitly encode them to bytes 2698 using UTF-8. This was problematic because it made assumptions about the 2699 correct encoding and because a variable-length encoding can fail when writing 2700 to fixed length segment of a structure. 2701 2702 Code such as ``struct.pack('<6sHHBBB', 'GIF87a', x, y)`` should be rewritten 2703 with to use bytes instead of text, ``struct.pack('<6sHHBBB', b'GIF87a', x, y)``. 2704 2705 (Discovered by David Beazley and fixed by Victor Stinner; :issue:`10783`.) 2706 2707* The :class:`xml.etree.ElementTree` class now raises an 2708 :exc:`xml.etree.ElementTree.ParseError` when a parse fails. Previously it 2709 raised an :exc:`xml.parsers.expat.ExpatError`. 2710 2711* The new, longer :func:`str` value on floats may break doctests which rely on 2712 the old output format. 2713 2714* In :class:`subprocess.Popen`, the default value for *close_fds* is now 2715 ``True`` under Unix; under Windows, it is ``True`` if the three standard 2716 streams are set to ``None``, ``False`` otherwise. Previously, *close_fds* 2717 was always ``False`` by default, which produced difficult to solve bugs 2718 or race conditions when open file descriptors would leak into the child 2719 process. 2720 2721* Support for legacy HTTP 0.9 has been removed from :mod:`urllib.request` 2722 and :mod:`http.client`. Such support is still present on the server side 2723 (in :mod:`http.server`). 2724 2725 (Contributed by Antoine Pitrou, :issue:`10711`.) 2726 2727* SSL sockets in timeout mode now raise :exc:`socket.timeout` when a timeout 2728 occurs, rather than a generic :exc:`~ssl.SSLError`. 2729 2730 (Contributed by Antoine Pitrou, :issue:`10272`.) 2731 2732* The misleading functions :c:func:`PyEval_AcquireLock()` and 2733 :c:func:`PyEval_ReleaseLock()` have been officially deprecated. The 2734 thread-state aware APIs (such as :c:func:`PyEval_SaveThread()` 2735 and :c:func:`PyEval_RestoreThread()`) should be used instead. 2736 2737* Due to security risks, :func:`asyncore.handle_accept` has been deprecated, and 2738 a new function, :func:`asyncore.handle_accepted`, was added to replace it. 2739 2740 (Contributed by Giampaolo Rodola in :issue:`6706`.) 2741 2742* Due to the new :term:`GIL` implementation, :c:func:`PyEval_InitThreads()` 2743 cannot be called before :c:func:`Py_Initialize()` anymore. 2744