1:mod:`venv` --- Creation of virtual environments 2================================================ 3 4.. module:: venv 5 :synopsis: Creation of virtual environments. 6 7.. moduleauthor:: Vinay Sajip <[email protected]> 8.. sectionauthor:: Vinay Sajip <[email protected]> 9 10.. versionadded:: 3.3 11 12**Source code:** :source:`Lib/venv/` 13 14.. index:: pair: Environments; virtual 15 16-------------- 17 18.. _venv-def: 19.. _venv-intro: 20 21The :mod:`!venv` module supports creating lightweight "virtual environments", 22each with their own independent set of Python packages installed in 23their :mod:`site` directories. 24A virtual environment is created on top of an existing 25Python installation, known as the virtual environment's "base" Python, and may 26optionally be isolated from the packages in the base environment, 27so only those explicitly installed in the virtual environment are available. 28 29When used from within a virtual environment, common installation tools such as 30`pip`_ will install Python packages into a virtual environment 31without needing to be told to do so explicitly. 32 33See :pep:`405` for more background on Python virtual environments. 34 35.. seealso:: 36 37 `Python Packaging User Guide: Creating and using virtual environments 38 <https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment>`__ 39 40.. include:: ../includes/wasm-notavail.rst 41 42Creating virtual environments 43----------------------------- 44 45.. include:: /using/venv-create.inc 46 47.. _venv-explanation: 48 49How venvs work 50-------------- 51 52When a Python interpreter is running from a virtual environment, 53:data:`sys.prefix` and :data:`sys.exec_prefix` 54point to the directories of the virtual environment, 55whereas :data:`sys.base_prefix` and :data:`sys.base_exec_prefix` 56point to those of the base Python used to create the environment. 57It is sufficient to check 58``sys.prefix != sys.base_prefix`` to determine if the current interpreter is 59running from a virtual environment. 60 61A virtual environment may be "activated" using a script in its binary directory 62(``bin`` on POSIX; ``Scripts`` on Windows). 63This will prepend that directory to your :envvar:`!PATH`, so that running 64:program:`python` will invoke the environment's Python interpreter 65and you can run installed scripts without having to use their full path. 66The invocation of the activation script is platform-specific 67(:samp:`{<venv>}` must be replaced by the path to the directory 68containing the virtual environment): 69 70+-------------+------------+--------------------------------------------------+ 71| Platform | Shell | Command to activate virtual environment | 72+=============+============+==================================================+ 73| POSIX | bash/zsh | :samp:`$ source {<venv>}/bin/activate` | 74| +------------+--------------------------------------------------+ 75| | fish | :samp:`$ source {<venv>}/bin/activate.fish` | 76| +------------+--------------------------------------------------+ 77| | csh/tcsh | :samp:`$ source {<venv>}/bin/activate.csh` | 78| +------------+--------------------------------------------------+ 79| | PowerShell | :samp:`$ {<venv>}/bin/Activate.ps1` | 80+-------------+------------+--------------------------------------------------+ 81| Windows | cmd.exe | :samp:`C:\\> {<venv>}\\Scripts\\activate.bat` | 82| +------------+--------------------------------------------------+ 83| | PowerShell | :samp:`PS C:\\> {<venv>}\\Scripts\\Activate.ps1` | 84+-------------+------------+--------------------------------------------------+ 85 86.. versionadded:: 3.4 87 :program:`fish` and :program:`csh` activation scripts. 88 89.. versionadded:: 3.8 90 PowerShell activation scripts installed under POSIX for PowerShell Core 91 support. 92 93You don't specifically *need* to activate a virtual environment, 94as you can just specify the full path to that environment's 95Python interpreter when invoking Python. 96Furthermore, all scripts installed in the environment 97should be runnable without activating it. 98 99In order to achieve this, scripts installed into virtual environments have 100a "shebang" line which points to the environment's Python interpreter, 101i.e. :samp:`#!/{<path-to-venv>}/bin/python`. 102This means that the script will run with that interpreter regardless of the 103value of :envvar:`!PATH`. On Windows, "shebang" line processing is supported if 104you have the :ref:`launcher` installed. Thus, double-clicking an installed 105script in a Windows Explorer window should run it with the correct interpreter 106without the environment needing to be activated or on the :envvar:`!PATH`. 107 108When a virtual environment has been activated, the :envvar:`!VIRTUAL_ENV` 109environment variable is set to the path of the environment. 110Since explicitly activating a virtual environment is not required to use it, 111:envvar:`!VIRTUAL_ENV` cannot be relied upon to determine 112whether a virtual environment is being used. 113 114.. warning:: Because scripts installed in environments should not expect the 115 environment to be activated, their shebang lines contain the absolute paths 116 to their environment's interpreters. Because of this, environments are 117 inherently non-portable, in the general case. You should always have a 118 simple means of recreating an environment (for example, if you have a 119 requirements file ``requirements.txt``, you can invoke ``pip install -r 120 requirements.txt`` using the environment's ``pip`` to install all of the 121 packages needed by the environment). If for any reason you need to move the 122 environment to a new location, you should recreate it at the desired 123 location and delete the one at the old location. If you move an environment 124 because you moved a parent directory of it, you should recreate the 125 environment in its new location. Otherwise, software installed into the 126 environment may not work as expected. 127 128You can deactivate a virtual environment by typing ``deactivate`` in your shell. 129The exact mechanism is platform-specific and is an internal implementation 130detail (typically, a script or shell function will be used). 131 132 133.. _venv-api: 134 135API 136--- 137 138.. highlight:: python 139 140The high-level method described above makes use of a simple API which provides 141mechanisms for third-party virtual environment creators to customize environment 142creation according to their needs, the :class:`EnvBuilder` class. 143 144.. class:: EnvBuilder(system_site_packages=False, clear=False, \ 145 symlinks=False, upgrade=False, with_pip=False, \ 146 prompt=None, upgrade_deps=False) 147 148 The :class:`EnvBuilder` class accepts the following keyword arguments on 149 instantiation: 150 151 * ``system_site_packages`` -- a Boolean value indicating that the system Python 152 site-packages should be available to the environment (defaults to ``False``). 153 154 * ``clear`` -- a Boolean value which, if true, will delete the contents of 155 any existing target directory, before creating the environment. 156 157 * ``symlinks`` -- a Boolean value indicating whether to attempt to symlink the 158 Python binary rather than copying. 159 160 * ``upgrade`` -- a Boolean value which, if true, will upgrade an existing 161 environment with the running Python - for use when that Python has been 162 upgraded in-place (defaults to ``False``). 163 164 * ``with_pip`` -- a Boolean value which, if true, ensures pip is 165 installed in the virtual environment. This uses :mod:`ensurepip` with 166 the ``--default-pip`` option. 167 168 * ``prompt`` -- a String to be used after virtual environment is activated 169 (defaults to ``None`` which means directory name of the environment would 170 be used). If the special string ``"."`` is provided, the basename of the 171 current directory is used as the prompt. 172 173 * ``upgrade_deps`` -- Update the base venv modules to the latest on PyPI 174 175 .. versionchanged:: 3.4 176 Added the ``with_pip`` parameter 177 178 .. versionadded:: 3.6 179 Added the ``prompt`` parameter 180 181 .. versionadded:: 3.9 182 Added the ``upgrade_deps`` parameter 183 184 Creators of third-party virtual environment tools will be free to use the 185 provided :class:`EnvBuilder` class as a base class. 186 187 The returned env-builder is an object which has a method, ``create``: 188 189 .. method:: create(env_dir) 190 191 Create a virtual environment by specifying the target directory 192 (absolute or relative to the current directory) which is to contain the 193 virtual environment. The ``create`` method will either create the 194 environment in the specified directory, or raise an appropriate 195 exception. 196 197 The ``create`` method of the :class:`EnvBuilder` class illustrates the 198 hooks available for subclass customization:: 199 200 def create(self, env_dir): 201 """ 202 Create a virtualized Python environment in a directory. 203 env_dir is the target directory to create an environment in. 204 """ 205 env_dir = os.path.abspath(env_dir) 206 context = self.ensure_directories(env_dir) 207 self.create_configuration(context) 208 self.setup_python(context) 209 self.setup_scripts(context) 210 self.post_setup(context) 211 212 Each of the methods :meth:`ensure_directories`, 213 :meth:`create_configuration`, :meth:`setup_python`, 214 :meth:`setup_scripts` and :meth:`post_setup` can be overridden. 215 216 .. method:: ensure_directories(env_dir) 217 218 Creates the environment directory and all necessary subdirectories that 219 don't already exist, and returns a context object. This context object 220 is just a holder for attributes (such as paths) for use by the other 221 methods. If the :class:`EnvBuilder` is created with the arg 222 ``clear=True``, contents of the environment directory will be cleared 223 and then all necessary subdirectories will be recreated. 224 225 The returned context object is a :class:`types.SimpleNamespace` with the 226 following attributes: 227 228 * ``env_dir`` - The location of the virtual environment. Used for 229 ``__VENV_DIR__`` in activation scripts (see :meth:`install_scripts`). 230 231 * ``env_name`` - The name of the virtual environment. Used for 232 ``__VENV_NAME__`` in activation scripts (see :meth:`install_scripts`). 233 234 * ``prompt`` - The prompt to be used by the activation scripts. Used for 235 ``__VENV_PROMPT__`` in activation scripts (see :meth:`install_scripts`). 236 237 * ``executable`` - The underlying Python executable used by the virtual 238 environment. This takes into account the case where a virtual environment 239 is created from another virtual environment. 240 241 * ``inc_path`` - The include path for the virtual environment. 242 243 * ``lib_path`` - The purelib path for the virtual environment. 244 245 * ``bin_path`` - The script path for the virtual environment. 246 247 * ``bin_name`` - The name of the script path relative to the virtual 248 environment location. Used for ``__VENV_BIN_NAME__`` in activation 249 scripts (see :meth:`install_scripts`). 250 251 * ``env_exe`` - The name of the Python interpreter in the virtual 252 environment. Used for ``__VENV_PYTHON__`` in activation scripts 253 (see :meth:`install_scripts`). 254 255 * ``env_exec_cmd`` - The name of the Python interpreter, taking into 256 account filesystem redirections. This can be used to run Python in 257 the virtual environment. 258 259 260 .. versionchanged:: 3.12 261 The attribute ``lib_path`` was added to the context, and the context 262 object was documented. 263 264 .. versionchanged:: 3.11 265 The *venv* 266 :ref:`sysconfig installation scheme <installation_paths>` 267 is used to construct the paths of the created directories. 268 269 .. method:: create_configuration(context) 270 271 Creates the ``pyvenv.cfg`` configuration file in the environment. 272 273 .. method:: setup_python(context) 274 275 Creates a copy or symlink to the Python executable in the environment. 276 On POSIX systems, if a specific executable ``python3.x`` was used, 277 symlinks to ``python`` and ``python3`` will be created pointing to that 278 executable, unless files with those names already exist. 279 280 .. method:: setup_scripts(context) 281 282 Installs activation scripts appropriate to the platform into the virtual 283 environment. 284 285 .. method:: upgrade_dependencies(context) 286 287 Upgrades the core venv dependency packages (currently ``pip`` and 288 ``setuptools``) in the environment. This is done by shelling out to the 289 ``pip`` executable in the environment. 290 291 .. versionadded:: 3.9 292 293 .. method:: post_setup(context) 294 295 A placeholder method which can be overridden in third party 296 implementations to pre-install packages in the virtual environment or 297 perform other post-creation steps. 298 299 .. versionchanged:: 3.7.2 300 Windows now uses redirector scripts for ``python[w].exe`` instead of 301 copying the actual binaries. In 3.7.2 only :meth:`setup_python` does 302 nothing unless running from a build in the source tree. 303 304 .. versionchanged:: 3.7.3 305 Windows copies the redirector scripts as part of :meth:`setup_python` 306 instead of :meth:`setup_scripts`. This was not the case in 3.7.2. 307 When using symlinks, the original executables will be linked. 308 309 In addition, :class:`EnvBuilder` provides this utility method that can be 310 called from :meth:`setup_scripts` or :meth:`post_setup` in subclasses to 311 assist in installing custom scripts into the virtual environment. 312 313 .. method:: install_scripts(context, path) 314 315 *path* is the path to a directory that should contain subdirectories 316 "common", "posix", "nt", each containing scripts destined for the bin 317 directory in the environment. The contents of "common" and the 318 directory corresponding to :data:`os.name` are copied after some text 319 replacement of placeholders: 320 321 * ``__VENV_DIR__`` is replaced with the absolute path of the environment 322 directory. 323 324 * ``__VENV_NAME__`` is replaced with the environment name (final path 325 segment of environment directory). 326 327 * ``__VENV_PROMPT__`` is replaced with the prompt (the environment 328 name surrounded by parentheses and with a following space) 329 330 * ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory 331 (either ``bin`` or ``Scripts``). 332 333 * ``__VENV_PYTHON__`` is replaced with the absolute path of the 334 environment's executable. 335 336 The directories are allowed to exist (for when an existing environment 337 is being upgraded). 338 339There is also a module-level convenience function: 340 341.. function:: create(env_dir, system_site_packages=False, clear=False, \ 342 symlinks=False, with_pip=False, prompt=None, \ 343 upgrade_deps=False) 344 345 Create an :class:`EnvBuilder` with the given keyword arguments, and call its 346 :meth:`~EnvBuilder.create` method with the *env_dir* argument. 347 348 .. versionadded:: 3.3 349 350 .. versionchanged:: 3.4 351 Added the ``with_pip`` parameter 352 353 .. versionchanged:: 3.6 354 Added the ``prompt`` parameter 355 356 .. versionchanged:: 3.9 357 Added the ``upgrade_deps`` parameter 358 359An example of extending ``EnvBuilder`` 360-------------------------------------- 361 362The following script shows how to extend :class:`EnvBuilder` by implementing a 363subclass which installs setuptools and pip into a created virtual environment:: 364 365 import os 366 import os.path 367 from subprocess import Popen, PIPE 368 import sys 369 from threading import Thread 370 from urllib.parse import urlparse 371 from urllib.request import urlretrieve 372 import venv 373 374 class ExtendedEnvBuilder(venv.EnvBuilder): 375 """ 376 This builder installs setuptools and pip so that you can pip or 377 easy_install other packages into the created virtual environment. 378 379 :param nodist: If true, setuptools and pip are not installed into the 380 created virtual environment. 381 :param nopip: If true, pip is not installed into the created 382 virtual environment. 383 :param progress: If setuptools or pip are installed, the progress of the 384 installation can be monitored by passing a progress 385 callable. If specified, it is called with two 386 arguments: a string indicating some progress, and a 387 context indicating where the string is coming from. 388 The context argument can have one of three values: 389 'main', indicating that it is called from virtualize() 390 itself, and 'stdout' and 'stderr', which are obtained 391 by reading lines from the output streams of a subprocess 392 which is used to install the app. 393 394 If a callable is not specified, default progress 395 information is output to sys.stderr. 396 """ 397 398 def __init__(self, *args, **kwargs): 399 self.nodist = kwargs.pop('nodist', False) 400 self.nopip = kwargs.pop('nopip', False) 401 self.progress = kwargs.pop('progress', None) 402 self.verbose = kwargs.pop('verbose', False) 403 super().__init__(*args, **kwargs) 404 405 def post_setup(self, context): 406 """ 407 Set up any packages which need to be pre-installed into the 408 virtual environment being created. 409 410 :param context: The information for the virtual environment 411 creation request being processed. 412 """ 413 os.environ['VIRTUAL_ENV'] = context.env_dir 414 if not self.nodist: 415 self.install_setuptools(context) 416 # Can't install pip without setuptools 417 if not self.nopip and not self.nodist: 418 self.install_pip(context) 419 420 def reader(self, stream, context): 421 """ 422 Read lines from a subprocess' output stream and either pass to a progress 423 callable (if specified) or write progress information to sys.stderr. 424 """ 425 progress = self.progress 426 while True: 427 s = stream.readline() 428 if not s: 429 break 430 if progress is not None: 431 progress(s, context) 432 else: 433 if not self.verbose: 434 sys.stderr.write('.') 435 else: 436 sys.stderr.write(s.decode('utf-8')) 437 sys.stderr.flush() 438 stream.close() 439 440 def install_script(self, context, name, url): 441 _, _, path, _, _, _ = urlparse(url) 442 fn = os.path.split(path)[-1] 443 binpath = context.bin_path 444 distpath = os.path.join(binpath, fn) 445 # Download script into the virtual environment's binaries folder 446 urlretrieve(url, distpath) 447 progress = self.progress 448 if self.verbose: 449 term = '\n' 450 else: 451 term = '' 452 if progress is not None: 453 progress('Installing %s ...%s' % (name, term), 'main') 454 else: 455 sys.stderr.write('Installing %s ...%s' % (name, term)) 456 sys.stderr.flush() 457 # Install in the virtual environment 458 args = [context.env_exe, fn] 459 p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) 460 t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) 461 t1.start() 462 t2 = Thread(target=self.reader, args=(p.stderr, 'stderr')) 463 t2.start() 464 p.wait() 465 t1.join() 466 t2.join() 467 if progress is not None: 468 progress('done.', 'main') 469 else: 470 sys.stderr.write('done.\n') 471 # Clean up - no longer needed 472 os.unlink(distpath) 473 474 def install_setuptools(self, context): 475 """ 476 Install setuptools in the virtual environment. 477 478 :param context: The information for the virtual environment 479 creation request being processed. 480 """ 481 url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py' 482 self.install_script(context, 'setuptools', url) 483 # clear up the setuptools archive which gets downloaded 484 pred = lambda o: o.startswith('setuptools-') and o.endswith('.tar.gz') 485 files = filter(pred, os.listdir(context.bin_path)) 486 for f in files: 487 f = os.path.join(context.bin_path, f) 488 os.unlink(f) 489 490 def install_pip(self, context): 491 """ 492 Install pip in the virtual environment. 493 494 :param context: The information for the virtual environment 495 creation request being processed. 496 """ 497 url = 'https://bootstrap.pypa.io/get-pip.py' 498 self.install_script(context, 'pip', url) 499 500 def main(args=None): 501 compatible = True 502 if sys.version_info < (3, 3): 503 compatible = False 504 elif not hasattr(sys, 'base_prefix'): 505 compatible = False 506 if not compatible: 507 raise ValueError('This script is only for use with ' 508 'Python 3.3 or later') 509 else: 510 import argparse 511 512 parser = argparse.ArgumentParser(prog=__name__, 513 description='Creates virtual Python ' 514 'environments in one or ' 515 'more target ' 516 'directories.') 517 parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', 518 help='A directory in which to create the ' 519 'virtual environment.') 520 parser.add_argument('--no-setuptools', default=False, 521 action='store_true', dest='nodist', 522 help="Don't install setuptools or pip in the " 523 "virtual environment.") 524 parser.add_argument('--no-pip', default=False, 525 action='store_true', dest='nopip', 526 help="Don't install pip in the virtual " 527 "environment.") 528 parser.add_argument('--system-site-packages', default=False, 529 action='store_true', dest='system_site', 530 help='Give the virtual environment access to the ' 531 'system site-packages dir.') 532 if os.name == 'nt': 533 use_symlinks = False 534 else: 535 use_symlinks = True 536 parser.add_argument('--symlinks', default=use_symlinks, 537 action='store_true', dest='symlinks', 538 help='Try to use symlinks rather than copies, ' 539 'when symlinks are not the default for ' 540 'the platform.') 541 parser.add_argument('--clear', default=False, action='store_true', 542 dest='clear', help='Delete the contents of the ' 543 'virtual environment ' 544 'directory if it already ' 545 'exists, before virtual ' 546 'environment creation.') 547 parser.add_argument('--upgrade', default=False, action='store_true', 548 dest='upgrade', help='Upgrade the virtual ' 549 'environment directory to ' 550 'use this version of ' 551 'Python, assuming Python ' 552 'has been upgraded ' 553 'in-place.') 554 parser.add_argument('--verbose', default=False, action='store_true', 555 dest='verbose', help='Display the output ' 556 'from the scripts which ' 557 'install setuptools and pip.') 558 options = parser.parse_args(args) 559 if options.upgrade and options.clear: 560 raise ValueError('you cannot supply --upgrade and --clear together.') 561 builder = ExtendedEnvBuilder(system_site_packages=options.system_site, 562 clear=options.clear, 563 symlinks=options.symlinks, 564 upgrade=options.upgrade, 565 nodist=options.nodist, 566 nopip=options.nopip, 567 verbose=options.verbose) 568 for d in options.dirs: 569 builder.create(d) 570 571 if __name__ == '__main__': 572 rc = 1 573 try: 574 main() 575 rc = 0 576 except Exception as e: 577 print('Error: %s' % e, file=sys.stderr) 578 sys.exit(rc) 579 580 581This script is also available for download `online 582<https://gist.github.com/vsajip/4673395>`_. 583 584 585.. _setuptools: https://pypi.org/project/setuptools/ 586.. _pip: https://pypi.org/project/pip/ 587