1.. currentmodule:: asyncio
2
3.. _asyncio-subprocess:
4
5============
6Subprocesses
7============
8
9**Source code:** :source:`Lib/asyncio/subprocess.py`,
10:source:`Lib/asyncio/base_subprocess.py`
11
12----------------------------------------
13
14This section describes high-level async/await asyncio APIs to
15create and manage subprocesses.
16
17.. _asyncio_example_subprocess_shell:
18
19Here's an example of how asyncio can run a shell command and
20obtain its result::
21
22    import asyncio
23
24    async def run(cmd):
25        proc = await asyncio.create_subprocess_shell(
26            cmd,
27            stdout=asyncio.subprocess.PIPE,
28            stderr=asyncio.subprocess.PIPE)
29
30        stdout, stderr = await proc.communicate()
31
32        print(f'[{cmd!r} exited with {proc.returncode}]')
33        if stdout:
34            print(f'[stdout]\n{stdout.decode()}')
35        if stderr:
36            print(f'[stderr]\n{stderr.decode()}')
37
38    asyncio.run(run('ls /zzz'))
39
40will print::
41
42    ['ls /zzz' exited with 1]
43    [stderr]
44    ls: /zzz: No such file or directory
45
46Because all asyncio subprocess functions are asynchronous and asyncio
47provides many tools to work with such functions, it is easy to execute
48and monitor multiple subprocesses in parallel.  It is indeed trivial
49to modify the above example to run several commands simultaneously::
50
51    async def main():
52        await asyncio.gather(
53            run('ls /zzz'),
54            run('sleep 1; echo "hello"'))
55
56    asyncio.run(main())
57
58See also the `Examples`_ subsection.
59
60
61Creating Subprocesses
62=====================
63
64.. coroutinefunction:: create_subprocess_exec(program, *args, stdin=None, \
65                          stdout=None, stderr=None, limit=None, **kwds)
66
67   Create a subprocess.
68
69   The *limit* argument sets the buffer limit for :class:`StreamReader`
70   wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
71   (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
72
73   Return a :class:`~asyncio.subprocess.Process` instance.
74
75   See the documentation of :meth:`loop.subprocess_exec` for other
76   parameters.
77
78   .. versionchanged:: 3.10
79      Removed the *loop* parameter.
80
81
82.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \
83                          stdout=None, stderr=None, limit=None, **kwds)
84
85   Run the *cmd* shell command.
86
87   The *limit* argument sets the buffer limit for :class:`StreamReader`
88   wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
89   (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
90
91   Return a :class:`~asyncio.subprocess.Process` instance.
92
93   See the documentation of :meth:`loop.subprocess_shell` for other
94   parameters.
95
96   .. important::
97
98      It is the application's responsibility to ensure that all whitespace and
99      special characters are quoted appropriately to avoid `shell injection
100      <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
101      vulnerabilities. The :func:`shlex.quote` function can be used to properly
102      escape whitespace and special shell characters in strings that are going
103      to be used to construct shell commands.
104
105   .. versionchanged:: 3.10
106      Removed the *loop* parameter.
107
108.. note::
109
110   Subprocesses are available for Windows if a :class:`ProactorEventLoop` is
111   used. See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>`
112   for details.
113
114.. seealso::
115
116   asyncio also has the following *low-level* APIs to work with subprocesses:
117   :meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`,
118   :meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`,
119   as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>`
120   and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`.
121
122
123Constants
124=========
125
126.. data:: asyncio.subprocess.PIPE
127   :module:
128
129   Can be passed to the *stdin*, *stdout* or *stderr* parameters.
130
131   If *PIPE* is passed to *stdin* argument, the
132   :attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute
133   will point to a :class:`StreamWriter` instance.
134
135   If *PIPE* is passed to *stdout* or *stderr* arguments, the
136   :attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and
137   :attr:`Process.stderr <asyncio.subprocess.Process.stderr>`
138   attributes will point to :class:`StreamReader` instances.
139
140.. data:: asyncio.subprocess.STDOUT
141   :module:
142
143   Special value that can be used as the *stderr* argument and indicates
144   that standard error should be redirected into standard output.
145
146.. data:: asyncio.subprocess.DEVNULL
147   :module:
148
149   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
150   to process creation functions.  It indicates that the special file
151   :data:`os.devnull` will be used for the corresponding subprocess stream.
152
153
154Interacting with Subprocesses
155=============================
156
157Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
158functions return instances of the *Process* class.  *Process* is a high-level
159wrapper that allows communicating with subprocesses and watching for
160their completion.
161
162.. class:: asyncio.subprocess.Process
163   :module:
164
165   An object that wraps OS processes created by the
166   :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
167   functions.
168
169   This class is designed to have a similar API to the
170   :class:`subprocess.Popen` class, but there are some
171   notable differences:
172
173   * unlike Popen, Process instances do not have an equivalent to
174     the :meth:`~subprocess.Popen.poll` method;
175
176   * the :meth:`~asyncio.subprocess.Process.communicate` and
177     :meth:`~asyncio.subprocess.Process.wait` methods don't have a
178     *timeout* parameter: use the :func:`~asyncio.wait_for` function;
179
180   * the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method
181     is asynchronous, whereas :meth:`subprocess.Popen.wait` method
182     is implemented as a blocking busy loop;
183
184   * the *universal_newlines* parameter is not supported.
185
186   This class is :ref:`not thread safe <asyncio-multithreading>`.
187
188   See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
189   section.
190
191   .. coroutinemethod:: wait()
192
193      Wait for the child process to terminate.
194
195      Set and return the :attr:`returncode` attribute.
196
197      .. note::
198
199         This method can deadlock when using ``stdout=PIPE`` or
200         ``stderr=PIPE`` and the child process generates so much output
201         that it blocks waiting for the OS pipe buffer to accept
202         more data. Use the :meth:`communicate` method when using pipes
203         to avoid this condition.
204
205   .. coroutinemethod:: communicate(input=None)
206
207      Interact with process:
208
209      1. send data to *stdin* (if *input* is not ``None``);
210      2. read data from *stdout* and *stderr*, until EOF is reached;
211      3. wait for process to terminate.
212
213      The optional *input* argument is the data (:class:`bytes` object)
214      that will be sent to the child process.
215
216      Return a tuple ``(stdout_data, stderr_data)``.
217
218      If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError`
219      exception is raised when writing *input* into *stdin*, the
220      exception is ignored.  This condition occurs when the process
221      exits before all data are written into *stdin*.
222
223      If it is desired to send data to the process' *stdin*,
224      the process needs to be created with ``stdin=PIPE``.  Similarly,
225      to get anything other than ``None`` in the result tuple, the
226      process has to be created with ``stdout=PIPE`` and/or
227      ``stderr=PIPE`` arguments.
228
229      Note, that the data read is buffered in memory, so do not use
230      this method if the data size is large or unlimited.
231
232   .. method:: send_signal(signal)
233
234      Sends the signal *signal* to the child process.
235
236      .. note::
237
238         On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`.
239         ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes
240         started with a *creationflags* parameter which includes
241         ``CREATE_NEW_PROCESS_GROUP``.
242
243   .. method:: terminate()
244
245      Stop the child process.
246
247      On POSIX systems this method sends :py:data:`signal.SIGTERM` to the
248      child process.
249
250      On Windows the Win32 API function :c:func:`TerminateProcess` is
251      called to stop the child process.
252
253   .. method:: kill()
254
255      Kill the child process.
256
257      On POSIX systems this method sends :py:data:`SIGKILL` to the child
258      process.
259
260      On Windows this method is an alias for :meth:`terminate`.
261
262   .. attribute:: stdin
263
264      Standard input stream (:class:`StreamWriter`) or ``None``
265      if the process was created with ``stdin=None``.
266
267   .. attribute:: stdout
268
269      Standard output stream (:class:`StreamReader`) or ``None``
270      if the process was created with ``stdout=None``.
271
272   .. attribute:: stderr
273
274      Standard error stream (:class:`StreamReader`) or ``None``
275      if the process was created with ``stderr=None``.
276
277   .. warning::
278
279      Use the :meth:`communicate` method rather than
280      :attr:`process.stdin.write() <stdin>`,
281      :attr:`await process.stdout.read() <stdout>` or
282      :attr:`await process.stderr.read() <stderr>`.
283      This avoids deadlocks due to streams pausing reading or writing
284      and blocking the child process.
285
286   .. attribute:: pid
287
288      Process identification number (PID).
289
290      Note that for processes created by the :func:`create_subprocess_shell`
291      function, this attribute is the PID of the spawned shell.
292
293   .. attribute:: returncode
294
295      Return code of the process when it exits.
296
297      A ``None`` value indicates that the process has not terminated yet.
298
299      A negative value ``-N`` indicates that the child was terminated
300      by signal ``N`` (POSIX only).
301
302
303.. _asyncio-subprocess-threads:
304
305Subprocess and Threads
306----------------------
307
308Standard asyncio event loop supports running subprocesses from different threads by
309default.
310
311On Windows subprocesses are provided by :class:`ProactorEventLoop` only (default),
312:class:`SelectorEventLoop` has no subprocess support.
313
314On UNIX *child watchers* are used for subprocess finish waiting, see
315:ref:`asyncio-watchers` for more info.
316
317
318.. versionchanged:: 3.8
319
320   UNIX switched to use :class:`ThreadedChildWatcher` for spawning subprocesses from
321   different threads without any limitation.
322
323   Spawning a subprocess with *inactive* current child watcher raises
324   :exc:`RuntimeError`.
325
326Note that alternative event loop implementations might have own limitations;
327please refer to their documentation.
328
329.. seealso::
330
331   The :ref:`Concurrency and multithreading in asyncio
332   <asyncio-multithreading>` section.
333
334
335Examples
336--------
337
338An example using the :class:`~asyncio.subprocess.Process` class to
339control a subprocess and the :class:`StreamReader` class to read from
340its standard output.
341
342.. _asyncio_example_create_subprocess_exec:
343
344The subprocess is created by the :func:`create_subprocess_exec`
345function::
346
347    import asyncio
348    import sys
349
350    async def get_date():
351        code = 'import datetime; print(datetime.datetime.now())'
352
353        # Create the subprocess; redirect the standard output
354        # into a pipe.
355        proc = await asyncio.create_subprocess_exec(
356            sys.executable, '-c', code,
357            stdout=asyncio.subprocess.PIPE)
358
359        # Read one line of output.
360        data = await proc.stdout.readline()
361        line = data.decode('ascii').rstrip()
362
363        # Wait for the subprocess exit.
364        await proc.wait()
365        return line
366
367    date = asyncio.run(get_date())
368    print(f"Current date: {date}")
369
370
371See also the :ref:`same example <asyncio_example_subprocess_proto>`
372written using low-level APIs.
373