1:mod:`mmap` --- Memory-mapped file support
2==========================================
3
4.. module:: mmap
5   :synopsis: Interface to memory-mapped files for Unix and Windows.
6
7--------------
8
9.. include:: ../includes/wasm-notavail.rst
10
11Memory-mapped file objects behave like both :class:`bytearray` and like
12:term:`file objects <file object>`.  You can use mmap objects in most places
13where :class:`bytearray` are expected; for example, you can use the :mod:`re`
14module to search through a memory-mapped file.  You can also change a single
15byte by doing ``obj[index] = 97``, or change a subsequence by assigning to a
16slice: ``obj[i1:i2] = b'...'``.  You can also read and write data starting at
17the current file position, and :meth:`seek` through the file to different positions.
18
19A memory-mapped file is created by the :class:`~mmap.mmap` constructor, which is
20different on Unix and on Windows.  In either case you must provide a file
21descriptor for a file opened for update. If you wish to map an existing Python
22file object, use its :meth:`fileno` method to obtain the correct value for the
23*fileno* parameter.  Otherwise, you can open the file using the
24:func:`os.open` function, which returns a file descriptor directly (the file
25still needs to be closed when done).
26
27.. note::
28   If you want to create a memory-mapping for a writable, buffered file, you
29   should :func:`~io.IOBase.flush` the file first.  This is necessary to ensure
30   that local modifications to the buffers are actually available to the
31   mapping.
32
33For both the Unix and Windows versions of the constructor, *access* may be
34specified as an optional keyword parameter. *access* accepts one of four
35values: :const:`ACCESS_READ`, :const:`ACCESS_WRITE`, or :const:`ACCESS_COPY` to
36specify read-only, write-through or copy-on-write memory respectively, or
37:const:`ACCESS_DEFAULT` to defer to *prot*.  *access* can be used on both Unix
38and Windows.  If *access* is not specified, Windows mmap returns a
39write-through mapping.  The initial memory values for all three access types
40are taken from the specified file.  Assignment to an :const:`ACCESS_READ`
41memory map raises a :exc:`TypeError` exception.  Assignment to an
42:const:`ACCESS_WRITE` memory map affects both memory and the underlying file.
43Assignment to an :const:`ACCESS_COPY` memory map affects memory but does not
44update the underlying file.
45
46.. versionchanged:: 3.7
47   Added :const:`ACCESS_DEFAULT` constant.
48
49To map anonymous memory, -1 should be passed as the fileno along with the length.
50
51.. class:: mmap(fileno, length, tagname=None, access=ACCESS_DEFAULT[, offset])
52
53   **(Windows version)** Maps *length* bytes from the file specified by the
54   file handle *fileno*, and creates a mmap object.  If *length* is larger
55   than the current size of the file, the file is extended to contain *length*
56   bytes.  If *length* is ``0``, the maximum length of the map is the current
57   size of the file, except that if the file is empty Windows raises an
58   exception (you cannot create an empty mapping on Windows).
59
60   *tagname*, if specified and not ``None``, is a string giving a tag name for
61   the mapping.  Windows allows you to have many different mappings against
62   the same file.  If you specify the name of an existing tag, that tag is
63   opened, otherwise a new tag of this name is created.  If this parameter is
64   omitted or ``None``, the mapping is created without a name.  Avoiding the
65   use of the tag parameter will assist in keeping your code portable between
66   Unix and Windows.
67
68   *offset* may be specified as a non-negative integer offset. mmap references
69   will be relative to the offset from the beginning of the file. *offset*
70   defaults to 0.  *offset* must be a multiple of the :const:`ALLOCATIONGRANULARITY`.
71
72   .. audit-event:: mmap.__new__ fileno,length,access,offset mmap.mmap
73
74.. class:: mmap(fileno, length, flags=MAP_SHARED, prot=PROT_WRITE|PROT_READ, access=ACCESS_DEFAULT[, offset])
75   :noindex:
76
77   **(Unix version)** Maps *length* bytes from the file specified by the file
78   descriptor *fileno*, and returns a mmap object.  If *length* is ``0``, the
79   maximum length of the map will be the current size of the file when
80   :class:`~mmap.mmap` is called.
81
82   *flags* specifies the nature of the mapping. :const:`MAP_PRIVATE` creates a
83   private copy-on-write mapping, so changes to the contents of the mmap
84   object will be private to this process, and :const:`MAP_SHARED` creates a
85   mapping that's shared with all other processes mapping the same areas of
86   the file.  The default value is :const:`MAP_SHARED`. Some systems have
87   additional possible flags with the full list specified in
88   :ref:`MAP_* constants <map-constants>`.
89
90   *prot*, if specified, gives the desired memory protection; the two most
91   useful values are :const:`PROT_READ` and :const:`PROT_WRITE`, to specify
92   that the pages may be read or written.  *prot* defaults to
93   :const:`PROT_READ \| PROT_WRITE`.
94
95   *access* may be specified in lieu of *flags* and *prot* as an optional
96   keyword parameter.  It is an error to specify both *flags*, *prot* and
97   *access*.  See the description of *access* above for information on how to
98   use this parameter.
99
100   *offset* may be specified as a non-negative integer offset. mmap references
101   will be relative to the offset from the beginning of the file. *offset*
102   defaults to 0. *offset* must be a multiple of :const:`ALLOCATIONGRANULARITY`
103   which is equal to :const:`PAGESIZE` on Unix systems.
104
105   To ensure validity of the created memory mapping the file specified
106   by the descriptor *fileno* is internally automatically synchronized
107   with the physical backing store on macOS.
108
109   This example shows a simple way of using :class:`~mmap.mmap`::
110
111      import mmap
112
113      # write a simple example file
114      with open("hello.txt", "wb") as f:
115          f.write(b"Hello Python!\n")
116
117      with open("hello.txt", "r+b") as f:
118          # memory-map the file, size 0 means whole file
119          mm = mmap.mmap(f.fileno(), 0)
120          # read content via standard file methods
121          print(mm.readline())  # prints b"Hello Python!\n"
122          # read content via slice notation
123          print(mm[:5])  # prints b"Hello"
124          # update content using slice notation;
125          # note that new content must have same size
126          mm[6:] = b" world!\n"
127          # ... and read again using standard file methods
128          mm.seek(0)
129          print(mm.readline())  # prints b"Hello  world!\n"
130          # close the map
131          mm.close()
132
133
134   :class:`~mmap.mmap` can also be used as a context manager in a :keyword:`with`
135   statement::
136
137      import mmap
138
139      with mmap.mmap(-1, 13) as mm:
140          mm.write(b"Hello world!")
141
142   .. versionadded:: 3.2
143      Context manager support.
144
145
146   The next example demonstrates how to create an anonymous map and exchange
147   data between the parent and child processes::
148
149      import mmap
150      import os
151
152      mm = mmap.mmap(-1, 13)
153      mm.write(b"Hello world!")
154
155      pid = os.fork()
156
157      if pid == 0:  # In a child process
158          mm.seek(0)
159          print(mm.readline())
160
161          mm.close()
162
163   .. audit-event:: mmap.__new__ fileno,length,access,offset mmap.mmap
164
165   Memory-mapped file objects support the following methods:
166
167   .. method:: close()
168
169      Closes the mmap. Subsequent calls to other methods of the object will
170      result in a ValueError exception being raised. This will not close
171      the open file.
172
173
174   .. attribute:: closed
175
176      ``True`` if the file is closed.
177
178      .. versionadded:: 3.2
179
180
181   .. method:: find(sub[, start[, end]])
182
183      Returns the lowest index in the object where the subsequence *sub* is
184      found, such that *sub* is contained in the range [*start*, *end*].
185      Optional arguments *start* and *end* are interpreted as in slice notation.
186      Returns ``-1`` on failure.
187
188      .. versionchanged:: 3.5
189         Writable :term:`bytes-like object` is now accepted.
190
191
192   .. method:: flush([offset[, size]])
193
194      Flushes changes made to the in-memory copy of a file back to disk. Without
195      use of this call there is no guarantee that changes are written back before
196      the object is destroyed.  If *offset* and *size* are specified, only
197      changes to the given range of bytes will be flushed to disk; otherwise, the
198      whole extent of the mapping is flushed.  *offset* must be a multiple of the
199      :const:`PAGESIZE` or :const:`ALLOCATIONGRANULARITY`.
200
201      ``None`` is returned to indicate success.  An exception is raised when the
202      call failed.
203
204      .. versionchanged:: 3.8
205         Previously, a nonzero value was returned on success; zero was returned
206         on error under Windows.  A zero value was returned on success; an
207         exception was raised on error under Unix.
208
209
210   .. method:: madvise(option[, start[, length]])
211
212      Send advice *option* to the kernel about the memory region beginning at
213      *start* and extending *length* bytes.  *option* must be one of the
214      :ref:`MADV_* constants <madvise-constants>` available on the system.  If
215      *start* and *length* are omitted, the entire mapping is spanned.  On
216      some systems (including Linux), *start* must be a multiple of the
217      :const:`PAGESIZE`.
218
219      Availability: Systems with the ``madvise()`` system call.
220
221      .. versionadded:: 3.8
222
223
224   .. method:: move(dest, src, count)
225
226      Copy the *count* bytes starting at offset *src* to the destination index
227      *dest*.  If the mmap was created with :const:`ACCESS_READ`, then calls to
228      move will raise a :exc:`TypeError` exception.
229
230
231   .. method:: read([n])
232
233      Return a :class:`bytes` containing up to *n* bytes starting from the
234      current file position. If the argument is omitted, ``None`` or negative,
235      return all bytes from the current file position to the end of the
236      mapping. The file position is updated to point after the bytes that were
237      returned.
238
239      .. versionchanged:: 3.3
240         Argument can be omitted or ``None``.
241
242   .. method:: read_byte()
243
244      Returns a byte at the current file position as an integer, and advances
245      the file position by 1.
246
247
248   .. method:: readline()
249
250      Returns a single line, starting at the current file position and up to the
251      next newline. The file position is updated to point after the bytes that were
252      returned.
253
254
255   .. method:: resize(newsize)
256
257      Resizes the map and the underlying file, if any. If the mmap was created
258      with :const:`ACCESS_READ` or :const:`ACCESS_COPY`, resizing the map will
259      raise a :exc:`TypeError` exception.
260
261      **On Windows**: Resizing the map will raise an :exc:`OSError` if there are other
262      maps against the same named file. Resizing an anonymous map (ie against the
263      pagefile) will silently create a new map with the original data copied over
264      up to the length of the new size.
265
266      .. versionchanged:: 3.11
267         Correctly fails if attempting to resize when another map is held
268         Allows resize against an anonymous map on Windows
269
270   .. method:: rfind(sub[, start[, end]])
271
272      Returns the highest index in the object where the subsequence *sub* is
273      found, such that *sub* is contained in the range [*start*, *end*].
274      Optional arguments *start* and *end* are interpreted as in slice notation.
275      Returns ``-1`` on failure.
276
277      .. versionchanged:: 3.5
278         Writable :term:`bytes-like object` is now accepted.
279
280
281   .. method:: seek(pos[, whence])
282
283      Set the file's current position.  *whence* argument is optional and
284      defaults to ``os.SEEK_SET`` or ``0`` (absolute file positioning); other
285      values are ``os.SEEK_CUR`` or ``1`` (seek relative to the current
286      position) and ``os.SEEK_END`` or ``2`` (seek relative to the file's end).
287
288
289   .. method:: size()
290
291      Return the length of the file, which can be larger than the size of the
292      memory-mapped area.
293
294
295   .. method:: tell()
296
297      Returns the current position of the file pointer.
298
299
300   .. method:: write(bytes)
301
302      Write the bytes in *bytes* into memory at the current position of the
303      file pointer and return the number of bytes written (never less than
304      ``len(bytes)``, since if the write fails, a :exc:`ValueError` will be
305      raised).  The file position is updated to point after the bytes that
306      were written.  If the mmap was created with :const:`ACCESS_READ`, then
307      writing to it will raise a :exc:`TypeError` exception.
308
309      .. versionchanged:: 3.5
310         Writable :term:`bytes-like object` is now accepted.
311
312      .. versionchanged:: 3.6
313         The number of bytes written is now returned.
314
315
316   .. method:: write_byte(byte)
317
318      Write the integer *byte* into memory at the current
319      position of the file pointer; the file position is advanced by ``1``. If
320      the mmap was created with :const:`ACCESS_READ`, then writing to it will
321      raise a :exc:`TypeError` exception.
322
323.. _madvise-constants:
324
325MADV_* Constants
326++++++++++++++++
327
328.. data:: MADV_NORMAL
329          MADV_RANDOM
330          MADV_SEQUENTIAL
331          MADV_WILLNEED
332          MADV_DONTNEED
333          MADV_REMOVE
334          MADV_DONTFORK
335          MADV_DOFORK
336          MADV_HWPOISON
337          MADV_MERGEABLE
338          MADV_UNMERGEABLE
339          MADV_SOFT_OFFLINE
340          MADV_HUGEPAGE
341          MADV_NOHUGEPAGE
342          MADV_DONTDUMP
343          MADV_DODUMP
344          MADV_FREE
345          MADV_NOSYNC
346          MADV_AUTOSYNC
347          MADV_NOCORE
348          MADV_CORE
349          MADV_PROTECT
350          MADV_FREE_REUSABLE
351          MADV_FREE_REUSE
352
353   These options can be passed to :meth:`mmap.madvise`.  Not every option will
354   be present on every system.
355
356   Availability: Systems with the madvise() system call.
357
358   .. versionadded:: 3.8
359
360.. _map-constants:
361
362MAP_* Constants
363+++++++++++++++
364
365.. data:: MAP_SHARED
366          MAP_PRIVATE
367          MAP_DENYWRITE
368          MAP_EXECUTABLE
369          MAP_ANON
370          MAP_ANONYMOUS
371          MAP_POPULATE
372          MAP_STACK
373
374    These are the various flags that can be passed to :meth:`mmap.mmap`. Note that some options might not be present on some systems.
375
376    .. versionchanged:: 3.10
377       Added MAP_POPULATE constant.
378
379    .. versionadded:: 3.11
380       Added MAP_STACK constant.
381