Using Python on macOS

5. Using Python on macOS¶
This document aims to give an overview of macOS-specific behavior you should know about to get started with Python on Mac computers. Python on a Mac running macOS is very similar to Python on other Unix-derived platforms, but there are some differences in installation and some features.
There are various ways to obtain and install Python for macOS. Pre-built versions of the most recent versions of Python are available from a number of distributors. Much of this document describes use of the Pythons provided by the CPython release team for download from the python.org website. See Alternative Distributions for some other options.
5.1. Using Python for macOS from python.org
¶
5.1.1. Installation steps¶
For current Python versions
(other than those in security
status), the release team produces a
Python for macOS installer package for each new release.
A list of available installers
is available here.
We recommend using the most recent supported Python version where possible.
Current installers provide a
universal2 binary build
of Python which runs natively on all Macs (Apple Silicon and Intel) that are
supported by a wide range of macOS versions,
currently typically from at least macOS 10.15 Catalina on.
The downloaded file is a standard macOS installer package file (.pkg
).
File integrity information (checksum, size, sigstore signature, etc) for each file is included
on the release download page. Installer packages and their contents are signed and notarized
with Python Software Foundation
Apple Developer ID certificates
to meet macOS Gatekeeper requirements.
For a default installation, double-click on the downloaded installer package file. This should launch the standard macOS Installer app and display the first of several installer windows steps.
Clicking on the Continue button brings up the Read Me for this installer.
Besides other important information, the Read Me documents which Python version is
going to be installed and on what versions of macOS it is supported. You may need
to scroll through to read the whole file. By default, this Read Me will also be
installed in /Applications/Python 3.14/
and available to read anytime.
Clicking on Continue proceeds to display the license for Python and for other included software. You will then need to Agree to the license terms before proceeding to the next step. This license file will also be installed and available to be read later.
After the license terms are accepted, the next step is the Installation Type display. For most uses, the standard set of installation operations is appropriate.
By pressing the Customize button, you can choose to omit or select certain package components of the installer. Click on each package name to see a description of what it installs. To also install support for the optional free-threaded feature, see Installing Free-threaded Binaries.
In either case, clicking Install will begin the install process by asking
permission to install new software. A macOS user name with Administrator
privilege
is needed as the installed Python will be available to all users of the Mac.
When the installation is complete, the Summary window will appear.
Double-click on the Install Certificates.command
icon or file in the /Applications/Python 3.14/
window to complete the
installation.
This will open a temporary Terminal shell window that will use the new Python to download and install SSL root certificates for its use.
If Successfully installed certifi
and update complete
appears
in the terminal window, the installation is complete.
Close this terminal window and the installer window.
A default install will include:
A
Python 3.14
folder in yourApplications
folder. In here you find IDLE, the development environment that is a standard part of official Python distributions; and Python Launcher, which handles double-clicking Python scripts from the macOS Finder.A framework
/Library/Frameworks/Python.framework
, which includes the Python executable and libraries. The installer adds this location to your shell path. To uninstall Python, you can remove these three things. Symlinks to the Python executable are placed in/usr/local/bin/
.
Note
Recent versions of macOS include a python3 command in /usr/bin/python3
that links to a usually older and incomplete version of Python provided by and for use by
the Apple development tools, Xcode or the Command Line Tools for Xcode.
You should never modify or attempt to delete this installation, as it is
Apple-controlled and is used by Apple-provided or third-party software. If
you choose to install a newer Python version from python.org
, you will have
two different but functional Python installations on your computer that
can co-exist. The default installer options should ensure that its python3
will be used instead of the system python3.
5.1.2. How to run a Python script¶
There are two ways to invoke the Python interpreter.
If you are familiar with using a Unix shell in a terminal
window, you can invoke python3.14
or python3
optionally
followed by one or more command line options (described in Command line and environment).
The Python tutorial also has a useful section on
using Python interactively from a shell.
You can also invoke the interpreter through an integrated development environment. IDLE — Python editor and shell is a basic editor and interpreter environment which is included with the standard distribution of Python. IDLE includes a Help menu that allows you to access Python documentation. If you are completely new to Python, you can read the tutorial introduction in that document.
There are many other editors and IDEs available, see Editors and IDEs for more information.
To run a Python script file from the terminal window, you can invoke the interpreter with the name of the script file:
python3.14
myscript.py
To run your script from the Finder, you can either:
Drag it to Python Launcher.
Select Python Launcher as the default application to open your script (or any
.py
script) through the Finder Info window and double-click it. Python Launcher has various preferences to control how your script is launched. Option-dragging allows you to change these for one invocation, or use itsPreferences
menu to change things globally.
Be aware that running the script directly from the macOS Finder might produce different results than when running from a terminal window as the script will not be run in the usual shell environment including any setting of environment variables in shell profiles. And, as with any other script or program, be certain of what you are about to run.
5.2. Alternative Distributions¶
Besides the standard python.org
for macOS installer, there are third-party
distributions for macOS that may include additional functionality.
Some popular distributions and their key features:
- ActivePython
Installer with multi-platform compatibility, documentation
- Anaconda
Popular scientific modules (such as numpy, scipy, and pandas) and the
conda
package manager.- Homebrew
Package manager for macOS including multiple versions of Python and many third-party Python-based packages (including numpy, scipy, and pandas).
- MacPorts
Another package manager for macOS including multiple versions of Python and many third-party Python-based packages. May include pre-built versions of Python and many packages for older versions of macOS.
Note that distributions might not include the latest versions of Python or other libraries, and are not maintained or supported by the core Python team.
5.3. Installing Additional Python Packages¶
Refer to the Python Packaging User Guide for more information.
5.4. GUI Programming¶
There are several options for building GUI applications on the Mac with Python.
The standard Python GUI toolkit is tkinter
, based on the cross-platform
Tk toolkit (https://www.tcl.tk). A macOS-native version of Tk is included with
the installer.
PyObjC is a Python binding to Apple’s Objective-C/Cocoa framework. Information on PyObjC is available from pyobjc.
A number of alternative macOS GUI toolkits are available including:
PySide: Official Python bindings to the Qt GUI toolkit.
PyQt: Alternative Python bindings to Qt.
Kivy: A cross-platform GUI toolkit that supports desktop and mobile platforms.
Toga: Part of the BeeWare Project; supports desktop, mobile, web and console apps.
wxPython: A cross-platform toolkit that supports desktop operating systems.
5.5. Advanced Topics¶
5.5.1. Installing Free-threaded Binaries¶
Added in version 3.13.
The python.org
Python for macOS
installer package can optionally install an additional build of
Python 3.14 that supports PEP 703, the free-threading feature
(running with the global interpreter lock disabled).
Check the release page on python.org
for possible updated information.
The free-threaded mode is working and continues to be improved, but there is some additional overhead in single-threaded workloads compared to the regular build. Additionally, third-party packages, in particular ones with an extension module, may not be ready for use in a free-threaded build, and will re-enable the GIL. Therefore, the support for free-threading is not installed by default. It is packaged as a separate install option, available by clicking the Customize button on the Installation Type step of the installer as described above.
If the box next to the Free-threaded Python package name is checked,
a separate PythonT.framework
will also be installed
alongside the normal Python.framework
in /Library/Frameworks
.
This configuration allows a free-threaded Python 3.14 build to co-exist
on your system with a traditional (GIL only) Python 3.14 build with
minimal risk while installing or testing. This installation layout may
change in future releases.
Known cautions and limitations:
The UNIX command-line tools package, which is selected by default, will install links in
/usr/local/bin
forpython3.14t
, the free-threaded interpreter, andpython3.14t-config
, a configuration utility which may be useful for package builders. Since/usr/local/bin
is typically included in your shellPATH
, in most cases no changes to yourPATH
environment variables should be needed to usepython3.14t
.For this release, the Shell profile updater package and the
Update Shell Profile.command
in/Applications/Python 3.14/
do not support the free-threaded package.The free-threaded build and the traditional build have separate search paths and separate
site-packages
directories so, by default, if you need a package available in both builds, it may need to be installed in both. The free-threaded package will install a separate instance of pip for use withpython3.14t
.To install a package using pip without a venv:
python3.14t -m pip install <package_name>
When working with multiple Python environments, it is usually safest and easiest to create and use virtual environments. This can avoid possible command name conflicts and confusion about which Python is in use:
python3.14t -m venv <venv_name>
then activate.
To run a free-threaded version of IDLE:
python3.14t -m idlelib
The interpreters in both builds respond to the same PYTHON environment variables which may have unexpected results, for example, if you have
PYTHONPATH
set in a shell profile. If necessary, there are command line options like-E
to ignore these environment variables.The free-threaded build links to the third-party shared libraries, such as
OpenSSL
andTk
, installed in the traditional framework. This means that both builds also share one set of trust certificates as installed by the Install Certificates.command script, thus it only needs to be run once.If you cannot depend on the link in
/usr/local/bin
pointing to thepython.org
free-threadedpython3.14t
(for example, if you want to install your own version there or some other distribution does), you can explicitly set your shellPATH
environment variable to include thePythonT
frameworkbin
directory:export PATH="/Library/Frameworks/PythonT.framework/Versions/3.14/bin":"$PATH"
The traditional framework installation by default does something similar, except for
Python.framework
. Be aware that having both frameworkbin
directories inPATH
can lead to confusion if there are duplicate names likepython3.14
in both; which one is actually used depends on the order they appear inPATH
. Thewhich python3.x
orwhich python3.xt
commands can show which path is being used. Using virtual environments can help avoid such ambiguities. Another option might be to create a shell alias to the desired interpreter, like:alias py3.14="/Library/Frameworks/Python.framework/Versions/3.14/bin/python3.14" alias py3.14t="/Library/Frameworks/PythonT.framework/Versions/3.14/bin/python3.14t"
5.5.2. Installing using the command line¶
If you want to use automation to install the python.org
installer package
(rather than by using the familiar macOS Installer GUI app),
the macOS command line installer utility lets you select non-default
options, too. If you are not familiar with installer, it can be
somewhat cryptic (see man installer for more information).
As an example, the following shell snippet shows one way to do it,
using the 3.14.0b2
release and selecting the free-threaded interpreter
option:
RELEASE="python-3.140b2-macos11.pkg" # download installer pkg curl -O https://www.python.org/ftp/python/3.14.0/${RELEASE} # create installer choicechanges to customize the install: # enable the PythonTFramework-3.14 package # while accepting the other defaults (install all other packages) cat > ./choicechanges.plist <<EOF <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <array> <dict> <key>attributeSetting</key> <integer>1</integer> <key>choiceAttribute</key> <string>selected</string> <key>choiceIdentifier</key> <string>org.python.Python.PythonTFramework-3.14</string> </dict> </array> </plist> EOF sudo installer -pkg ./${RELEASE} -applyChoiceChangesXML ./choicechanges.plist -target /
You can then test that both installer builds are now available with something like:
$ # test that the free-threaded interpreter was installed if the Unix Command Tools package was enabled $ /usr/local/bin/python3.14t -VV Python 3.14.0b2 free-threading build (v3.14.0b2:3a83b172af, Jun 5 2024, 12:57:31) [Clang 15.0.0 (clang-1500.3.9.4)] $ # and the traditional interpreter $ /usr/local/bin/python3.14 -VV Python 3.14.0b2 (v3.14.0b2:3a83b172af, Jun 5 2024, 12:50:24) [Clang 15.0.0 (clang-1500.3.9.4)] $ # test that they are also available without the prefix if /usr/local/bin is on $PATH $ python3.14t -VV Python 3.14.0b2 free-threading build (v3.14.0b2:3a83b172af, Jun 5 2024, 12:57:31) [Clang 15.0.0 (clang-1500.3.9.4)] $ python3.14 -VV Python 3.14.0b2 (v3.14.0b2:3a83b172af, Jun 5 2024, 12:50:24) [Clang 15.0.0 (clang-1500.3.9.4)]
Note
Current python.org
installers only install to fixed locations like
/Library/Frameworks/
, /Applications
, and /usr/local/bin
.
You cannot use the installer -domain
option to install to
other locations.
5.5.3. Distributing Python Applications¶
A range of tools exist for converting your Python code into a standalone distributable application:
py2app: Supports creating macOS
.app
bundles from a Python project.Briefcase: Part of the BeeWare Project; a cross-platform packaging tool that supports creation of
.app
bundles on macOS, as well as managing signing and notarization.PyInstaller: A cross-platform packaging tool that creates a single file or folder as a distributable artifact.
5.5.4. App Store Compliance¶
Apps submitted for distribution through the macOS App Store must pass Apple’s app review process. This process includes a set of automated validation rules that inspect the submitted application bundle for problematic code.
The Python standard library contains some code that is known to violate these automated rules. While these violations appear to be false positives, Apple’s review rules cannot be challenged. Therefore, it is necessary to modify the Python standard library for an app to pass App Store review.
The Python source tree contains
a patch file that will remove
all code that is known to cause issues with the App Store review process. This
patch is applied automatically when CPython is configured with the
--with-app-store-compliance
option.
This patch is not normally required to use CPython on a Mac; nor is it required if you are distributing an app outside the macOS App Store. It is only required if you are using the macOS App Store as a distribution channel.
5.6. Other Resources¶
The python.org Help page has links to many useful resources. The Pythonmac-SIG mailing list is another support resource specifically for Python users and developers on the Mac.


Exception Handling

Exception Handling¶
The functions described in this chapter will let you handle and raise Python
exceptions. It is important to understand some of the basics of Python
exception handling. It works somewhat like the POSIX errno
variable:
there is a global indicator (per thread) of the last error that occurred. Most
C API functions don’t clear this on success, but will set it to indicate the
cause of the error on failure. Most C API functions also return an error
indicator, usually NULL
if they are supposed to return a pointer, or -1
if they return an integer (exception: the PyArg_*
functions
return 1
for success and 0
for failure).
Concretely, the error indicator consists of three object pointers: the
exception’s type, the exception’s value, and the traceback object. Any
of those pointers can be NULL
if non-set (although some combinations are
forbidden, for example you can’t have a non-NULL
traceback if the exception
type is NULL
).
When a function must fail because some function it called failed, it generally doesn’t set the error indicator; the function it called already set it. It is responsible for either handling the error and clearing the exception or returning after cleaning up any resources it holds (such as object references or memory allocations); it should not continue normally if it is not prepared to handle the error. If returning due to an error, it is important to indicate to the caller that an error has been set. If the error is not handled or carefully propagated, additional calls into the Python/C API may not behave as intended and may fail in mysterious ways.
Note
The error indicator is not the result of sys.exc_info()
.
The former corresponds to an exception that is not yet caught (and is
therefore still propagating), while the latter returns an exception after
it is caught (and has therefore stopped propagating).
Printing and clearing¶
-
void PyErr_Clear()¶
- Part of the Stable ABI.
Clear the error indicator. If the error indicator is not set, there is no effect.
-
void PyErr_PrintEx(int set_sys_last_vars)¶
- Part of the Stable ABI.
Print a standard traceback to
sys.stderr
and clear the error indicator. Unless the error is aSystemExit
, in that case no traceback is printed and the Python process will exit with the error code specified by theSystemExit
instance.Call this function only when the error indicator is set. Otherwise it will cause a fatal error!
If set_sys_last_vars is nonzero, the variable
sys.last_exc
is set to the printed exception. For backwards compatibility, the deprecated variablessys.last_type
,sys.last_value
andsys.last_traceback
are also set to the type, value and traceback of this exception, respectively.Changed in version 3.12: The setting of
sys.last_exc
was added.
-
void PyErr_Print()¶
- Part of the Stable ABI.
Alias for
PyErr_PrintEx(1)
.
-
void PyErr_WriteUnraisable(PyObject *obj)¶
- Part of the Stable ABI.
Call
sys.unraisablehook()
using the current exception and obj argument.This utility function prints a warning message to
sys.stderr
when an exception has been set but it is impossible for the interpreter to actually raise the exception. It is used, for example, when an exception occurs in an__del__()
method.The function is called with a single argument obj that identifies the context in which the unraisable exception occurred. If possible, the repr of obj will be printed in the warning message. If obj is
NULL
, only the traceback is printed.An exception must be set when calling this function.
Changed in version 3.4: Print a traceback. Print only traceback if obj is
NULL
.Changed in version 3.8: Use
sys.unraisablehook()
.
-
void PyErr_FormatUnraisable(const char *format, ...)¶
Similar to
PyErr_WriteUnraisable()
, but the format and subsequent parameters help format the warning message; they have the same meaning and values as inPyUnicode_FromFormat()
.PyErr_WriteUnraisable(obj)
is roughly equivalent toPyErr_FormatUnraisable("Exception ignored in: %R", obj)
. If format isNULL
, only the traceback is printed.Added in version 3.13.
-
void PyErr_DisplayException(PyObject *exc)¶
- Part of the Stable ABI since version 3.12.
Print the standard traceback display of
exc
tosys.stderr
, including chained exceptions and notes.Added in version 3.12.
Raising exceptions¶
These functions help you set the current thread’s error indicator.
For convenience, some of these functions will always return a
NULL
pointer for use in a return
statement.
-
void PyErr_SetString(PyObject *type, const char *message)¶
- Part of the Stable ABI.
This is the most common way to set the error indicator. The first argument specifies the exception type; it is normally one of the standard exceptions, e.g.
PyExc_RuntimeError
. You need not create a new strong reference to it (e.g. withPy_INCREF()
). The second argument is an error message; it is decoded from'utf-8'
.
-
void PyErr_SetObject(PyObject *type, PyObject *value)¶
- Part of the Stable ABI.
This function is similar to
PyErr_SetString()
but lets you specify an arbitrary Python object for the “value” of the exception.
-
PyObject *PyErr_Format(PyObject *exception, const char *format, ...)¶
- Return value: Always NULL. Part of the Stable ABI.
This function sets the error indicator and returns
NULL
. exception should be a Python exception class. The format and subsequent parameters help format the error message; they have the same meaning and values as inPyUnicode_FromFormat()
. format is an ASCII-encoded string.
-
PyObject *PyErr_FormatV(PyObject *exception, const char *format, va_list vargs)¶
- Return value: Always NULL. Part of the Stable ABI since version 3.5.
Same as
PyErr_Format()
, but taking ava_list
argument rather than a variable number of arguments.Added in version 3.5.
-
void PyErr_SetNone(PyObject *type)¶
- Part of the Stable ABI.
This is a shorthand for
PyErr_SetObject(type, Py_None)
.
-
int PyErr_BadArgument()¶
- Part of the Stable ABI.
This is a shorthand for
PyErr_SetString(PyExc_TypeError, message)
, where message indicates that a built-in operation was invoked with an illegal argument. It is mostly for internal use.
-
PyObject *PyErr_NoMemory()¶
- Return value: Always NULL. Part of the Stable ABI.
This is a shorthand for
PyErr_SetNone(PyExc_MemoryError)
; it returnsNULL
so an object allocation function can writereturn PyErr_NoMemory();
when it runs out of memory.
-
PyObject *PyErr_SetFromErrno(PyObject *type)¶
- Return value: Always NULL. Part of the Stable ABI.
This is a convenience function to raise an exception when a C library function has returned an error and set the C variable
errno
. It constructs a tuple object whose first item is the integererrno
value and whose second item is the corresponding error message (gotten fromstrerror()
), and then callsPyErr_SetObject(type, object)
. On Unix, when theerrno
value isEINTR
, indicating an interrupted system call, this callsPyErr_CheckSignals()
, and if that set the error indicator, leaves it set to that. The function always returnsNULL
, so a wrapper function around a system call can writereturn PyErr_SetFromErrno(type);
when the system call returns an error.
-
PyObject *PyErr_SetFromErrnoWithFilenameObject(PyObject *type, PyObject *filenameObject)¶
- Return value: Always NULL. Part of the Stable ABI.
Similar to
PyErr_SetFromErrno()
, with the additional behavior that if filenameObject is notNULL
, it is passed to the constructor of type as a third parameter. In the case ofOSError
exception, this is used to define thefilename
attribute of the exception instance.
-
PyObject *PyErr_SetFromErrnoWithFilenameObjects(PyObject *type, PyObject *filenameObject, PyObject *filenameObject2)¶
- Return value: Always NULL. Part of the Stable ABI since version 3.7.
Similar to
PyErr_SetFromErrnoWithFilenameObject()
, but takes a second filename object, for raising errors when a function that takes two filenames fails.Added in version 3.4.
-
PyObject *PyErr_SetFromErrnoWithFilename(PyObject *type, const char *filename)¶
- Return value: Always NULL. Part of the Stable ABI.
Similar to
PyErr_SetFromErrnoWithFilenameObject()
, but the filename is given as a C string. filename is decoded from the filesystem encoding and error handler.
-
PyObject *PyErr_SetFromWindowsErr(int ierr)¶
- Return value: Always NULL. Part of the Stable ABI on Windows since version 3.7.
This is a convenience function to raise
OSError
. If called with ierr of0
, the error code returned by a call toGetLastError()
is used instead. It calls the Win32 functionFormatMessage()
to retrieve the Windows description of error code given by ierr orGetLastError()
, then it constructs aOSError
object with thewinerror
attribute set to the error code, thestrerror
attribute set to the corresponding error message (gotten fromFormatMessage()
), and then callsPyErr_SetObject(PyExc_OSError, object)
. This function always returnsNULL
.Availability: Windows.
-
PyObject *PyErr_SetExcFromWindowsErr(PyObject *type, int ierr)¶
- Return value: Always NULL. Part of the Stable ABI on Windows since version 3.7.
Similar to
PyErr_SetFromWindowsErr()
, with an additional parameter specifying the exception type to be raised.Availability: Windows.
-
PyObject *PyErr_SetFromWindowsErrWithFilename(int ierr, const char *filename)¶
- Return value: Always NULL. Part of the Stable ABI on Windows since version 3.7.
Similar to
PyErr_SetFromWindowsErr()
, with the additional behavior that if filename is notNULL
, it is decoded from the filesystem encoding (os.fsdecode()
) and passed to the constructor ofOSError
as a third parameter to be used to define thefilename
attribute of the exception instance.Availability: Windows.
-
PyObject *PyErr_SetExcFromWindowsErrWithFilenameObject(PyObject *type, int ierr, PyObject *filename)¶
- Return value: Always NULL. Part of the Stable ABI on Windows since version 3.7.
Similar to
PyErr_SetExcFromWindowsErr()
, with the additional behavior that if filename is notNULL
, it is passed to the constructor ofOSError
as a third parameter to be used to define thefilename
attribute of the exception instance.Availability: Windows.
-
PyObject *PyErr_SetExcFromWindowsErrWithFilenameObjects(PyObject *type, int ierr, PyObject *filename, PyObject *filename2)¶
- Return value: Always NULL. Part of the Stable ABI on Windows since version 3.7.
Similar to
PyErr_SetExcFromWindowsErrWithFilenameObject()
, but accepts a second filename object.Availability: Windows.
Added in version 3.4.
-
PyObject *PyErr_SetExcFromWindowsErrWithFilename(PyObject *type, int ierr, const char *filename)¶
- Return value: Always NULL. Part of the Stable ABI on Windows since version 3.7.
Similar to
PyErr_SetFromWindowsErrWithFilename()
, with an additional parameter specifying the exception type to be raised.Availability: Windows.
-
PyObject *PyErr_SetImportError(PyObject *msg, PyObject *name, PyObject *path)¶
- Return value: Always NULL. Part of the Stable ABI since version 3.7.
This is a convenience function to raise
ImportError
. msg will be set as the exception’s message string. name and path, both of which can beNULL
, will be set as theImportError
’s respectivename
andpath
attributes.Added in version 3.3.
-
PyObject *PyErr_SetImportErrorSubclass(PyObject *exception, PyObject *msg, PyObject *name, PyObject *path)¶
- Return value: Always NULL. Part of the Stable ABI since version 3.6.
Much like
PyErr_SetImportError()
but this function allows for specifying a subclass ofImportError
to raise.Added in version 3.6.
-
void PyErr_SyntaxLocationObject(PyObject *filename, int lineno, int col_offset)¶
Set file, line, and offset information for the current exception. If the current exception is not a
SyntaxError
, then it sets additional attributes, which make the exception printing subsystem think the exception is aSyntaxError
.Added in version 3.4.
-
void PyErr_RangedSyntaxLocationObject(PyObject *filename, int lineno, int col_offset, int end_lineno, int end_col_offset)¶
Similar to
PyErr_SyntaxLocationObject()
, but also sets the end_lineno and end_col_offset information for the current exception.Added in version 3.10.
-
void PyErr_SyntaxLocationEx(const char *filename, int lineno, int col_offset)¶
- Part of the Stable ABI since version 3.7.
Like
PyErr_SyntaxLocationObject()
, but filename is a byte string decoded from the filesystem encoding and error handler.Added in version 3.2.
-
void PyErr_SyntaxLocation(const char *filename, int lineno)¶
- Part of the Stable ABI.
Like
PyErr_SyntaxLocationEx()
, but the col_offset parameter is omitted.
-
void PyErr_BadInternalCall()¶
- Part of the Stable ABI.
This is a shorthand for
PyErr_SetString(PyExc_SystemError, message)
, where message indicates that an internal operation (e.g. a Python/C API function) was invoked with an illegal argument. It is mostly for internal use.
-
PyObject *PyErr_ProgramTextObject(PyObject *filename, int lineno)¶
Get the source line in filename at line lineno. filename should be a Python
str
object.On success, this function returns a Python string object with the found line. On failure, this function returns
NULL
without an exception set.
-
PyObject *PyErr_ProgramText(const char *filename, int lineno)¶
- Part of the Stable ABI.
Similar to
PyErr_ProgramTextObject()
, but filename is a const char*, which is decoded with the filesystem encoding and error handler, instead of a Python object reference.
Issuing warnings¶
Use these functions to issue warnings from C code. They mirror similar
functions exported by the Python warnings
module. They normally
print a warning message to sys.stderr; however, it is
also possible that the user has specified that warnings are to be turned into
errors, and in that case they will raise an exception. It is also possible that
the functions raise an exception because of a problem with the warning machinery.
The return value is 0
if no exception is raised, or -1
if an exception
is raised. (It is not possible to determine whether a warning message is
actually printed, nor what the reason is for the exception; this is
intentional.) If an exception is raised, the caller should do its normal
exception handling (for example, Py_DECREF()
owned references and return
an error value).
-
int PyErr_WarnEx(PyObject *category, const char *message, Py_ssize_t stack_level)¶
- Part of the Stable ABI.
Issue a warning message. The category argument is a warning category (see below) or
NULL
; the message argument is a UTF-8 encoded string. stack_level is a positive number giving a number of stack frames; the warning will be issued from the currently executing line of code in that stack frame. A stack_level of 1 is the function callingPyErr_WarnEx()
, 2 is the function above that, and so forth.Warning categories must be subclasses of
PyExc_Warning
;PyExc_Warning
is a subclass ofPyExc_Exception
; the default warning category isPyExc_RuntimeWarning
. The standard Python warning categories are available as global variables whose names are enumerated at Warning types.For information about warning control, see the documentation for the
warnings
module and the-W
option in the command line documentation. There is no C API for warning control.
-
int PyErr_WarnExplicitObject(PyObject *category, PyObject *message, PyObject *filename, int lineno, PyObject *module, PyObject *registry)¶
Issue a warning message with explicit control over all warning attributes. This is a straightforward wrapper around the Python function
warnings.warn_explicit()
; see there for more information. The module and registry arguments may be set toNULL
to get the default effect described there.Added in version 3.4.
-
int PyErr_WarnExplicit(PyObject *category, const char *message, const char *filename, int lineno, const char *module, PyObject *registry)¶
- Part of the Stable ABI.
Similar to
PyErr_WarnExplicitObject()
except that message and module are UTF-8 encoded strings, and filename is decoded from the filesystem encoding and error handler.
-
int PyErr_WarnFormat(PyObject *category, Py_ssize_t stack_level, const char *format, ...)¶
- Part of the Stable ABI.
Function similar to
PyErr_WarnEx()
, but usePyUnicode_FromFormat()
to format the warning message. format is an ASCII-encoded string.Added in version 3.2.
-
int PyErr_WarnExplicitFormat(PyObject *category, const char *filename, int lineno, const char *module, PyObject *registry, const char *format, ...)¶
Similar to
PyErr_WarnExplicit()
, but usesPyUnicode_FromFormat()
to format the warning message. format is an ASCII-encoded string.Added in version 3.2.
-
int PyErr_ResourceWarning(PyObject *source, Py_ssize_t stack_level, const char *format, ...)¶
- Part of the Stable ABI since version 3.6.
Function similar to
PyErr_WarnFormat()
, but category isResourceWarning
and it passes source towarnings.WarningMessage
.Added in version 3.6.
Querying the error indicator¶
-
PyObject *PyErr_Occurred()¶
- Return value: Borrowed reference. Part of the Stable ABI.
Test whether the error indicator is set. If set, return the exception type (the first argument to the last call to one of the
PyErr_Set*
functions or toPyErr_Restore()
). If not set, returnNULL
. You do not own a reference to the return value, so you do not need toPy_DECREF()
it.The caller must have an attached thread state.
Note
Do not compare the return value to a specific exception; use
PyErr_ExceptionMatches()
instead, shown below. (The comparison could easily fail since the exception may be an instance instead of a class, in the case of a class exception, or it may be a subclass of the expected exception.)
-
int PyErr_ExceptionMatches(PyObject *exc)¶
- Part of the Stable ABI.
Equivalent to
PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)
. This should only be called when an exception is actually set; a memory access violation will occur if no exception has been raised.
-
int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc)¶
- Part of the Stable ABI.
Return true if the given exception matches the exception type in exc. If exc is a class object, this also returns true when given is an instance of a subclass. If exc is a tuple, all exception types in the tuple (and recursively in subtuples) are searched for a match.
-
PyObject *PyErr_GetRaisedException(void)¶
- Return value: New reference. Part of the Stable ABI since version 3.12.
Return the exception currently being raised, clearing the error indicator at the same time. Return
NULL
if the error indicator is not set.This function is used by code that needs to catch exceptions, or code that needs to save and restore the error indicator temporarily.
For example:
{ PyObject *exc = PyErr_GetRaisedException(); /* ... code that might produce other errors ... */ PyErr_SetRaisedException(exc); }
See also
PyErr_GetHandledException()
, to save the exception currently being handled.Added in version 3.12.
-
void PyErr_SetRaisedException(PyObject *exc)¶
- Part of the Stable ABI since version 3.12.
Set exc as the exception currently being raised, clearing the existing exception if one is set.
Warning
This call steals a reference to exc, which must be a valid exception.
Added in version 3.12.
-
void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback)¶
- Part of the Stable ABI.
Deprecated since version 3.12: Use
PyErr_GetRaisedException()
instead.Retrieve the error indicator into three variables whose addresses are passed. If the error indicator is not set, set all three variables to
NULL
. If it is set, it will be cleared and you own a reference to each object retrieved. The value and traceback object may beNULL
even when the type object is not.Note
This function is normally only used by legacy code that needs to catch exceptions or save and restore the error indicator temporarily.
For example:
{ PyObject *type, *value, *traceback; PyErr_Fetch(&type, &value, &traceback); /* ... code that might produce other errors ... */ PyErr_Restore(type, value, traceback); }
-
void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback)¶
- Part of the Stable ABI.
Deprecated since version 3.12: Use
PyErr_SetRaisedException()
instead.Set the error indicator from the three objects, type, value, and traceback, clearing the existing exception if one is set. If the objects are
NULL
, the error indicator is cleared. Do not pass aNULL
type and non-NULL
value or traceback. The exception type should be a class. Do not pass an invalid exception type or value. (Violating these rules will cause subtle problems later.) This call takes away a reference to each object: you must own a reference to each object before the call and after the call you no longer own these references. (If you don’t understand this, don’t use this function. I warned you.)Note
This function is normally only used by legacy code that needs to save and restore the error indicator temporarily. Use
PyErr_Fetch()
to save the current error indicator.
-
void PyErr_NormalizeException(PyObject **exc, PyObject **val, PyObject **tb)¶
- Part of the Stable ABI.
Deprecated since version 3.12: Use
PyErr_GetRaisedException()
instead, to avoid any possible de-normalization.Under certain circumstances, the values returned by
PyErr_Fetch()
below can be “unnormalized”, meaning that*exc
is a class object but*val
is not an instance of the same class. This function can be used to instantiate the class in that case. If the values are already normalized, nothing happens. The delayed normalization is implemented to improve performance.Note
This function does not implicitly set the
__traceback__
attribute on the exception value. If setting the traceback appropriately is desired, the following additional snippet is needed:if (tb != NULL) { PyException_SetTraceback(val, tb); }
-
PyObject *PyErr_GetHandledException(void)¶
- Part of the Stable ABI since version 3.11.
Retrieve the active exception instance, as would be returned by
sys.exception()
. This refers to an exception that was already caught, not to an exception that was freshly raised. Returns a new reference to the exception orNULL
. Does not modify the interpreter’s exception state.Note
This function is not normally used by code that wants to handle exceptions. Rather, it can be used when code needs to save and restore the exception state temporarily. Use
PyErr_SetHandledException()
to restore or clear the exception state.Added in version 3.11.
-
void PyErr_SetHandledException(PyObject *exc)¶
- Part of the Stable ABI since version 3.11.
Set the active exception, as known from
sys.exception()
. This refers to an exception that was already caught, not to an exception that was freshly raised. To clear the exception state, passNULL
.Note
This function is not normally used by code that wants to handle exceptions. Rather, it can be used when code needs to save and restore the exception state temporarily. Use
PyErr_GetHandledException()
to get the exception state.Added in version 3.11.
-
void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback)¶
- Part of the Stable ABI since version 3.7.
Retrieve the old-style representation of the exception info, as known from
sys.exc_info()
. This refers to an exception that was already caught, not to an exception that was freshly raised. Returns new references for the three objects, any of which may beNULL
. Does not modify the exception info state. This function is kept for backwards compatibility. Prefer usingPyErr_GetHandledException()
.Note
This function is not normally used by code that wants to handle exceptions. Rather, it can be used when code needs to save and restore the exception state temporarily. Use
PyErr_SetExcInfo()
to restore or clear the exception state.Added in version 3.3.
-
void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback)¶
- Part of the Stable ABI since version 3.7.
Set the exception info, as known from
sys.exc_info()
. This refers to an exception that was already caught, not to an exception that was freshly raised. This function steals the references of the arguments. To clear the exception state, passNULL
for all three arguments. This function is kept for backwards compatibility. Prefer usingPyErr_SetHandledException()
.Note
This function is not normally used by code that wants to handle exceptions. Rather, it can be used when code needs to save and restore the exception state temporarily. Use
PyErr_GetExcInfo()
to read the exception state.Added in version 3.3.
Changed in version 3.11: The
type
andtraceback
arguments are no longer used and can be NULL. The interpreter now derives them from the exception instance (thevalue
argument). The function still steals references of all three arguments.
Signal Handling¶
-
int PyErr_CheckSignals()¶
- Part of the Stable ABI.
This function interacts with Python’s signal handling.
If the function is called from the main thread and under the main Python interpreter, it checks whether a signal has been sent to the processes and if so, invokes the corresponding signal handler. If the
signal
module is supported, this can invoke a signal handler written in Python.The function attempts to handle all pending signals, and then returns
0
. However, if a Python signal handler raises an exception, the error indicator is set and the function returns-1
immediately (such that other pending signals may not have been handled yet: they will be on the nextPyErr_CheckSignals()
invocation).If the function is called from a non-main thread, or under a non-main Python interpreter, it does nothing and returns
0
.This function can be called by long-running C code that wants to be interruptible by user requests (such as by pressing Ctrl-C).
Note
The default Python signal handler for
SIGINT
raises theKeyboardInterrupt
exception.
-
void PyErr_SetInterrupt()¶
- Part of the Stable ABI.
Simulate the effect of a
SIGINT
signal arriving. This is equivalent toPyErr_SetInterruptEx(SIGINT)
.Note
This function is async-signal-safe. It can be called without an attached thread state and from a C signal handler.
-
int PyErr_SetInterruptEx(int signum)¶
- Part of the Stable ABI since version 3.10.
Simulate the effect of a signal arriving. The next time
PyErr_CheckSignals()
is called, the Python signal handler for the given signal number will be called.This function can be called by C code that sets up its own signal handling and wants Python signal handlers to be invoked as expected when an interruption is requested (for example when the user presses Ctrl-C to interrupt an operation).
If the given signal isn’t handled by Python (it was set to
signal.SIG_DFL
orsignal.SIG_IGN
), it will be ignored.If signum is outside of the allowed range of signal numbers,
-1
is returned. Otherwise,0
is returned. The error indicator is never changed by this function.Note
This function is async-signal-safe. It can be called without an attached thread state and from a C signal handler.
Added in version 3.10.
-
int PySignal_SetWakeupFd(int fd)¶
This utility function specifies a file descriptor to which the signal number is written as a single byte whenever a signal is received. fd must be non-blocking. It returns the previous such file descriptor.
The value
-1
disables the feature; this is the initial state. This is equivalent tosignal.set_wakeup_fd()
in Python, but without any error checking. fd should be a valid file descriptor. The function should only be called from the main thread.Changed in version 3.5: On Windows, the function now also supports socket handles.
Exception Classes¶
-
PyObject *PyErr_NewException(const char *name, PyObject *base, PyObject *dict)¶
- Return value: New reference. Part of the Stable ABI.
This utility function creates and returns a new exception class. The name argument must be the name of the new exception, a C string of the form
module.classname
. The base and dict arguments are normallyNULL
. This creates a class object derived fromException
(accessible in C asPyExc_Exception
).The
__module__
attribute of the new class is set to the first part (up to the last dot) of the name argument, and the class name is set to the last part (after the last dot). The base argument can be used to specify alternate base classes; it can either be only one class or a tuple of classes. The dict argument can be used to specify a dictionary of class variables and methods.
-
PyObject *PyErr_NewExceptionWithDoc(const char *name, const char *doc, PyObject *base, PyObject *dict)¶
- Return value: New reference. Part of the Stable ABI.
Same as
PyErr_NewException()
, except that the new exception class can easily be given a docstring: If doc is non-NULL
, it will be used as the docstring for the exception class.Added in version 3.2.
-
int PyExceptionClass_Check(PyObject *ob)¶
Return non-zero if ob is an exception class, zero otherwise. This function always succeeds.
-
const char *PyExceptionClass_Name(PyObject *ob)¶
- Part of the Stable ABI since version 3.8.
Return
tp_name
of the exception class ob.
Exception Objects¶
-
int PyExceptionInstance_Check(PyObject *op)¶
Return true if op is an instance of
BaseException
, false otherwise. This function always succeeds.
-
PyExceptionInstance_Class(op)¶
Equivalent to
Py_TYPE(op)
.
-
PyObject *PyException_GetTraceback(PyObject *ex)¶
- Return value: New reference. Part of the Stable ABI.
Return the traceback associated with the exception as a new reference, as accessible from Python through the
__traceback__
attribute. If there is no traceback associated, this returnsNULL
.
-
int PyException_SetTraceback(PyObject *ex, PyObject *tb)¶
- Part of the Stable ABI.
Set the traceback associated with the exception to tb. Use
Py_None
to clear it.
-
PyObject *PyException_GetContext(PyObject *ex)¶
- Return value: New reference. Part of the Stable ABI.
Return the context (another exception instance during whose handling ex was raised) associated with the exception as a new reference, as accessible from Python through the
__context__
attribute. If there is no context associated, this returnsNULL
.
-
void PyException_SetContext(PyObject *ex, PyObject *ctx)¶
- Part of the Stable ABI.
Set the context associated with the exception to ctx. Use
NULL
to clear it. There is no type check to make sure that ctx is an exception instance. This steals a reference to ctx.
-
PyObject *PyException_GetCause(PyObject *ex)¶
- Return value: New reference. Part of the Stable ABI.
Return the cause (either an exception instance, or
None
, set byraise ... from ...
) associated with the exception as a new reference, as accessible from Python through the__cause__
attribute.
-
void PyException_SetCause(PyObject *ex, PyObject *cause)¶
- Part of the Stable ABI.
Set the cause associated with the exception to cause. Use
NULL
to clear it. There is no type check to make sure that cause is either an exception instance orNone
. This steals a reference to cause.The
__suppress_context__
attribute is implicitly set toTrue
by this function.
-
PyObject *PyException_GetArgs(PyObject *ex)¶
- Return value: New reference. Part of the Stable ABI since version 3.12.
Return
args
of exception ex.
-
void PyException_SetArgs(PyObject *ex, PyObject *args)¶
- Part of the Stable ABI since version 3.12.
Set
args
of exception ex to args.
-
PyObject *PyUnstable_Exc_PrepReraiseStar(PyObject *orig, PyObject *excs)¶
- This is Unstable API. It may change without warning in minor releases.
Implement part of the interpreter’s implementation of
except*
. orig is the original exception that was caught, and excs is the list of the exceptions that need to be raised. This list contains the unhandled part of orig, if any, as well as the exceptions that were raised from theexcept*
clauses (so they have a different traceback from orig) and those that were reraised (and have the same traceback as orig). Return theExceptionGroup
that needs to be reraised in the end, orNone
if there is nothing to reraise.Added in version 3.12.
Unicode Exception Objects¶
The following functions are used to create and modify Unicode exceptions from C.
-
PyObject *PyUnicodeDecodeError_Create(const char *encoding, const char *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason)¶
- Return value: New reference. Part of the Stable ABI.
Create a
UnicodeDecodeError
object with the attributes encoding, object, length, start, end and reason. encoding and reason are UTF-8 encoded strings.
-
PyObject *PyUnicodeDecodeError_GetEncoding(PyObject *exc)¶
-
PyObject *PyUnicodeEncodeError_GetEncoding(PyObject *exc)¶
- Return value: New reference. Part of the Stable ABI.
Return the encoding attribute of the given exception object.
-
PyObject *PyUnicodeDecodeError_GetObject(PyObject *exc)¶
-
PyObject *PyUnicodeEncodeError_GetObject(PyObject *exc)¶
-
PyObject *PyUnicodeTranslateError_GetObject(PyObject *exc)¶
- Return value: New reference. Part of the Stable ABI.
Return the object attribute of the given exception object.
-
int PyUnicodeDecodeError_GetStart(PyObject *exc, Py_ssize_t *start)¶
-
int PyUnicodeEncodeError_GetStart(PyObject *exc, Py_ssize_t *start)¶
-
int PyUnicodeTranslateError_GetStart(PyObject *exc, Py_ssize_t *start)¶
- Part of the Stable ABI.
Get the start attribute of the given exception object and place it into *start. start must not be
NULL
. Return0
on success,-1
on failure.If the
UnicodeError.object
is an empty sequence, the resulting start is0
. Otherwise, it is clipped to[0, len(object) - 1]
.See also
-
int PyUnicodeDecodeError_SetStart(PyObject *exc, Py_ssize_t start)¶
-
int PyUnicodeEncodeError_SetStart(PyObject *exc, Py_ssize_t start)¶
-
int PyUnicodeTranslateError_SetStart(PyObject *exc, Py_ssize_t start)¶
- Part of the Stable ABI.
Set the start attribute of the given exception object to start. Return
0
on success,-1
on failure.Note
While passing a negative start does not raise an exception, the corresponding getters will not consider it as a relative offset.
-
int PyUnicodeDecodeError_GetEnd(PyObject *exc, Py_ssize_t *end)¶
-
int PyUnicodeEncodeError_GetEnd(PyObject *exc, Py_ssize_t *end)¶
-
int PyUnicodeTranslateError_GetEnd(PyObject *exc, Py_ssize_t *end)¶
- Part of the Stable ABI.
Get the end attribute of the given exception object and place it into *end. end must not be
NULL
. Return0
on success,-1
on failure.If the
UnicodeError.object
is an empty sequence, the resulting end is0
. Otherwise, it is clipped to[1, len(object)]
.
-
int PyUnicodeDecodeError_SetEnd(PyObject *exc, Py_ssize_t end)¶
-
int PyUnicodeEncodeError_SetEnd(PyObject *exc, Py_ssize_t end)¶
-
int PyUnicodeTranslateError_SetEnd(PyObject *exc, Py_ssize_t end)¶
- Part of the Stable ABI.
Set the end attribute of the given exception object to end. Return
0
on success,-1
on failure.See also
-
PyObject *PyUnicodeDecodeError_GetReason(PyObject *exc)¶
-
PyObject *PyUnicodeEncodeError_GetReason(PyObject *exc)¶
-
PyObject *PyUnicodeTranslateError_GetReason(PyObject *exc)¶
- Return value: New reference. Part of the Stable ABI.
Return the reason attribute of the given exception object.
-
int PyUnicodeDecodeError_SetReason(PyObject *exc, const char *reason)¶
-
int PyUnicodeEncodeError_SetReason(PyObject *exc, const char *reason)¶
-
int PyUnicodeTranslateError_SetReason(PyObject *exc, const char *reason)¶
- Part of the Stable ABI.
Set the reason attribute of the given exception object to reason. Return
0
on success,-1
on failure.
Recursion Control¶
These two functions provide a way to perform safe recursive calls at the C level, both in the core and in extension modules. They are needed if the recursive code does not necessarily invoke Python code (which tracks its recursion depth automatically). They are also not needed for tp_call implementations because the call protocol takes care of recursion handling.
-
int Py_EnterRecursiveCall(const char *where)¶
- Part of the Stable ABI since version 3.9.
Marks a point where a recursive C-level call is about to be performed.
The function then checks if the stack limit is reached. If this is the case, a
RecursionError
is set and a nonzero value is returned. Otherwise, zero is returned.where should be a UTF-8 encoded string such as
" in instance check"
to be concatenated to theRecursionError
message caused by the recursion depth limit.See also
The
PyUnstable_ThreadState_SetStackProtection()
function.Changed in version 3.9: This function is now also available in the limited API.
-
void Py_LeaveRecursiveCall(void)¶
- Part of the Stable ABI since version 3.9.
Ends a
Py_EnterRecursiveCall()
. Must be called once for each successful invocation ofPy_EnterRecursiveCall()
.Changed in version 3.9: This function is now also available in the limited API.
Properly implementing tp_repr
for container types requires
special recursion handling. In addition to protecting the stack,
tp_repr
also needs to track objects to prevent cycles. The
following two functions facilitate this functionality. Effectively,
these are the C equivalent to reprlib.recursive_repr()
.
-
int Py_ReprEnter(PyObject *object)¶
- Part of the Stable ABI.
Called at the beginning of the
tp_repr
implementation to detect cycles.If the object has already been processed, the function returns a positive integer. In that case the
tp_repr
implementation should return a string object indicating a cycle. As examples,dict
objects return{...}
andlist
objects return[...]
.The function will return a negative integer if the recursion limit is reached. In that case the
tp_repr
implementation should typically returnNULL
.Otherwise, the function returns zero and the
tp_repr
implementation can continue normally.
-
void Py_ReprLeave(PyObject *object)¶
- Part of the Stable ABI.
Ends a
Py_ReprEnter()
. Must be called once for each invocation ofPy_ReprEnter()
that returns zero.
-
int Py_GetRecursionLimit(void)¶
- Part of the Stable ABI.
Get the recursion limit for the current interpreter. It can be set with
Py_SetRecursionLimit()
. The recursion limit prevents the Python interpreter stack from growing infinitely.This function cannot fail, and the caller must hold an attached thread state.
See also
-
void Py_SetRecursionLimit(int new_limit)¶
- Part of the Stable ABI.
Set the recursion limit for the current interpreter.
This function cannot fail, and the caller must hold an attached thread state.
See also
Exception and warning types¶
All standard Python exceptions and warning categories are available as global
variables whose names are PyExc_
followed by the Python exception name.
These have the type PyObject*; they are all class objects.
For completeness, here are all the variables:
Exception types¶
C name |
Python name |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Added in version 3.3: PyExc_BlockingIOError
, PyExc_BrokenPipeError
,
PyExc_ChildProcessError
, PyExc_ConnectionError
,
PyExc_ConnectionAbortedError
, PyExc_ConnectionRefusedError
,
PyExc_ConnectionResetError
, PyExc_FileExistsError
,
PyExc_FileNotFoundError
, PyExc_InterruptedError
,
PyExc_IsADirectoryError
, PyExc_NotADirectoryError
,
PyExc_PermissionError
, PyExc_ProcessLookupError
and PyExc_TimeoutError
were introduced following PEP 3151.
Added in version 3.5: PyExc_StopAsyncIteration
and PyExc_RecursionError
.
Added in version 3.6: PyExc_ModuleNotFoundError
.
Added in version 3.11: PyExc_BaseExceptionGroup
.
OSError aliases¶
The following are a compatibility aliases to PyExc_OSError
.
Changed in version 3.3: These aliases used to be separate exception types.
C name |
Python name |
Notes |
|---|---|---|
|
||
|
||
|
Notes:
PyExc_WindowsError
is only defined on Windows; protect code that
uses this by testing that the preprocessor macro MS_WINDOWS
is defined.
Warning types¶
C name |
Python name |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Added in version 3.2: PyExc_ResourceWarning
.
Added in version 3.10: PyExc_EncodingWarning
.
Tracebacks¶
-
PyTypeObject PyTraceBack_Type¶
- Part of the Stable ABI.
Type object for traceback objects. This is available as
types.TracebackType
in the Python layer.
-
int PyTraceBack_Check(PyObject *op)¶
Return true if op is a traceback object, false otherwise. This function does not account for subtypes.
-
int PyTraceBack_Here(PyFrameObject *f)¶
- Part of the Stable ABI.
Replace the
__traceback__
attribute on the current exception with a new traceback prepending f to the existing chain.Calling this function without an exception set is undefined behavior.
This function returns
0
on success, and returns-1
with an exception set on failure.
-
int PyTraceBack_Print(PyObject *tb, PyObject *f)¶
- Part of the Stable ABI.
Write the traceback tb into the file f.
This function returns
0
on success, and returns-1
with an exception set on failure.


Python support for the Linux

Python support for the Linux perf
profiler¶
- author:
Pablo Galindo
The Linux perf profiler
is a very powerful tool that allows you to profile and obtain
information about the performance of your application.
perf
also has a very vibrant ecosystem of tools
that aid with the analysis of the data that it produces.
The main problem with using the perf
profiler with Python applications is that
perf
only gets information about native symbols, that is, the names of
functions and procedures written in C. This means that the names and file names
of Python functions in your code will not appear in the output of perf
.
Since Python 3.12, the interpreter can run in a special mode that allows Python
functions to appear in the output of the perf
profiler. When this mode is
enabled, the interpreter will interpose a small piece of code compiled on the
fly before the execution of every Python function and it will teach perf
the
relationship between this piece of code and the associated Python function using
perf map files.
Note
Support for the perf
profiler is currently only available for Linux on
select architectures. Check the output of the configure
build step or
check the output of python -m sysconfig | grep HAVE_PERF_TRAMPOLINE
to see if your system is supported.
For example, consider the following script:
def foo(n):
result = 0
for _ in range(n):
result += 1
return result
def bar(n):
foo(n)
def baz(n):
bar(n)
if __name__ == "__main__":
baz(1000000)
We can run perf
to sample CPU stack traces at 9999 hertz:
$ perf record -F 9999 -g -o perf.data python my_script.py
Then we can use perf report
to analyze the data:
$ perf report --stdio -n -g
# Children Self Samples Command Shared Object Symbol
# ........ ........ ............ .......... .................. ..........................................
#
91.08% 0.00% 0 python.exe python.exe [.] _start
|
---_start
|
--90.71%--__libc_start_main
Py_BytesMain
|
|--56.88%--pymain_run_python.constprop.0
| |
| |--56.13%--_PyRun_AnyFileObject
| | _PyRun_SimpleFileObject
| | |
| | |--55.02%--run_mod
| | | |
| | | --54.65%--PyEval_EvalCode
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | |
| | | |--51.67%--_PyEval_EvalFrameDefault
| | | | |
| | | | |--11.52%--_PyLong_Add
| | | | | |
| | | | | |--2.97%--_PyObject_Malloc
...
As you can see, the Python functions are not shown in the output, only _PyEval_EvalFrameDefault
(the function that evaluates the Python bytecode) shows up. Unfortunately that’s not very useful because all Python
functions use the same C function to evaluate bytecode so we cannot know which Python function corresponds to which
bytecode-evaluating function.
Instead, if we run the same experiment with perf
support enabled we get:
$ perf report --stdio -n -g
# Children Self Samples Command Shared Object Symbol
# ........ ........ ............ .......... .................. .....................................................................
#
90.58% 0.36% 1 python.exe python.exe [.] _start
|
---_start
|
--89.86%--__libc_start_main
Py_BytesMain
|
|--55.43%--pymain_run_python.constprop.0
| |
| |--54.71%--_PyRun_AnyFileObject
| | _PyRun_SimpleFileObject
| | |
| | |--53.62%--run_mod
| | | |
| | | --53.26%--PyEval_EvalCode
| | | py::<module>:/src/script.py
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | py::baz:/src/script.py
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | py::bar:/src/script.py
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | py::foo:/src/script.py
| | | |
| | | |--51.81%--_PyEval_EvalFrameDefault
| | | | |
| | | | |--13.77%--_PyLong_Add
| | | | | |
| | | | | |--3.26%--_PyObject_Malloc
How to enable perf
profiling support¶
perf
profiling support can be enabled either from the start using
the environment variable PYTHONPERFSUPPORT
or the
-X perf
option,
or dynamically using sys.activate_stack_trampoline()
and
sys.deactivate_stack_trampoline()
.
The sys
functions take precedence over the -X
option,
the -X
option takes precedence over the environment variable.
Example, using the environment variable:
$ PYTHONPERFSUPPORT=1 perf record -F 9999 -g -o perf.data python my_script.py
$ perf report -g -i perf.data
Example, using the -X
option:
$ perf record -F 9999 -g -o perf.data python -X perf my_script.py
$ perf report -g -i perf.data
Example, using the sys
APIs in file example.py
:
import sys
sys.activate_stack_trampoline("perf")
do_profiled_stuff()
sys.deactivate_stack_trampoline()
non_profiled_stuff()
…then:
$ perf record -F 9999 -g -o perf.data python ./example.py
$ perf report -g -i perf.data
How to obtain the best results¶
For best results, Python should be compiled with
CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
as this allows
profilers to unwind using only the frame pointer and not on DWARF debug
information. This is because as the code that is interposed to allow perf
support is dynamically generated it doesn’t have any DWARF debugging information
available.
You can check if your system has been compiled with this flag by running:
$ python -m sysconfig | grep 'no-omit-frame-pointer'
If you don’t see any output it means that your interpreter has not been compiled with
frame pointers and therefore it may not be able to show Python functions in the output
of perf
.
How to work without frame pointers¶
If you are working with a Python interpreter that has been compiled without
frame pointers, you can still use the perf
profiler, but the overhead will be
a bit higher because Python needs to generate unwinding information for every
Python function call on the fly. Additionally, perf
will take more time to
process the data because it will need to use the DWARF debugging information to
unwind the stack and this is a slow process.
To enable this mode, you can use the environment variable
PYTHON_PERF_JIT_SUPPORT
or the -X perf_jit
option,
which will enable the JIT mode for the perf
profiler.
Note
Due to a bug in the perf
tool, only perf
versions higher than v6.8
will work with the JIT mode. The fix was also backported to the v6.7.2
version of the tool.
Note that when checking the version of the perf
tool (which can be done
by running perf version
) you must take into account that some distros
add some custom version numbers including a -
character. This means
that perf 6.7-3
is not necessarily perf 6.7.3
.
When using the perf JIT mode, you need an extra step before you can run perf
report
. You need to call the perf inject
command to inject the JIT
information into the perf.data
file.:
$ perf record -F 9999 -g -k 1 --call-graph dwarf -o perf.data python -Xperf_jit my_script.py
$ perf inject -i perf.data --jit --output perf.jit.data
$ perf report -g -i perf.jit.data
or using the environment variable:
$ PYTHON_PERF_JIT_SUPPORT=1 perf record -F 9999 -g --call-graph dwarf -o perf.data python my_script.py
$ perf inject -i perf.data --jit --output perf.jit.data
$ perf report -g -i perf.jit.data
perf inject --jit
command will read perf.data
,
automatically pick up the perf dump file that Python creates (in
/tmp/perf-$PID.dump
), and then create perf.jit.data
which merges all the
JIT information together. It should also create a lot of jitted-XXXX-N.so
files in the current directory which are ELF images for all the JIT trampolines
that were created by Python.
Warning
When using --call-graph dwarf
, the perf
tool will take
snapshots of the stack of the process being profiled and save the
information in the perf.data
file. By default, the size of the stack dump
is 8192 bytes, but you can change the size by passing it after
a comma like --call-graph dwarf,16384
.
The size of the stack dump is important because if the size is too small
perf
will not be able to unwind the stack and the output will be
incomplete. On the other hand, if the size is too big, then perf
won’t
be able to sample the process as frequently as it would like as the overhead
will be higher.
The stack size is particularly important when profiling Python code compiled
with low optimization levels (like -O0
), as these builds tend to have
larger stack frames. If you are compiling Python with -O0
and not seeing
Python functions in your profiling output, try increasing the stack dump
size to 65528 bytes (the maximum):
$ perf record -F 9999 -g -k 1 --call-graph dwarf,65528 -o perf.data python -Xperf_jit my_script.py
Different compilation flags can significantly impact stack sizes:
Builds with
-O0
typically have much larger stack frames than those with-O1
or higherAdding optimizations (
-O1
,-O2
, etc.) typically reduces stack sizeFrame pointers (
-fno-omit-frame-pointer
) generally provide more reliable stack unwinding


C API Extension Support for Free Threading¶
Starting with the 3.13 release, CPython has support for running with the global interpreter lock (GIL) disabled in a configuration called free threading. This document describes how to adapt C API extensions to support free threading.
Identifying the Free-Threaded Build in C¶
The CPython C API exposes the Py_GIL_DISABLED
macro: in the free-threaded
build it’s defined to 1
, and in the regular build it’s not defined.
You can use it to enable code that only runs under the free-threaded build:
#ifdef Py_GIL_DISABLED
/* code that only runs in the free-threaded build */
#endif
Note
On Windows, this macro is not defined automatically, but must be specified
to the compiler when building. The sysconfig.get_config_var()
function
can be used to determine whether the current running interpreter had the
macro defined.
Module Initialization¶
Extension modules need to explicitly indicate that they support running with the GIL disabled; otherwise importing the extension will raise a warning and enable the GIL at runtime.
There are two ways to indicate that an extension module supports running with the GIL disabled depending on whether the extension uses multi-phase or single-phase initialization.
Multi-Phase Initialization¶
Extensions that use multi-phase initialization (i.e.,
PyModuleDef_Init()
) should add a Py_mod_gil
slot in the
module definition. If your extension supports older versions of CPython,
you should guard the slot with a PY_VERSION_HEX
check.
static struct PyModuleDef_Slot module_slots[] = {
...
#if PY_VERSION_HEX >= 0x030D0000
{Py_mod_gil, Py_MOD_GIL_NOT_USED},
#endif
{0, NULL}
};
static struct PyModuleDef moduledef = {
PyModuleDef_HEAD_INIT,
.m_slots = module_slots,
...
};
Single-Phase Initialization¶
Extensions that use single-phase initialization (i.e.,
PyModule_Create()
) should call PyUnstable_Module_SetGIL()
to
indicate that they support running with the GIL disabled. The function is
only defined in the free-threaded build, so you should guard the call with
#ifdef Py_GIL_DISABLED
to avoid compilation errors in the regular build.
static struct PyModuleDef moduledef = {
PyModuleDef_HEAD_INIT,
...
};
PyMODINIT_FUNC
PyInit_mymodule(void)
{
PyObject *m = PyModule_Create(&moduledef);
if (m == NULL) {
return NULL;
}
#ifdef Py_GIL_DISABLED
PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);
#endif
return m;
}
General API Guidelines¶
Most of the C API is thread-safe, but there are some exceptions.
Struct Fields: Accessing fields in Python C API objects or structs directly is not thread-safe if the field may be concurrently modified.
Macros: Accessor macros like
PyList_GET_ITEM
,PyList_SET_ITEM
, and macros likePySequence_Fast_GET_SIZE
that use the object returned byPySequence_Fast()
do not perform any error checking or locking. These macros are not thread-safe if the container object may be modified concurrently.Borrowed References: C API functions that return borrowed references may not be thread-safe if the containing object is modified concurrently. See the section on borrowed references for more information.
Container Thread Safety¶
Containers like PyListObject
,
PyDictObject
, and PySetObject
perform internal locking
in the free-threaded build. For example, the PyList_Append()
will
lock the list before appending an item.
PyDict_Next
¶
A notable exception is PyDict_Next()
, which does not lock the
dictionary. You should use Py_BEGIN_CRITICAL_SECTION
to protect
the dictionary while iterating over it if the dictionary may be concurrently
modified:
Py_BEGIN_CRITICAL_SECTION(dict);
PyObject *key, *value;
Py_ssize_t pos = 0;
while (PyDict_Next(dict, &pos, &key, &value)) {
...
}
Py_END_CRITICAL_SECTION();
Borrowed References¶
Some C API functions return borrowed references.
These APIs are not thread-safe if the containing object is modified
concurrently. For example, it’s not safe to use PyList_GetItem()
if the list may be modified concurrently.
The following table lists some borrowed reference APIs and their replacements that return strong references.
Borrowed reference API |
Strong reference API |
|---|---|
none (see PyDict_Next) |
|
Not all APIs that return borrowed references are problematic. For
example, PyTuple_GetItem()
is safe because tuples are immutable.
Similarly, not all uses of the above APIs are problematic. For example,
PyDict_GetItem()
is often used for parsing keyword argument
dictionaries in function calls; those keyword argument dictionaries are
effectively private (not accessible by other threads), so using borrowed
references in that context is safe.
Some of these functions were added in Python 3.13. You can use the pythoncapi-compat package to provide implementations of these functions for older Python versions.
Memory Allocation APIs¶
Python’s memory management C API provides functions in three different allocation domains: “raw”, “mem”, and “object”. For thread-safety, the free-threaded build requires that only Python objects are allocated using the object domain, and that all Python objects are allocated using that domain. This differs from the prior Python versions, where this was only a best practice and not a hard requirement.
Note
Search for uses of PyObject_Malloc()
in your
extension and check that the allocated memory is used for Python objects.
Use PyMem_Malloc()
to allocate buffers instead of
PyObject_Malloc()
.
Thread State and GIL APIs¶
Python provides a set of functions and macros to manage thread state and the GIL, such as:
These functions should still be used in the free-threaded build to manage
thread state even when the GIL is disabled. For example, if you
create a thread outside of Python, you must call PyGILState_Ensure()
before calling into the Python API to ensure that the thread has a valid
Python thread state.
You should continue to call PyEval_SaveThread()
or
Py_BEGIN_ALLOW_THREADS
around blocking operations, such as I/O or
lock acquisitions, to allow other threads to run the
cyclic garbage collector.
Protecting Internal Extension State¶
Your extension may have internal state that was previously protected by the GIL. You may need to add locking to protect this state. The approach will depend on your extension, but some common patterns include:
Caches: global caches are a common source of shared state. Consider using a lock to protect the cache or disabling it in the free-threaded build if the cache is not critical for performance.
Global State: global state may need to be protected by a lock or moved to thread local storage. C11 and C++11 provide the
thread_local
or_Thread_local
for thread-local storage.
Critical Sections¶
In the free-threaded build, CPython provides a mechanism called “critical sections” to protect data that would otherwise be protected by the GIL. While extension authors may not interact with the internal critical section implementation directly, understanding their behavior is crucial when using certain C API functions or managing shared state in the free-threaded build.
What Are Critical Sections?¶
Conceptually, critical sections act as a deadlock avoidance layer built on
top of simple mutexes. Each thread maintains a stack of active critical
sections. When a thread needs to acquire a lock associated with a critical
section (e.g., implicitly when calling a thread-safe C API function like
PyDict_SetItem()
, or explicitly using macros), it attempts to acquire
the underlying mutex.
Using Critical Sections¶
The primary APIs for using critical sections are:
Py_BEGIN_CRITICAL_SECTION
andPy_END_CRITICAL_SECTION
- For locking a single objectPy_BEGIN_CRITICAL_SECTION2
andPy_END_CRITICAL_SECTION2
- For locking two objects simultaneously
These macros must be used in matching pairs and must appear in the same C scope, since they establish a new local scope. These macros are no-ops in non-free-threaded builds, so they can be safely added to code that needs to support both build types.
A common use of a critical section would be to lock an object while accessing an internal attribute of it. For example, if an extension type has an internal count field, you could use a critical section while reading or writing that field:
// read the count, returns new reference to internal count value
PyObject *result;
Py_BEGIN_CRITICAL_SECTION(obj);
result = Py_NewRef(obj->count);
Py_END_CRITICAL_SECTION();
return result;
// write the count, consumes reference from new_count
Py_BEGIN_CRITICAL_SECTION(obj);
obj->count = new_count;
Py_END_CRITICAL_SECTION();
How Critical Sections Work¶
Unlike traditional locks, critical sections do not guarantee exclusive access throughout their entire duration. If a thread would block while holding a critical section (e.g., by acquiring another lock or performing I/O), the critical section is temporarily suspended—all locks are released—and then resumed when the blocking operation completes.
This behavior is similar to what happens with the GIL when a thread makes a blocking call. The key differences are:
Critical sections operate on a per-object basis rather than globally
Critical sections follow a stack discipline within each thread (the “begin” and “end” macros enforce this since they must be paired and within the same scope)
Critical sections automatically release and reacquire locks around potential blocking operations
Deadlock Avoidance¶
Critical sections help avoid deadlocks in two ways:
If a thread tries to acquire a lock that’s already held by another thread, it first suspends all of its active critical sections, temporarily releasing their locks
When the blocking operation completes, only the top-most critical section is reacquired first
This means you cannot rely on nested critical sections to lock multiple objects
at once, as the inner critical section may suspend the outer ones. Instead, use
Py_BEGIN_CRITICAL_SECTION2
to lock two objects simultaneously.
Note that the locks described above are only PyMutex
based locks.
The critical section implementation does not know about or affect other locking
mechanisms that might be in use, like POSIX mutexes. Also note that while
blocking on any PyMutex
causes the critical sections to be
suspended, only the mutexes that are part of the critical sections are
released. If PyMutex
is used without a critical section, it will
not be released and therefore does not get the same deadlock avoidance.
Important Considerations¶
Critical sections may temporarily release their locks, allowing other threads to modify the protected data. Be careful about making assumptions about the state of the data after operations that might block.
Because locks can be temporarily released (suspended), entering a critical section does not guarantee exclusive access to the protected resource throughout the section’s duration. If code within a critical section calls another function that blocks (e.g., acquires another lock, performs blocking I/O), all locks held by the thread via critical sections will be released. This is similar to how the GIL can be released during blocking calls.
Only the lock(s) associated with the most recently entered (top-most) critical section are guaranteed to be held at any given time. Locks for outer, nested critical sections might have been suspended.
You can lock at most two objects simultaneously with these APIs. If you need to lock more objects, you’ll need to restructure your code.
While critical sections will not deadlock if you attempt to lock the same object twice, they are less efficient than purpose-built reentrant locks for this use case.
When using
Py_BEGIN_CRITICAL_SECTION2
, the order of the objects doesn’t affect correctness (the implementation handles deadlock avoidance), but it’s good practice to always lock objects in a consistent order.Remember that the critical section macros are primarily for protecting access to Python objects that might be involved in internal CPython operations susceptible to the deadlock scenarios described above. For protecting purely internal extension state, standard mutexes or other synchronization primitives might be more appropriate.
Building Extensions for the Free-Threaded Build¶
C API extensions need to be built specifically for the free-threaded build.
The wheels, shared libraries, and binaries are indicated by a t
suffix.
pypa/manylinux supports the free-threaded build, with the
t
suffix, such aspython3.13t
.pypa/cibuildwheel supports the free-threaded build on Python 3.13 and 3.14. On Python 3.14, free-threaded wheels will be built by default. On Python 3.13, you will need to set CIBW_ENABLE to cpython-freethreading.
Limited C API and Stable ABI¶
The free-threaded build does not currently support the
Limited C API or the stable ABI. If you use
setuptools to build
your extension and currently set py_limited_api=True
you can use
py_limited_api=not sysconfig.get_config_var("Py_GIL_DISABLED")
to opt out
of the limited API when building with the free-threaded build.
Note
You will need to build separate wheels specifically for the free-threaded build. If you currently use the stable ABI, you can continue to build a single wheel for multiple non-free-threaded Python versions.
Windows¶
Due to a limitation of the official Windows installer, you will need to
manually define Py_GIL_DISABLED=1
when building extensions from source.
See also
Porting Extension Modules to Support Free-Threading: A community-maintained porting guide for extension authors.


The Python 2.3 Method Resolution Order

The Python 2.3 Method Resolution Order¶
Note
This is a historical document, provided as an appendix to the official documentation. The Method Resolution Order discussed here was introduced in Python 2.3, but it is still used in later versions – including Python 3.
- Abstract:
This document is intended for Python programmers who want to understand the C3 Method Resolution Order used in Python 2.3. Although it is not intended for newbies, it is quite pedagogical with many worked out examples. I am not aware of other publicly available documents with the same scope, therefore it should be useful.
Disclaimer:
I donate this document to the Python Software Foundation, under the Python 2.3 license. As usual in these circumstances, I warn the reader that what follows should be correct, but I don’t give any warranty. Use it at your own risk and peril!
Acknowledgments:
All the people of the Python mailing list who sent me their support. Paul Foley who pointed out various imprecisions and made me to add the part on local precedence ordering. David Goodger for help with the formatting in reStructuredText. David Mertz for help with the editing. Finally, Guido van Rossum who enthusiastically added this document to the official Python 2.3 home-page.
The beginning¶
Felix qui potuit rerum cognoscere causas – Virgilius
Everything started with a post by Samuele Pedroni to the Python development mailing list [1]. In his post, Samuele showed that the Python 2.2 method resolution order is not monotonic and he proposed to replace it with the C3 method resolution order. Guido agreed with his arguments and therefore now Python 2.3 uses C3. The C3 method itself has nothing to do with Python, since it was invented by people working on Dylan and it is described in a paper intended for lispers [2]. The present paper gives a (hopefully) readable discussion of the C3 algorithm for Pythonistas who want to understand the reasons for the change.
First of all, let me point out that what I am going to say only applies to the new style classes introduced in Python 2.2: classic classes maintain their old method resolution order, depth first and then left to right. Therefore, there is no breaking of old code for classic classes; and even if in principle there could be breaking of code for Python 2.2 new style classes, in practice the cases in which the C3 resolution order differs from the Python 2.2 method resolution order are so rare that no real breaking of code is expected. Therefore:
Don’t be scared!
Moreover, unless you make strong use of multiple inheritance and you have non-trivial hierarchies, you don’t need to understand the C3 algorithm, and you can easily skip this paper. On the other hand, if you really want to know how multiple inheritance works, then this paper is for you. The good news is that things are not as complicated as you might expect.
Let me begin with some basic definitions.
Given a class C in a complicated multiple inheritance hierarchy, it is a non-trivial task to specify the order in which methods are overridden, i.e. to specify the order of the ancestors of C.
The list of the ancestors of a class C, including the class itself, ordered from the nearest ancestor to the furthest, is called the class precedence list or the linearization of C.
The Method Resolution Order (MRO) is the set of rules that construct the linearization. In the Python literature, the idiom “the MRO of C” is also used as a synonymous for the linearization of the class C.
For instance, in the case of single inheritance hierarchy, if C is a subclass of C1, and C1 is a subclass of C2, then the linearization of C is simply the list [C, C1 , C2]. However, with multiple inheritance hierarchies, the construction of the linearization is more cumbersome, since it is more difficult to construct a linearization that respects local precedence ordering and monotonicity.
I will discuss the local precedence ordering later, but I can give the definition of monotonicity here. A MRO is monotonic when the following is true: if C1 precedes C2 in the linearization of C, then C1 precedes C2 in the linearization of any subclass of C. Otherwise, the innocuous operation of deriving a new class could change the resolution order of methods, potentially introducing very subtle bugs. Examples where this happens will be shown later.
Not all classes admit a linearization. There are cases, in complicated hierarchies, where it is not possible to derive a class such that its linearization respects all the desired properties.
Here I give an example of this situation. Consider the hierarchy
>>> O = object
>>> class X(O): pass
>>> class Y(O): pass
>>> class A(X,Y): pass
>>> class B(Y,X): pass
which can be represented with the following inheritance graph, where I
have denoted with O the object
class, which is the beginning of any
hierarchy for new style classes:
----------- | | | O | | / \ | - X Y / | / | / | / |/ A B \ / ?
In this case, it is not possible to derive a new class C from A and B, since X precedes Y in A, but Y precedes X in B, therefore the method resolution order would be ambiguous in C.
Python 2.3 raises an exception in this situation (TypeError: MRO conflict among bases Y, X) forbidding the naive programmer from creating ambiguous hierarchies. Python 2.2 instead does not raise an exception, but chooses an ad hoc ordering (CABXYO in this case).
The C3 Method Resolution Order¶
Let me introduce a few simple notations which will be useful for the following discussion. I will use the shortcut notation:
C1 C2 ... CN
to indicate the list of classes [C1, C2, … , CN].
The head of the list is its first element:
head = C1
whereas the tail is the rest of the list:
tail = C2 ... CN.
I shall also use the notation:
C + (C1 C2 ... CN) = C C1 C2 ... CN
to denote the sum of the lists [C] + [C1, C2, … ,CN].
Now I can explain how the MRO works in Python 2.3.
Consider a class C in a multiple inheritance hierarchy, with C inheriting from the base classes B1, B2, … , BN. We want to compute the linearization L[C] of the class C. The rule is the following:
the linearization of C is the sum of C plus the merge of the linearizations of the parents and the list of the parents.
In symbolic notation:
L[C(B1 ... BN)] = C + merge(L[B1] ... L[BN], B1 ... BN)
In particular, if C is the object
class, which has no parents, the
linearization is trivial:
L[object] = object.
However, in general one has to compute the merge according to the following prescription:
take the head of the first list, i.e L[B1][0]; if this head is not in the tail of any of the other lists, then add it to the linearization of C and remove it from the lists in the merge, otherwise look at the head of the next list and take it, if it is a good head. Then repeat the operation until all the class are removed or it is impossible to find good heads. In this case, it is impossible to construct the merge, Python 2.3 will refuse to create the class C and will raise an exception.
This prescription ensures that the merge operation preserves the ordering, if the ordering can be preserved. On the other hand, if the order cannot be preserved (as in the example of serious order disagreement discussed above) then the merge cannot be computed.
The computation of the merge is trivial if C has only one parent (single inheritance); in this case:
L[C(B)] = C + merge(L[B],B) = C + L[B]
However, in the case of multiple inheritance things are more cumbersome and I don’t expect you can understand the rule without a couple of examples ;-)
Examples¶
First example. Consider the following hierarchy:
>>> O = object
>>> class F(O): pass
>>> class E(O): pass
>>> class D(O): pass
>>> class C(D,F): pass
>>> class B(D,E): pass
>>> class A(B,C): pass
In this case the inheritance graph can be drawn as:
6 --- Level 3 | O | (more general) / --- \ / | \ | / | \ | / | \ | --- --- --- | Level 2 3 | D | 4| E | | F | 5 | --- --- --- | \ \ _ / | | \ / \ _ | | \ / \ | | --- --- | Level 1 1 | B | | C | 2 | --- --- | \ / | \ / \ / --- Level 0 0 | A | (more specialized) ---
The linearizations of O,D,E and F are trivial:
L[O] = O
L[D] = D O
L[E] = E O
L[F] = F O
The linearization of B can be computed as:
L[B] = B + merge(DO, EO, DE)
We see that D is a good head, therefore we take it and we are reduced to
compute merge(O,EO,E)
. Now O is not a good head, since it is in the
tail of the sequence EO. In this case the rule says that we have to
skip to the next sequence. Then we see that E is a good head; we take
it and we are reduced to compute merge(O,O)
which gives O. Therefore:
L[B] = B D E O
Using the same procedure one finds:
L[C] = C + merge(DO,FO,DF)
= C + D + merge(O,FO,F)
= C + D + F + merge(O,O)
= C D F O
Now we can compute:
L[A] = A + merge(BDEO,CDFO,BC)
= A + B + merge(DEO,CDFO,C)
= A + B + C + merge(DEO,DFO)
= A + B + C + D + merge(EO,FO)
= A + B + C + D + E + merge(O,FO)
= A + B + C + D + E + F + merge(O,O)
= A B C D E F O
In this example, the linearization is ordered in a pretty nice way according to the inheritance level, in the sense that lower levels (i.e. more specialized classes) have higher precedence (see the inheritance graph). However, this is not the general case.
I leave as an exercise for the reader to compute the linearization for my second example:
>>> O = object
>>> class F(O): pass
>>> class E(O): pass
>>> class D(O): pass
>>> class C(D,F): pass
>>> class B(E,D): pass
>>> class A(B,C): pass
The only difference with the previous example is the change B(D,E) –> B(E,D); however even such a little modification completely changes the ordering of the hierarchy:
6 --- Level 3 | O | / --- \ / | \ / | \ / | \ --- --- --- Level 2 2 | E | 4 | D | | F | 5 --- --- --- \ / \ / \ / \ / \ / \ / --- --- Level 1 1 | B | | C | 3 --- --- \ / \ / --- Level 0 0 | A | ---
Notice that the class E, which is in the second level of the hierarchy, precedes the class C, which is in the first level of the hierarchy, i.e. E is more specialized than C, even if it is in a higher level.
A lazy programmer can obtain the MRO directly from Python 2.2, since in
this case it coincides with the Python 2.3 linearization. It is enough
to invoke the mro()
method of class A:
>>> A.mro()
[<class 'A'>, <class 'B'>, <class 'E'>,
<class 'C'>, <class 'D'>, <class 'F'>,
<class 'object'>]
Finally, let me consider the example discussed in the first section, involving a serious order disagreement. In this case, it is straightforward to compute the linearizations of O, X, Y, A and B:
L[O] = 0 L[X] = X O L[Y] = Y O L[A] = A X Y O L[B] = B Y X O
However, it is impossible to compute the linearization for a class C that inherits from A and B:
L[C] = C + merge(AXYO, BYXO, AB)
= C + A + merge(XYO, BYXO, B)
= C + A + B + merge(XYO, YXO)
At this point we cannot merge the lists XYO and YXO, since X is in the tail of YXO whereas Y is in the tail of XYO: therefore there are no good heads and the C3 algorithm stops. Python 2.3 raises an error and refuses to create the class C.
Bad Method Resolution Orders¶
A MRO is bad when it breaks such fundamental properties as local precedence ordering and monotonicity. In this section, I will show that both the MRO for classic classes and the MRO for new style classes in Python 2.2 are bad.
It is easier to start with the local precedence ordering. Consider the following example:
>>> F=type('Food',(),{'remember2buy':'spam'})
>>> E=type('Eggs',(F,),{'remember2buy':'eggs'})
>>> G=type('GoodFood',(F,E),{}) # under Python 2.3 this is an error!
with inheritance diagram
O | (buy spam) F | \ | E (buy eggs) | / G (buy eggs or spam ?)
We see that class G inherits from F and E, with F before E: therefore we would expect the attribute G.remember2buy to be inherited by F.remember2buy and not by E.remember2buy: nevertheless Python 2.2 gives
>>> G.remember2buy
'eggs'
This is a breaking of local precedence ordering since the order in the local precedence list, i.e. the list of the parents of G, is not preserved in the Python 2.2 linearization of G:
L[G,P22]= G E F object # F *follows* E
One could argue that the reason why F follows E in the Python 2.2 linearization is that F is less specialized than E, since F is the superclass of E; nevertheless the breaking of local precedence ordering is quite non-intuitive and error prone. This is particularly true since it is a different from old style classes:
>>> class F: remember2buy='spam'
>>> class E(F): remember2buy='eggs'
>>> class G(F,E): pass
>>> G.remember2buy
'spam'
In this case the MRO is GFEF and the local precedence ordering is preserved.
As a general rule, hierarchies such as the previous one should be avoided, since it is unclear if F should override E or vice-versa. Python 2.3 solves the ambiguity by raising an exception in the creation of class G, effectively stopping the programmer from generating ambiguous hierarchies. The reason for that is that the C3 algorithm fails when the merge:
merge(FO,EFO,FE)
cannot be computed, because F is in the tail of EFO and E is in the tail of FE.
The real solution is to design a non-ambiguous hierarchy, i.e. to derive G from E and F (the more specific first) and not from F and E; in this case the MRO is GEF without any doubt.
O | F (spam) / | (eggs) E | \ | G (eggs, no doubt)
Python 2.3 forces the programmer to write good hierarchies (or, at least, less error-prone ones).
On a related note, let me point out that the Python 2.3 algorithm is smart enough to recognize obvious mistakes, as the duplication of classes in the list of parents:
>>> class A(object): pass
>>> class C(A,A): pass # error
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: duplicate base class A
Python 2.2 (both for classic classes and new style classes) in this situation, would not raise any exception.
Finally, I would like to point out two lessons we have learned from this example:
despite the name, the MRO determines the resolution order of attributes, not only of methods;
the default food for Pythonistas is spam ! (but you already knew that ;-)
Having discussed the issue of local precedence ordering, let me now consider the issue of monotonicity. My goal is to show that neither the MRO for classic classes nor that for Python 2.2 new style classes is monotonic.
To prove that the MRO for classic classes is non-monotonic is rather trivial, it is enough to look at the diamond diagram:
C / \ / \ A B \ / \ / D
One easily discerns the inconsistency:
L[B,P21] = B C # B precedes C : B's methods win
L[D,P21] = D A C B C # B follows C : C's methods win!
On the other hand, there are no problems with the Python 2.2 and 2.3 MROs, they give both:
L[D] = D A B C
Guido points out in his essay [3] that the classic MRO is not so bad in
practice, since one can typically avoids diamonds for classic classes.
But all new style classes inherit from object
, therefore diamonds are
unavoidable and inconsistencies shows up in every multiple inheritance
graph.
The MRO of Python 2.2 makes breaking monotonicity difficult, but not impossible. The following example, originally provided by Samuele Pedroni, shows that the MRO of Python 2.2 is non-monotonic:
>>> class A(object): pass
>>> class B(object): pass
>>> class C(object): pass
>>> class D(object): pass
>>> class E(object): pass
>>> class K1(A,B,C): pass
>>> class K2(D,B,E): pass
>>> class K3(D,A): pass
>>> class Z(K1,K2,K3): pass
Here are the linearizations according to the C3 MRO (the reader should verify these linearizations as an exercise and draw the inheritance diagram ;-)
L[A] = A O
L[B] = B O
L[C] = C O
L[D] = D O
L[E] = E O
L[K1]= K1 A B C O
L[K2]= K2 D B E O
L[K3]= K3 D A O
L[Z] = Z K1 K2 K3 D A B C E O
Python 2.2 gives exactly the same linearizations for A, B, C, D, E, K1, K2 and K3, but a different linearization for Z:
L[Z,P22] = Z K1 K3 A K2 D B C E O
It is clear that this linearization is wrong, since A comes before D whereas in the linearization of K3 A comes after D. In other words, in K3 methods derived by D override methods derived by A, but in Z, which still is a subclass of K3, methods derived by A override methods derived by D! This is a violation of monotonicity. Moreover, the Python 2.2 linearization of Z is also inconsistent with local precedence ordering, since the local precedence list of the class Z is [K1, K2, K3] (K2 precedes K3), whereas in the linearization of Z K2 follows K3. These problems explain why the 2.2 rule has been dismissed in favor of the C3 rule.
The end¶
This section is for the impatient reader, who skipped all the previous sections and jumped immediately to the end. This section is for the lazy programmer too, who didn’t want to exercise her/his brain. Finally, it is for the programmer with some hubris, otherwise s/he would not be reading a paper on the C3 method resolution order in multiple inheritance hierarchies ;-) These three virtues taken all together (and not separately) deserve a prize: the prize is a short Python 2.2 script that allows you to compute the 2.3 MRO without risk to your brain. Simply change the last line to play with the various examples I have discussed in this paper.:
#<mro.py>
"""C3 algorithm by Samuele Pedroni (with readability enhanced by me)."""
class __metaclass__(type):
"All classes are metamagically modified to be nicely printed"
__repr__ = lambda cls: cls.__name__
class ex_2:
"Serious order disagreement" #From Guido
class O: pass
class X(O): pass
class Y(O): pass
class A(X,Y): pass
class B(Y,X): pass
try:
class Z(A,B): pass #creates Z(A,B) in Python 2.2
except TypeError:
pass # Z(A,B) cannot be created in Python 2.3
class ex_5:
"My first example"
class O: pass
class F(O): pass
class E(O): pass
class D(O): pass
class C(D,F): pass
class B(D,E): pass
class A(B,C): pass
class ex_6:
"My second example"
class O: pass
class F(O): pass
class E(O): pass
class D(O): pass
class C(D,F): pass
class B(E,D): pass
class A(B,C): pass
class ex_9:
"Difference between Python 2.2 MRO and C3" #From Samuele
class O: pass
class A(O): pass
class B(O): pass
class C(O): pass
class D(O): pass
class E(O): pass
class K1(A,B,C): pass
class K2(D,B,E): pass
class K3(D,A): pass
class Z(K1,K2,K3): pass
def merge(seqs):
print '\n\nCPL[%s]=%s' % (seqs[0][0],seqs),
res = []; i=0
while 1:
nonemptyseqs=[seq for seq in seqs if seq]
if not nonemptyseqs: return res
i+=1; print '\n',i,'round: candidates...',
for seq in nonemptyseqs: # find merge candidates among seq heads
cand = seq[0]; print ' ',cand,
nothead=[s for s in nonemptyseqs if cand in s[1:]]
if nothead: cand=None #reject candidate
else: break
if not cand: raise "Inconsistent hierarchy"
res.append(cand)
for seq in nonemptyseqs: # remove cand
if seq[0] == cand: del seq[0]
def mro(C):
"Compute the class precedence list (mro) according to C3"
return merge([[C]]+map(mro,C.__bases__)+[list(C.__bases__)])
def print_mro(C):
print '\nMRO[%s]=%s' % (C,mro(C))
print '\nP22 MRO[%s]=%s' % (C,C.mro())
print_mro(ex_9.Z)
#</mro.py>
That’s all folks,
enjoy !