Remote debugging attachment protocol

Remote debugging attachment protocol¶
This protocol enables external tools to attach to a running CPython process and execute Python code remotely.
Most platforms require elevated privileges to attach to another Python process.
Disabling remote debugging¶
To disable remote debugging support, use any of the following:
Set the
PYTHON_DISABLE_REMOTE_DEBUG
environment variable to1
before starting the interpreter.Use the
-X disable_remote_debug
command-line option.Compile Python with the
--without-remote-debug
build flag.
Permission requirements¶
Attaching to a running Python process for remote debugging requires elevated privileges on most platforms. The specific requirements and troubleshooting steps depend on your operating system:
Linux
The tracer process must have the CAP_SYS_PTRACE
capability or equivalent
privileges. You can only trace processes you own and can signal. Tracing may
fail if the process is already being traced, or if it is running with
set-user-ID or set-group-ID. Security modules like Yama may further restrict
tracing.
To temporarily relax ptrace restrictions (until reboot), run:
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
Note
Disabling ptrace_scope
reduces system hardening and should only be done
in trusted environments.
If running inside a container, use --cap-add=SYS_PTRACE
or
--privileged
, and run as root if needed.
Try re-running the command with elevated privileges:
sudo -E !!
macOS
To attach to another process, you typically need to run your debugging tool
with elevated privileges. This can be done by using sudo
or running as
root.
Even when attaching to processes you own, macOS may block debugging unless the debugger is run with root privileges due to system security restrictions.
Windows
To attach to another process, you usually need to run your debugging tool with administrative privileges. Start the command prompt or terminal as Administrator.
Some processes may still be inaccessible even with Administrator rights,
unless you have the SeDebugPrivilege
privilege enabled.
To resolve file or folder access issues, adjust the security permissions:
Right-click the file or folder and select Properties.
Go to the Security tab to view users and groups with access.
Click Edit to modify permissions.
Select your user account.
In Permissions, check Read or Full control as needed.
Click Apply, then OK to confirm.
Note
Ensure you’ve satisfied all Permission requirements before proceeding.
This section describes the low-level protocol that enables external tools to inject and execute a Python script within a running CPython process.
This mechanism forms the basis of the sys.remote_exec()
function, which
instructs a remote Python process to execute a .py
file. However, this
section does not document the usage of that function. Instead, it provides a
detailed explanation of the underlying protocol, which takes as input the
pid
of a target Python process and the path to a Python source file to be
executed. This information supports independent reimplementation of the
protocol, regardless of programming language.
Warning
The execution of the injected script depends on the interpreter reaching a safe evaluation point. As a result, execution may be delayed depending on the runtime state of the target process.
Once injected, the script is executed by the interpreter within the target process the next time a safe evaluation point is reached. This approach enables remote execution capabilities without modifying the behavior or structure of the running Python application.
Subsequent sections provide a step-by-step description of the protocol, including techniques for locating interpreter structures in memory, safely accessing internal fields, and triggering code execution. Platform-specific variations are noted where applicable, and example implementations are included to clarify each operation.
Locating the PyRuntime structure¶
CPython places the PyRuntime
structure in a dedicated binary section to
help external tools find it at runtime. The name and format of this section
vary by platform. For example, .PyRuntime
is used on ELF systems, and
__DATA,__PyRuntime
is used on macOS. Tools can find the offset of this
structure by examining the binary on disk.
The PyRuntime
structure contains CPython’s global interpreter state and
provides access to other internal data, including the list of interpreters,
thread states, and debugger support fields.
To work with a remote Python process, a debugger must first find the memory
address of the PyRuntime
structure in the target process. This address
can’t be hardcoded or calculated from a symbol name, because it depends on
where the operating system loaded the binary.
The method for finding PyRuntime
depends on the platform, but the steps are
the same in general:
Find the base address where the Python binary or shared library was loaded in the target process.
Use the on-disk binary to locate the offset of the
.PyRuntime
section.Add the section offset to the base address to compute the address in memory.
The sections below explain how to do this on each supported platform and include example code.
Linux (ELF)
To find the PyRuntime
structure on Linux:
Read the process’s memory map (for example,
/proc/<pid>/maps
) to find the address where the Python executable orlibpython
was loaded.Parse the ELF section headers in the binary to get the offset of the
.PyRuntime
section.Add that offset to the base address from step 1 to get the memory address of
PyRuntime
.
The following is an example implementation:
def find_py_runtime_linux(pid: int) -> int:
# Step 1: Try to find the Python executable in memory
binary_path, base_address = find_mapped_binary(
pid, name_contains="python"
)
# Step 2: Fallback to shared library if executable is not found
if binary_path is None:
binary_path, base_address = find_mapped_binary(
pid, name_contains="libpython"
)
# Step 3: Parse ELF headers to get .PyRuntime section offset
section_offset = parse_elf_section_offset(
binary_path, ".PyRuntime"
)
# Step 4: Compute PyRuntime address in memory
return base_address + section_offset
On Linux systems, there are two main approaches to read memory from another
process. The first is through the /proc
filesystem, specifically by reading from
/proc/[pid]/mem
which provides direct access to the process’s memory. This
requires appropriate permissions - either being the same user as the target
process or having root access. The second approach is using the
process_vm_readv()
system call which provides a more efficient way to copy
memory between processes. While ptrace’s PTRACE_PEEKTEXT
operation can also be
used to read memory, it is significantly slower as it only reads one word at a
time and requires multiple context switches between the tracer and tracee
processes.
For parsing ELF sections, the process involves reading and interpreting the ELF file format structures from the binary file on disk. The ELF header contains a pointer to the section header table. Each section header contains metadata about a section including its name (stored in a separate string table), offset, and size. To find a specific section like .PyRuntime, you need to walk through these headers and match the section name. The section header then provides the offset where that section exists in the file, which can be used to calculate its runtime address when the binary is loaded into memory.
You can read more about the ELF file format in the ELF specification.
macOS (Mach-O)
To find the PyRuntime
structure on macOS:
Call
task_for_pid()
to get themach_port_t
task port for the target process. This handle is needed to read memory using APIs likemach_vm_read_overwrite
andmach_vm_region
.Scan the memory regions to find the one containing the Python executable or
libpython
.Load the binary file from disk and parse the Mach-O headers to find the section named
PyRuntime
in the__DATA
segment. On macOS, symbol names are automatically prefixed with an underscore, so thePyRuntime
symbol appears as_PyRuntime
in the symbol table, but the section name is not affected.
The following is an example implementation:
def find_py_runtime_macos(pid: int) -> int:
# Step 1: Get access to the process's memory
handle = get_memory_access_handle(pid)
# Step 2: Try to find the Python executable in memory
binary_path, base_address = find_mapped_binary(
handle, name_contains="python"
)
# Step 3: Fallback to libpython if the executable is not found
if binary_path is None:
binary_path, base_address = find_mapped_binary(
handle, name_contains="libpython"
)
# Step 4: Parse Mach-O headers to get __DATA,__PyRuntime section offset
section_offset = parse_macho_section_offset(
binary_path, "__DATA", "__PyRuntime"
)
# Step 5: Compute the PyRuntime address in memory
return base_address + section_offset
On macOS, accessing another process’s memory requires using Mach-O specific APIs
and file formats. The first step is obtaining a task_port
handle via
task_for_pid()
, which provides access to the target process’s memory space.
This handle enables memory operations through APIs like
mach_vm_read_overwrite()
.
The process memory can be examined using mach_vm_region()
to scan through the
virtual memory space, while proc_regionfilename()
helps identify which binary
files are loaded at each memory region. When the Python binary or library is
found, its Mach-O headers need to be parsed to locate the PyRuntime
structure.
The Mach-O format organizes code and data into segments and sections. The
PyRuntime
structure lives in a section named __PyRuntime
within the
__DATA
segment. The actual runtime address calculation involves finding the
__TEXT
segment which serves as the binary’s base address, then locating the
__DATA
segment containing our target section. The final address is computed by
combining the base address with the appropriate section offsets from the Mach-O
headers.
Note that accessing another process’s memory on macOS typically requires elevated privileges - either root access or special security entitlements granted to the debugging process.
Windows (PE)
To find the PyRuntime
structure on Windows:
Use the ToolHelp API to enumerate all modules loaded in the target process. This is done using functions such as CreateToolhelp32Snapshot, Module32First, and Module32Next.
Identify the module corresponding to
python.exe
orpythonXY.dll
, whereX
andY
are the major and minor version numbers of the Python version, and record its base address.Locate the
PyRuntim
section. Due to the PE format’s 8-character limit on section names (defined asIMAGE_SIZEOF_SHORT_NAME
), the original namePyRuntime
is truncated. This section contains thePyRuntime
structure.Retrieve the section’s relative virtual address (RVA) and add it to the base address of the module.
The following is an example implementation:
def find_py_runtime_windows(pid: int) -> int:
# Step 1: Try to find the Python executable in memory
binary_path, base_address = find_loaded_module(
pid, name_contains="python"
)
# Step 2: Fallback to shared pythonXY.dll if the executable is not
# found
if binary_path is None:
binary_path, base_address = find_loaded_module(
pid, name_contains="python3"
)
# Step 3: Parse PE section headers to get the RVA of the PyRuntime
# section. The section name appears as "PyRuntim" due to the
# 8-character limit defined by the PE format (IMAGE_SIZEOF_SHORT_NAME).
section_rva = parse_pe_section_offset(binary_path, "PyRuntim")
# Step 4: Compute PyRuntime address in memory
return base_address + section_rva
On Windows, accessing another process’s memory requires using the Windows API
functions like CreateToolhelp32Snapshot()
and Module32First()/Module32Next()
to enumerate loaded modules. The OpenProcess()
function provides a handle to
access the target process’s memory space, enabling memory operations through
ReadProcessMemory()
.
The process memory can be examined by enumerating loaded modules to find the
Python binary or DLL. When found, its PE headers need to be parsed to locate the
PyRuntime
structure.
The PE format organizes code and data into sections. The PyRuntime
structure
lives in a section named “PyRuntim” (truncated from “PyRuntime” due to PE’s
8-character name limit). The actual runtime address calculation involves finding
the module’s base address from the module entry, then locating our target
section in the PE headers. The final address is computed by combining the base
address with the section’s virtual address from the PE section headers.
Note that accessing another process’s memory on Windows typically requires
appropriate privileges - either administrative access or the SeDebugPrivilege
privilege granted to the debugging process.
Reading _Py_DebugOffsets¶
Once the address of the PyRuntime
structure has been determined, the next
step is to read the _Py_DebugOffsets
structure located at the beginning of
the PyRuntime
block.
This structure provides version-specific field offsets that are needed to safely read interpreter and thread state memory. These offsets vary between CPython versions and must be checked before use to ensure they are compatible.
To read and check the debug offsets, follow these steps:
Read memory from the target process starting at the
PyRuntime
address, covering the same number of bytes as the_Py_DebugOffsets
structure. This structure is located at the very start of thePyRuntime
memory block. Its layout is defined in CPython’s internal headers and stays the same within a given minor version, but may change in major versions.Check that the structure contains valid data:
The
cookie
field must match the expected debug marker.The
version
field must match the version of the Python interpreter used by the debugger.If either the debugger or the target process is using a pre-release version (for example, an alpha, beta, or release candidate), the versions must match exactly.
The
free_threaded
field must have the same value in both the debugger and the target process.
If the structure is valid, the offsets it contains can be used to locate fields in memory. If any check fails, the debugger should stop the operation to avoid reading memory in the wrong format.
The following is an example implementation that reads and checks
_Py_DebugOffsets
:
def read_debug_offsets(pid: int, py_runtime_addr: int) -> DebugOffsets:
# Step 1: Read memory from the target process at the PyRuntime address
data = read_process_memory(
pid, address=py_runtime_addr, size=DEBUG_OFFSETS_SIZE
)
# Step 2: Deserialize the raw bytes into a _Py_DebugOffsets structure
debug_offsets = parse_debug_offsets(data)
# Step 3: Validate the contents of the structure
if debug_offsets.cookie != EXPECTED_COOKIE:
raise RuntimeError("Invalid or missing debug cookie")
if debug_offsets.version != LOCAL_PYTHON_VERSION:
raise RuntimeError(
"Mismatch between caller and target Python versions"
)
if debug_offsets.free_threaded != LOCAL_FREE_THREADED:
raise RuntimeError("Mismatch in free-threaded configuration")
return debug_offsets
Warning
Process suspension recommended
To avoid race conditions and ensure memory consistency, it is strongly recommended that the target process be suspended before performing any operations that read or write internal interpreter state. The Python runtime may concurrently mutate interpreter data structures—such as creating or destroying threads—during normal execution. This can result in invalid memory reads or writes.
A debugger may suspend execution by attaching to the process with ptrace
or by sending a SIGSTOP
signal. Execution should only be resumed after
debugger-side memory operations are complete.
Note
Some tools, such as profilers or sampling-based debuggers, may operate on a running process without suspension. In such cases, tools must be explicitly designed to handle partially updated or inconsistent memory. For most debugger implementations, suspending the process remains the safest and most robust approach.
Locating the interpreter and thread state¶
Before code can be injected and executed in a remote Python process, the
debugger must choose a thread in which to schedule execution. This is necessary
because the control fields used to perform remote code injection are located in
the _PyRemoteDebuggerSupport
structure, which is embedded in a
PyThreadState
object. These fields are modified by the debugger to request
execution of injected scripts.
The PyThreadState
structure represents a thread running inside a Python
interpreter. It maintains the thread’s evaluation context and contains the
fields required for debugger coordination. Locating a valid PyThreadState
is therefore a key prerequisite for triggering execution remotely.
A thread is typically selected based on its role or ID. In most cases, the main thread is used, but some tools may target a specific thread by its native thread ID. Once the target thread is chosen, the debugger must locate both the interpreter and the associated thread state structures in memory.
The relevant internal structures are defined as follows:
PyInterpreterState
represents an isolated Python interpreter instance. Each interpreter maintains its own set of imported modules, built-in state, and thread state list. Although most Python applications use a single interpreter, CPython supports multiple interpreters in the same process.PyThreadState
represents a thread running within an interpreter. It contains execution state and the control fields used by the debugger.
To locate a thread:
Use the offset
runtime_state.interpreters_head
to obtain the address of the first interpreter in thePyRuntime
structure. This is the entry point to the linked list of active interpreters.Use the offset
interpreter_state.threads_main
to access the main thread state associated with the selected interpreter. This is typically the most reliable thread to target.Optionally, use the offset
interpreter_state.threads_head
to iterate through the linked list of all thread states. EachPyThreadState
structure contains anative_thread_id
field, which may be compared to a target thread ID to find a specific thread.Once a valid
PyThreadState
has been found, its address can be used in later steps of the protocol, such as writing debugger control fields and scheduling execution.
The following is an example implementation that locates the main thread state:
def find_main_thread_state(
pid: int, py_runtime_addr: int, debug_offsets: DebugOffsets,
) -> int:
# Step 1: Read interpreters_head from PyRuntime
interp_head_ptr = (
py_runtime_addr + debug_offsets.runtime_state.interpreters_head
)
interp_addr = read_pointer(pid, interp_head_ptr)
if interp_addr == 0:
raise RuntimeError("No interpreter found in the target process")
# Step 2: Read the threads_main pointer from the interpreter
threads_main_ptr = (
interp_addr + debug_offsets.interpreter_state.threads_main
)
thread_state_addr = read_pointer(pid, threads_main_ptr)
if thread_state_addr == 0:
raise RuntimeError("Main thread state is not available")
return thread_state_addr
The following example demonstrates how to locate a thread by its native thread ID:
def find_thread_by_id(
pid: int,
interp_addr: int,
debug_offsets: DebugOffsets,
target_tid: int,
) -> int:
# Start at threads_head and walk the linked list
thread_ptr = read_pointer(
pid,
interp_addr + debug_offsets.interpreter_state.threads_head
)
while thread_ptr:
native_tid_ptr = (
thread_ptr + debug_offsets.thread_state.native_thread_id
)
native_tid = read_int(pid, native_tid_ptr)
if native_tid == target_tid:
return thread_ptr
thread_ptr = read_pointer(
pid,
thread_ptr + debug_offsets.thread_state.next
)
raise RuntimeError("Thread with the given ID was not found")
Once a valid thread state has been located, the debugger can proceed with modifying its control fields and scheduling execution, as described in the next section.
Writing control information¶
Once a valid PyThreadState
structure has been identified, the debugger may
modify control fields within it to schedule the execution of a specified Python
script. These control fields are checked periodically by the interpreter, and
when set correctly, they trigger the execution of remote code at a safe point
in the evaluation loop.
Each PyThreadState
contains a _PyRemoteDebuggerSupport
structure used
for communication between the debugger and the interpreter. The locations of
its fields are defined by the _Py_DebugOffsets
structure and include the
following:
debugger_script_path
: A fixed-size buffer that holds the full path to a Python source file (.py
). This file must be accessible and readable by the target process when execution is triggered.debugger_pending_call
: An integer flag. Setting this to1
tells the interpreter that a script is ready to be executed.eval_breaker
: A field checked by the interpreter during execution. Setting bit 5 (_PY_EVAL_PLEASE_STOP_BIT
, value1U << 5
) in this field causes the interpreter to pause and check for debugger activity.
To complete the injection, the debugger must perform the following steps:
Write the full script path into the
debugger_script_path
buffer.Set
debugger_pending_call
to1
.Read the current value of
eval_breaker
, set bit 5 (_PY_EVAL_PLEASE_STOP_BIT
), and write the updated value back. This signals the interpreter to check for debugger activity.
The following is an example implementation:
def inject_script(
pid: int,
thread_state_addr: int,
debug_offsets: DebugOffsets,
script_path: str
) -> None:
# Compute the base offset of _PyRemoteDebuggerSupport
support_base = (
thread_state_addr +
debug_offsets.debugger_support.remote_debugger_support
)
# Step 1: Write the script path into debugger_script_path
script_path_ptr = (
support_base +
debug_offsets.debugger_support.debugger_script_path
)
write_string(pid, script_path_ptr, script_path)
# Step 2: Set debugger_pending_call to 1
pending_ptr = (
support_base +
debug_offsets.debugger_support.debugger_pending_call
)
write_int(pid, pending_ptr, 1)
# Step 3: Set _PY_EVAL_PLEASE_STOP_BIT (bit 5, value 1 << 5) in
# eval_breaker
eval_breaker_ptr = (
thread_state_addr +
debug_offsets.debugger_support.eval_breaker
)
breaker = read_int(pid, eval_breaker_ptr)
breaker |= (1 << 5)
write_int(pid, eval_breaker_ptr, breaker)
Once these fields are set, the debugger may resume the process (if it was suspended). The interpreter will process the request at the next safe evaluation point, load the script from disk, and execute it.
It is the responsibility of the debugger to ensure that the script file remains present and accessible to the target process during execution.
Note
Script execution is asynchronous. The script file cannot be deleted immediately after injection. The debugger should wait until the injected script has produced an observable effect before removing the file. This effect depends on what the script is designed to do. For example, a debugger might wait until the remote process connects back to a socket before removing the script. Once such an effect is observed, it is safe to assume the file is no longer needed.
Summary¶
To inject and execute a Python script in a remote process:
Locate the
PyRuntime
structure in the target process’s memory.Read and validate the
_Py_DebugOffsets
structure at the beginning ofPyRuntime
.Use the offsets to locate a valid
PyThreadState
.Write the path to a Python script into
debugger_script_path
.Set the
debugger_pending_call
flag to1
.Set
_PY_EVAL_PLEASE_STOP_BIT
in theeval_breaker
field.Resume the process (if suspended). The script will execute at the next safe evaluation point.


Input and Output

7. Input and Output¶
There are several ways to present the output of a program; data can be printed in a human-readable form, or written to a file for future use. This chapter will discuss some of the possibilities.
7.1. Fancier Output Formatting¶
So far we’ve encountered two ways of writing values: expression statements and
the print()
function. (A third way is using the write()
method
of file objects; the standard output file can be referenced as sys.stdout
.
See the Library Reference for more information on this.)
Often you’ll want more control over the formatting of your output than simply printing space-separated values. There are several ways to format output.
To use formatted string literals, begin a string with
f
orF
before the opening quotation mark or triple quotation mark. Inside this string, you can write a Python expression between{
and}
characters that can refer to variables or literal values.>>> year = 2016 >>> event = 'Referendum' >>> f'Results of the {year} {event}' 'Results of the 2016 Referendum'
The
str.format()
method of strings requires more manual effort. You’ll still use{
and}
to mark where a variable will be substituted and can provide detailed formatting directives, but you’ll also need to provide the information to be formatted. In the following code block there are two examples of how to format variables:>>> yes_votes = 42_572_654 >>> total_votes = 85_705_149 >>> percentage = yes_votes / total_votes >>> '{:-9} YES votes {:2.2%}'.format(yes_votes, percentage) ' 42572654 YES votes 49.67%'
Notice how the
yes_votes
are padded with spaces and a negative sign only for negative numbers. The example also printspercentage
multiplied by 100, with 2 decimal places and followed by a percent sign (see Format Specification Mini-Language for details).Finally, you can do all the string handling yourself by using string slicing and concatenation operations to create any layout you can imagine. The string type has some methods that perform useful operations for padding strings to a given column width.
When you don’t need fancy output but just want a quick display of some
variables for debugging purposes, you can convert any value to a string with
the repr()
or str()
functions.
The str()
function is meant to return representations of values which are
fairly human-readable, while repr()
is meant to generate representations
which can be read by the interpreter (or will force a SyntaxError
if
there is no equivalent syntax). For objects which don’t have a particular
representation for human consumption, str()
will return the same value as
repr()
. Many values, such as numbers or structures like lists and
dictionaries, have the same representation using either function. Strings, in
particular, have two distinct representations.
Some examples:
>>> s = 'Hello, world.'
>>> str(s)
'Hello, world.'
>>> repr(s)
"'Hello, world.'"
>>> str(1/7)
'0.14285714285714285'
>>> x = 10 * 3.25
>>> y = 200 * 200
>>> s = 'The value of x is ' + repr(x) + ', and y is ' + repr(y) + '...'
>>> print(s)
The value of x is 32.5, and y is 40000...
>>> # The repr() of a string adds string quotes and backslashes:
>>> hello = 'hello, world\n'
>>> hellos = repr(hello)
>>> print(hellos)
'hello, world\n'
>>> # The argument to repr() may be any Python object:
>>> repr((x, y, ('spam', 'eggs')))
"(32.5, 40000, ('spam', 'eggs'))"
The string
module contains support for a simple templating approach
based upon regular expressions, via string.Template
.
This offers yet another way to substitute values into strings,
using placeholders like $x
and replacing them with values from a dictionary.
This syntax is easy to use, although it offers much less control for formatting.
7.1.1. Formatted String Literals¶
Formatted string literals (also called f-strings for
short) let you include the value of Python expressions inside a string by
prefixing the string with f
or F
and writing expressions as
{expression}
.
An optional format specifier can follow the expression. This allows greater control over how the value is formatted. The following example rounds pi to three places after the decimal:
>>> import math
>>> print(f'The value of pi is approximately {math.pi:.3f}.')
The value of pi is approximately 3.142.
Passing an integer after the ':'
will cause that field to be a minimum
number of characters wide. This is useful for making columns line up.
>>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 7678}
>>> for name, phone in table.items():
... print(f'{name:10} ==> {phone:10d}')
...
Sjoerd ==> 4127
Jack ==> 4098
Dcab ==> 7678
Other modifiers can be used to convert the value before it is formatted.
'!a'
applies ascii()
, '!s'
applies str()
, and '!r'
applies repr()
:
>>> animals = 'eels'
>>> print(f'My hovercraft is full of {animals}.')
My hovercraft is full of eels.
>>> print(f'My hovercraft is full of {animals!r}.')
My hovercraft is full of 'eels'.
The =
specifier can be used to expand an expression to the text of the
expression, an equal sign, then the representation of the evaluated expression:
>>> bugs = 'roaches'
>>> count = 13
>>> area = 'living room'
>>> print(f'Debugging {bugs=} {count=} {area=}')
Debugging bugs='roaches' count=13 area='living room'
See self-documenting expressions for more information
on the =
specifier. For a reference on these format specifications, see
the reference guide for the Format Specification Mini-Language.
7.1.2. The String format() Method¶
Basic usage of the str.format()
method looks like this:
>>> print('We are the {} who say "{}!"'.format('knights', 'Ni'))
We are the knights who say "Ni!"
The brackets and characters within them (called format fields) are replaced with
the objects passed into the str.format()
method. A number in the
brackets can be used to refer to the position of the object passed into the
str.format()
method.
>>> print('{0} and {1}'.format('spam', 'eggs'))
spam and eggs
>>> print('{1} and {0}'.format('spam', 'eggs'))
eggs and spam
If keyword arguments are used in the str.format()
method, their values
are referred to by using the name of the argument.
>>> print('This {food} is {adjective}.'.format(
... food='spam', adjective='absolutely horrible'))
This spam is absolutely horrible.
Positional and keyword arguments can be arbitrarily combined:
>>> print('The story of {0}, {1}, and {other}.'.format('Bill', 'Manfred',
... other='Georg'))
The story of Bill, Manfred, and Georg.
If you have a really long format string that you don’t want to split up, it
would be nice if you could reference the variables to be formatted by name
instead of by position. This can be done by simply passing the dict and using
square brackets '[]'
to access the keys.
>>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678}
>>> print('Jack: {0[Jack]:d}; Sjoerd: {0[Sjoerd]:d}; '
... 'Dcab: {0[Dcab]:d}'.format(table))
Jack: 4098; Sjoerd: 4127; Dcab: 8637678
This could also be done by passing the table
dictionary as keyword arguments with the **
notation.
>>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678}
>>> print('Jack: {Jack:d}; Sjoerd: {Sjoerd:d}; Dcab: {Dcab:d}'.format(**table))
Jack: 4098; Sjoerd: 4127; Dcab: 8637678
This is particularly useful in combination with the built-in function
vars()
, which returns a dictionary containing all local variables:
>>> table = {k: str(v) for k, v in vars().items()}
>>> message = " ".join([f'{k}: ' + '{' + k +'};' for k in table.keys()])
>>> print(message.format(**table))
__name__: __main__; __doc__: None; __package__: None; __loader__: ...
As an example, the following lines produce a tidily aligned set of columns giving integers and their squares and cubes:
>>> for x in range(1, 11):
... print('{0:2d} {1:3d} {2:4d}'.format(x, x*x, x*x*x))
...
1 1 1
2 4 8
3 9 27
4 16 64
5 25 125
6 36 216
7 49 343
8 64 512
9 81 729
10 100 1000
For a complete overview of string formatting with str.format()
, see
Format String Syntax.
7.1.3. Manual String Formatting¶
Here’s the same table of squares and cubes, formatted manually:
>>> for x in range(1, 11):
... print(repr(x).rjust(2), repr(x*x).rjust(3), end=' ')
... # Note use of 'end' on previous line
... print(repr(x*x*x).rjust(4))
...
1 1 1
2 4 8
3 9 27
4 16 64
5 25 125
6 36 216
7 49 343
8 64 512
9 81 729
10 100 1000
(Note that the one space between each column was added by the
way print()
works: it always adds spaces between its arguments.)
The str.rjust()
method of string objects right-justifies a string in a
field of a given width by padding it with spaces on the left. There are
similar methods str.ljust()
and str.center()
. These methods do
not write anything, they just return a new string. If the input string is too
long, they don’t truncate it, but return it unchanged; this will mess up your
column lay-out but that’s usually better than the alternative, which would be
lying about a value. (If you really want truncation you can always add a
slice operation, as in x.ljust(n)[:n]
.)
There is another method, str.zfill()
, which pads a numeric string on the
left with zeros. It understands about plus and minus signs:
>>> '12'.zfill(5)
'00012'
>>> '-3.14'.zfill(7)
'-003.14'
>>> '3.14159265359'.zfill(5)
'3.14159265359'
7.1.4. Old string formatting¶
The % operator (modulo) can also be used for string formatting.
Given format % values
(where format is a string),
%
conversion specifications in format are replaced with
zero or more elements of values.
This operation is commonly known as string
interpolation. For example:
>>> import math
>>> print('The value of pi is approximately %5.3f.' % math.pi)
The value of pi is approximately 3.142.
More information can be found in the printf-style String Formatting section.
7.2. Reading and Writing Files¶
open()
returns a file object, and is most commonly used with
two positional arguments and one keyword argument:
open(filename, mode, encoding=None)
>>> f = open('workfile', 'w', encoding="utf-8")
The first argument is a string containing the filename. The second argument is
another string containing a few characters describing the way in which the file
will be used. mode can be 'r'
when the file will only be read, 'w'
for only writing (an existing file with the same name will be erased), and
'a'
opens the file for appending; any data written to the file is
automatically added to the end. 'r+'
opens the file for both reading and
writing. The mode argument is optional; 'r'
will be assumed if it’s
omitted.
Normally, files are opened in text mode, that means, you read and write
strings from and to the file, which are encoded in a specific encoding.
If encoding is not specified, the default is platform dependent
(see open()
).
Because UTF-8 is the modern de-facto standard, encoding="utf-8"
is
recommended unless you know that you need to use a different encoding.
Appending a 'b'
to the mode opens the file in binary mode.
Binary mode data is read and written as bytes
objects.
You can not specify encoding when opening file in binary mode.
In text mode, the default when reading is to convert platform-specific line
endings (\n
on Unix, \r\n
on Windows) to just \n
. When writing in
text mode, the default is to convert occurrences of \n
back to
platform-specific line endings. This behind-the-scenes modification
to file data is fine for text files, but will corrupt binary data like that in
JPEG
or EXE
files. Be very careful to use binary mode when
reading and writing such files.
It is good practice to use the with
keyword when dealing
with file objects. The advantage is that the file is properly closed
after its suite finishes, even if an exception is raised at some
point. Using with
is also much shorter than writing
equivalent try
-finally
blocks:
>>> with open('workfile', encoding="utf-8") as f:
... read_data = f.read()
>>> # We can check that the file has been automatically closed.
>>> f.closed
True
If you’re not using the with
keyword, then you should call
f.close()
to close the file and immediately free up any system
resources used by it.
Warning
Calling f.write()
without using the with
keyword or calling
f.close()
might result in the arguments
of f.write()
not being completely written to the disk, even if the
program exits successfully.
After a file object is closed, either by a with
statement
or by calling f.close()
, attempts to use the file object will
automatically fail.
>>> f.close()
>>> f.read()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: I/O operation on closed file.
7.2.1. Methods of File Objects¶
The rest of the examples in this section will assume that a file object called
f
has already been created.
To read a file’s contents, call f.read(size)
, which reads some quantity of
data and returns it as a string (in text mode) or bytes object (in binary mode).
size is an optional numeric argument. When size is omitted or negative, the
entire contents of the file will be read and returned; it’s your problem if the
file is twice as large as your machine’s memory. Otherwise, at most size
characters (in text mode) or size bytes (in binary mode) are read and returned.
If the end of the file has been reached, f.read()
will return an empty
string (''
).
>>> f.read()
'This is the entire file.\n'
>>> f.read()
''
f.readline()
reads a single line from the file; a newline character (\n
)
is left at the end of the string, and is only omitted on the last line of the
file if the file doesn’t end in a newline. This makes the return value
unambiguous; if f.readline()
returns an empty string, the end of the file
has been reached, while a blank line is represented by '\n'
, a string
containing only a single newline.
>>> f.readline()
'This is the first line of the file.\n'
>>> f.readline()
'Second line of the file\n'
>>> f.readline()
''
For reading lines from a file, you can loop over the file object. This is memory efficient, fast, and leads to simple code:
>>> for line in f:
... print(line, end='')
...
This is the first line of the file.
Second line of the file
If you want to read all the lines of a file in a list you can also use
list(f)
or f.readlines()
.
f.write(string)
writes the contents of string to the file, returning
the number of characters written.
>>> f.write('This is a test\n')
15
Other types of objects need to be converted – either to a string (in text mode) or a bytes object (in binary mode) – before writing them:
>>> value = ('the answer', 42)
>>> s = str(value) # convert the tuple to string
>>> f.write(s)
18
f.tell()
returns an integer giving the file object’s current position in the file
represented as number of bytes from the beginning of the file when in binary mode and
an opaque number when in text mode.
To change the file object’s position, use f.seek(offset, whence)
. The position is computed
from adding offset to a reference point; the reference point is selected by
the whence argument. A whence value of 0 measures from the beginning
of the file, 1 uses the current file position, and 2 uses the end of the file as
the reference point. whence can be omitted and defaults to 0, using the
beginning of the file as the reference point.
>>> f = open('workfile', 'rb+')
>>> f.write(b'0123456789abcdef')
16
>>> f.seek(5) # Go to the 6th byte in the file
5
>>> f.read(1)
b'5'
>>> f.seek(-3, 2) # Go to the 3rd byte before the end
13
>>> f.read(1)
b'd'
In text files (those opened without a b
in the mode string), only seeks
relative to the beginning of the file are allowed (the exception being seeking
to the very file end with seek(0, 2)
) and the only valid offset values are
those returned from the f.tell()
, or zero. Any other offset value produces
undefined behaviour.
File objects have some additional methods, such as isatty()
and
truncate()
which are less frequently used; consult the Library
Reference for a complete guide to file objects.
7.2.2. Saving structured data with json
¶
Strings can easily be written to and read from a file. Numbers take a bit more
effort, since the read()
method only returns strings, which will have to
be passed to a function like int()
, which takes a string like '123'
and returns its numeric value 123. When you want to save more complex data
types like nested lists and dictionaries, parsing and serializing by hand
becomes complicated.
Rather than having users constantly writing and debugging code to save
complicated data types to files, Python allows you to use the popular data
interchange format called JSON (JavaScript Object Notation). The standard module called json
can take Python
data hierarchies, and convert them to string representations; this process is
called serializing. Reconstructing the data from the string representation
is called deserializing. Between serializing and deserializing, the
string representing the object may have been stored in a file or data, or
sent over a network connection to some distant machine.
Note
The JSON format is commonly used by modern applications to allow for data exchange. Many programmers are already familiar with it, which makes it a good choice for interoperability.
If you have an object x
, you can view its JSON string representation with a
simple line of code:
>>> import json
>>> x = [1, 'simple', 'list']
>>> json.dumps(x)
'[1, "simple", "list"]'
Another variant of the dumps()
function, called dump()
,
simply serializes the object to a text file. So if f
is a
text file object opened for writing, we can do this:
json.dump(x, f)
To decode the object again, if f
is a binary file or
text file object which has been opened for reading:
x = json.load(f)
Note
JSON files must be encoded in UTF-8. Use encoding="utf-8"
when opening
JSON file as a text file for both of reading and writing.
This simple serialization technique can handle lists and dictionaries, but
serializing arbitrary class instances in JSON requires a bit of extra effort.
The reference for the json
module contains an explanation of this.
See also
pickle
- the pickle module
Contrary to JSON, pickle is a protocol which allows the serialization of arbitrarily complex Python objects. As such, it is specific to Python and cannot be used to communicate with applications written in other languages. It is also insecure by default: deserializing pickle data coming from an untrusted source can execute arbitrary code, if the data was crafted by a skilled attacker.


Configure Python

3. Configure Python¶
3.1. Build Requirements¶
To build CPython, you will need:
A C11 compiler. Optional C11 features are not required.
On Windows, Microsoft Visual Studio 2017 or later is required.
Support for IEEE 754 floating-point numbers and floating-point Not-a-Number (NaN).
Support for threads.
Changed in version 3.5: On Windows, Visual Studio 2015 or later is now required.
Changed in version 3.6: Selected C99 features, like <stdint.h>
and static inline
functions,
are now required.
Changed in version 3.7: Thread support is now required.
Changed in version 3.11: C11 compiler, IEEE 754 and NaN support are now required. On Windows, Visual Studio 2017 or later is required.
See also PEP 7 “Style Guide for C Code” and PEP 11 “CPython platform support”.
3.1.1. Requirements for optional modules¶
Some optional modules of the standard library require third-party libraries installed for development (for example, header files must be available).
Missing requirements are reported in the configure
output.
Modules that are missing due to missing dependencies are listed near the end
of the make
output,
sometimes using an internal name, for example, _ctypes
for ctypes
module.
If you distribute a CPython interpreter without optional modules, it’s best practice to advise users, who generally expect that standard library modules are available.
Dependencies to build optional modules are:
Dependency |
Minimum version |
Python module |
|---|---|---|
3.3.0 recommended |
||
2.5.0 |
||
|
||
3.0.18 recommended
(1.1.1 minimum)
|
||
3.15.2 |
||
8.5.12 |
||
1.2.2.1 |
||
1.4.5 |
Note that the table does not include all optional modules; in particular,
platform-specific modules like winreg
are not listed here.
See also
The devguide includes a full list of dependencies required to build all modules and instructions on how to install them on common platforms.
--with-system-expat
allows building with an external libexpat library.
Changed in version 3.1: Tcl/Tk version 8.3.1 is now required for tkinter
.
Changed in version 3.5: Tcl/Tk version 8.4 is now required for tkinter
.
Changed in version 3.10: OpenSSL 1.1.1 is now required for hashlib
and ssl
.
SQLite 3.7.15 is now required for sqlite3
.
Changed in version 3.11: Tcl/Tk version 8.5.12 is now required for tkinter
.
Changed in version 3.13: SQLite 3.15.2 is now required for sqlite3
.
3.2. Generated files¶
To reduce build dependencies, Python source code contains multiple generated files. Commands to regenerate all generated files:
make regen-all
make regen-stdlib-module-names
make regen-limited-abi
make regen-configure
The Makefile.pre.in
file documents generated files, their inputs, and tools used
to regenerate them. Search for regen-*
make targets.
3.2.1. configure script¶
The make regen-configure
command regenerates the aclocal.m4
file and
the configure
script using the Tools/build/regen-configure.sh
shell
script which uses an Ubuntu container to get the same tools versions and have a
reproducible output.
The container is optional, the following command can be run locally:
autoreconf -ivf -Werror
The generated files can change depending on the exact versions of the
tools used.
The container that CPython uses has
Autoconf 2.72,
aclocal
from Automake 1.16.5,
and pkg-config 1.8.1.
Changed in version 3.13: Autoconf 2.71 and aclocal 1.16.5 and are now used to regenerate
configure
.
Changed in version 3.14: Autoconf 2.72 is now used to regenerate configure
.
3.3. Configure Options¶
List all configure
script options using:
./configure --help
See also the Misc/SpecialBuilds.txt
in the Python source distribution.
3.3.1. General Options¶
- --enable-loadable-sqlite-extensions¶
Support loadable extensions in the
_sqlite
extension module (default is no) of thesqlite3
module.See the
sqlite3.Connection.enable_load_extension()
method of thesqlite3
module.Added in version 3.6.
- --enable-big-digits=[15|30]¶
Define the size in bits of Python
int
digits: 15 or 30 bits.By default, the digit size is 30.
Define the
PYLONG_BITS_IN_DIGIT
to15
or30
.
- --with-suffix=SUFFIX¶
Set the Python executable suffix to SUFFIX.
The default suffix is
.exe
on Windows and macOS (python.exe
executable),.js
on Emscripten node,.html
on Emscripten browser,.wasm
on WASI, and an empty string on other platforms (python
executable).Changed in version 3.11: The default suffix on WASM platform is one of
.js
,.html
or.wasm
.
- --with-tzpath=<list of absolute paths separated by pathsep>¶
Select the default time zone search path for
zoneinfo.TZPATH
. See the Compile-time configuration of thezoneinfo
module.Default:
/usr/share/zoneinfo:/usr/lib/zoneinfo:/usr/share/lib/zoneinfo:/etc/zoneinfo
.See
os.pathsep
path separator.Added in version 3.9.
- --without-decimal-contextvar¶
Build the
_decimal
extension module using a thread-local context rather than a coroutine-local context (default), see thedecimal
module.See
decimal.HAVE_CONTEXTVAR
and thecontextvars
module.Added in version 3.9.
- --with-dbmliborder=<list of backend names>¶
Override order to check db backends for the
dbm
moduleA valid value is a colon (
:
) separated string with the backend names:ndbm
;gdbm
;bdb
.
- --without-c-locale-coercion¶
Disable C locale coercion to a UTF-8 based locale (enabled by default).
Don’t define the
PY_COERCE_C_LOCALE
macro.See
PYTHONCOERCECLOCALE
and the PEP 538.
- --with-platlibdir=DIRNAME¶
Python library directory name (default is
lib
).Fedora and SuSE use
lib64
on 64-bit platforms.See
sys.platlibdir
.Added in version 3.9.
- --with-wheel-pkg-dir=PATH¶
Directory of wheel packages used by the
ensurepip
module (none by default).Some Linux distribution packaging policies recommend against bundling dependencies. For example, Fedora installs wheel packages in the
/usr/share/python-wheels/
directory and don’t install theensurepip._bundled
package.Added in version 3.10.
- --with-pkg-config=[check|yes|no]¶
Whether configure should use pkg-config to detect build dependencies.
check
(default): pkg-config is optionalyes
: pkg-config is mandatoryno
: configure does not use pkg-config even when present
Added in version 3.11.
- --enable-pystats¶
Turn on internal Python performance statistics gathering.
By default, statistics gathering is off. Use
python3 -X pystats
command or setPYTHONSTATS=1
environment variable to turn on statistics gathering at Python startup.At Python exit, dump statistics if statistics gathering was on and not cleared.
Effects:
Add
-X pystats
command line option.Add
PYTHONSTATS
environment variable.Define the
Py_STATS
macro.Add functions to the
sys
module:sys._stats_on()
: Turns on statistics gathering.sys._stats_off()
: Turns off statistics gathering.sys._stats_clear()
: Clears the statistics.sys._stats_dump()
: Dump statistics to file, and clears the statistics.
The statistics will be dumped to a arbitrary (probably unique) file in
/tmp/py_stats/
(Unix) orC:\temp\py_stats\
(Windows). If that directory does not exist, results will be printed on stderr.Use
Tools/scripts/summarize_stats.py
to read the stats.Statistics:
Opcode:
Specialization: success, failure, hit, deferred, miss, deopt, failures;
Execution count;
Pair count.
Call:
Inlined Python calls;
PyEval calls;
Frames pushed;
Frame object created;
Eval calls: vector, generator, legacy, function VECTORCALL, build class, slot, function “ex”, API, method.
Object:
incref and decref;
interpreter incref and decref;
allocations: all, 512 bytes, 4 kiB, big;
free;
to/from free lists;
dictionary materialized/dematerialized;
type cache;
optimization attempts;
optimization traces created/executed;
uops executed.
Garbage collector:
Garbage collections;
Objects visited;
Objects collected.
Added in version 3.11.
- --disable-gil¶
Enables support for running Python without the global interpreter lock (GIL): free-threaded build.
Defines the
Py_GIL_DISABLED
macro and adds"t"
tosys.abiflags
.See Free-threaded CPython for more detail.
Added in version 3.13.
- --enable-experimental-jit=[no|yes|yes-off|interpreter]¶
Indicate how to integrate the experimental just-in-time compiler.
no
: Don’t build the JIT.yes
: Enable the JIT. To disable it at runtime, set the environment variablePYTHON_JIT=0
.yes-off
: Build the JIT, but disable it by default. To enable it at runtime, set the environment variablePYTHON_JIT=1
.interpreter
: Enable the “JIT interpreter” (only useful for those debugging the JIT itself). To disable it at runtime, set the environment variablePYTHON_JIT=0
.
--enable-experimental-jit=no
is the default behavior if the option is not provided, and--enable-experimental-jit
is shorthand for--enable-experimental-jit=yes
. SeeTools/jit/README.md
for more information, including how to install the necessary build-time dependencies.Note
When building CPython with JIT enabled, ensure that your system has Python 3.11 or later installed.
Added in version 3.13.
- PKG_CONFIG¶
Path to
pkg-config
utility.
- PKG_CONFIG_LIBDIR¶
- PKG_CONFIG_PATH¶
pkg-config
options.
3.3.2. C compiler options¶
- CC¶
C compiler command.
- CFLAGS¶
C compiler flags.
- CPP¶
C preprocessor command.
- CPPFLAGS¶
C preprocessor flags, e.g.
-Iinclude_dir
.
3.3.3. Linker options¶
- LDFLAGS¶
Linker flags, e.g.
-Llibrary_directory
.
- LIBS¶
Libraries to pass to the linker, e.g.
-llibrary
.
- MACHDEP¶
Name for machine-dependent library files.
3.3.4. Options for third-party dependencies¶
Added in version 3.11.
- BZIP2_CFLAGS¶
- BZIP2_LIBS¶
C compiler and linker flags to link Python to
libbz2
, used bybz2
module, overridingpkg-config
.
- CURSES_CFLAGS¶
- CURSES_LIBS¶
C compiler and linker flags for
libncurses
orlibncursesw
, used bycurses
module, overridingpkg-config
.
- GDBM_CFLAGS¶
- GDBM_LIBS¶
C compiler and linker flags for
gdbm
.
- LIBEDIT_CFLAGS¶
- LIBEDIT_LIBS¶
C compiler and linker flags for
libedit
, used byreadline
module, overridingpkg-config
.
- LIBFFI_CFLAGS¶
- LIBMPDEC_CFLAGS¶
- LIBMPDEC_LIBS¶
C compiler and linker flags for
libmpdec
, used bydecimal
module, overridingpkg-config
.Note
These environment variables have no effect unless
--with-system-libmpdec
is specified.
- LIBLZMA_CFLAGS¶
- LIBREADLINE_CFLAGS¶
- LIBREADLINE_LIBS¶
C compiler and linker flags for
libreadline
, used byreadline
module, overridingpkg-config
.
- LIBSQLITE3_CFLAGS¶
- LIBSQLITE3_LIBS¶
C compiler and linker flags for
libsqlite3
, used bysqlite3
module, overridingpkg-config
.
- LIBUUID_CFLAGS¶
- LIBZSTD_CFLAGS¶
- LIBZSTD_LIBS¶
C compiler and linker flags for
libzstd
, used bycompression.zstd
module, overridingpkg-config
.Added in version 3.14.
- PANEL_CFLAGS¶
- PANEL_LIBS¶
C compiler and linker flags for PANEL, overriding
pkg-config
.C compiler and linker flags for
libpanel
orlibpanelw
, used bycurses.panel
module, overridingpkg-config
.
- TCLTK_CFLAGS¶
- TCLTK_LIBS¶
C compiler and linker flags for TCLTK, overriding
pkg-config
.
- ZLIB_CFLAGS¶
3.3.5. WebAssembly Options¶
- --enable-wasm-dynamic-linking¶
Turn on dynamic linking support for WASM.
Dynamic linking enables
dlopen
. File size of the executable increases due to limited dead code elimination and additional features.Added in version 3.11.
- --enable-wasm-pthreads¶
Turn on pthreads support for WASM.
Added in version 3.11.
3.3.6. Install Options¶
- --prefix=PREFIX¶
Install architecture-independent files in PREFIX. On Unix, it defaults to
/usr/local
.This value can be retrieved at runtime using
sys.prefix
.As an example, one can use
--prefix="$HOME/.local/"
to install a Python in its home directory.
- --exec-prefix=EPREFIX¶
Install architecture-dependent files in EPREFIX, defaults to
--prefix
.This value can be retrieved at runtime using
sys.exec_prefix
.
3.3.7. Performance options¶
Configuring Python using --enable-optimizations --with-lto
(PGO + LTO) is
recommended for best performance. The experimental --enable-bolt
flag can
also be used to improve performance.
- --enable-optimizations¶
Enable Profile Guided Optimization (PGO) using
PROFILE_TASK
(disabled by default).The C compiler Clang requires
llvm-profdata
program for PGO. On macOS, GCC also requires it: GCC is just an alias to Clang on macOS.Disable also semantic interposition in libpython if
--enable-shared
and GCC is used: add-fno-semantic-interposition
to the compiler and linker flags.Note
During the build, you may encounter compiler warnings about profile data not being available for some source files. These warnings are harmless, as only a subset of the code is exercised during profile data acquisition. To disable these warnings on Clang, manually suppress them by adding
-Wno-profile-instr-unprofiled
toCFLAGS
.Added in version 3.6.
Changed in version 3.10: Use
-fno-semantic-interposition
on GCC.
- PROFILE_TASK¶
Environment variable used in the Makefile: Python command line arguments for the PGO generation task.
Default:
-m test --pgo --timeout=$(TESTTIMEOUT)
.Added in version 3.8.
Changed in version 3.13: Task failure is no longer ignored silently.
- --with-lto=[full|thin|no|yes]¶
Enable Link Time Optimization (LTO) in any build (disabled by default).
The C compiler Clang requires
llvm-ar
for LTO (ar
on macOS), as well as an LTO-aware linker (ld.gold
orlld
).Added in version 3.6.
Added in version 3.11: To use ThinLTO feature, use
--with-lto=thin
on Clang.Changed in version 3.12: Use ThinLTO as the default optimization policy on Clang if the compiler accepts the flag.
- --enable-bolt¶
Enable usage of the BOLT post-link binary optimizer (disabled by default).
BOLT is part of the LLVM project but is not always included in their binary distributions. This flag requires that
llvm-bolt
andmerge-fdata
are available.BOLT is still a fairly new project so this flag should be considered experimental for now. Because this tool operates on machine code its success is dependent on a combination of the build environment + the other optimization configure args + the CPU architecture, and not all combinations are supported. BOLT versions before LLVM 16 are known to crash BOLT under some scenarios. Use of LLVM 16 or newer for BOLT optimization is strongly encouraged.
The
BOLT_INSTRUMENT_FLAGS
andBOLT_APPLY_FLAGS
configure variables can be defined to override the default set of arguments for llvm-bolt to instrument and apply BOLT data to binaries, respectively.Added in version 3.12.
- BOLT_APPLY_FLAGS¶
Arguments to
llvm-bolt
when creating a BOLT optimized binary.Added in version 3.12.
- BOLT_INSTRUMENT_FLAGS¶
Arguments to
llvm-bolt
when instrumenting binaries.Added in version 3.12.
- --with-computed-gotos¶
Enable computed gotos in evaluation loop (enabled by default on supported compilers).
- --with-tail-call-interp¶
Enable interpreters using tail calls in CPython. If enabled, enabling PGO (
--enable-optimizations
) is highly recommended. This option specifically requires a C compiler with proper tail call support, and the preserve_none calling convention. For example, Clang 19 and newer supports this feature.Added in version 3.14.
- --without-mimalloc¶
Disable the fast mimalloc allocator (enabled by default).
See also
PYTHONMALLOC
environment variable.
- --without-pymalloc¶
Disable the specialized Python memory allocator pymalloc (enabled by default).
See also
PYTHONMALLOC
environment variable.
- --without-doc-strings¶
Disable static documentation strings to reduce the memory footprint (enabled by default). Documentation strings defined in Python are not affected.
Don’t define the
WITH_DOC_STRINGS
macro.See the
PyDoc_STRVAR()
macro.
- --enable-profiling¶
Enable C-level code profiling with
gprof
(disabled by default).
- --with-strict-overflow¶
Add
-fstrict-overflow
to the C compiler flags (by default we add-fno-strict-overflow
instead).
- --without-remote-debug¶
Deactivate remote debugging support described in PEP 768 (enabled by default). When this flag is provided the code that allows the interpreter to schedule the execution of a Python file in a separate process as described in PEP 768 is not compiled. This includes both the functionality to schedule code to be executed and the functionality to receive code to be executed.
-
Py_REMOTE_DEBUG¶
This macro is defined by default, unless Python is configured with
--without-remote-debug
.Note that even if the macro is defined, remote debugging may not be available (for example, on an incompatible platform).
Added in version 3.14.
-
Py_REMOTE_DEBUG¶
3.3.8. Python Debug Build¶
A debug build is Python built with the --with-pydebug
configure
option.
Effects of a debug build:
Display all warnings by default: the list of default warning filters is empty in the
warnings
module.Add
d
tosys.abiflags
.Add
sys.gettotalrefcount()
function.Add
-X showrefcount
command line option.Add
-d
command line option andPYTHONDEBUG
environment variable to debug the parser.Add support for the
__lltrace__
variable: enable low-level tracing in the bytecode evaluation loop if the variable is defined.Install debug hooks on memory allocators to detect buffer overflow and other memory errors.
Define
Py_DEBUG
andPy_REF_DEBUG
macros.Add runtime checks: code surrounded by
#ifdef Py_DEBUG
and#endif
. Enableassert(...)
and_PyObject_ASSERT(...)
assertions: don’t set theNDEBUG
macro (see also the--with-assertions
configure option). Main runtime checks:Add sanity checks on the function arguments.
Unicode and int objects are created with their memory filled with a pattern to detect usage of uninitialized objects.
Ensure that functions which can clear or replace the current exception are not called with an exception raised.
Check that deallocator functions don’t change the current exception.
The garbage collector (
gc.collect()
function) runs some basic checks on objects consistency.The
Py_SAFE_DOWNCAST()
macro checks for integer underflow and overflow when downcasting from wide types to narrow types.
See also the Python Development Mode and the
--with-trace-refs
configure option.
Changed in version 3.8: Release builds and debug builds are now ABI compatible: defining the
Py_DEBUG
macro no longer implies the Py_TRACE_REFS
macro (see the
--with-trace-refs
option).
3.3.9. Debug options¶
- --with-pydebug¶
Build Python in debug mode: define the
Py_DEBUG
macro (disabled by default).
- --with-trace-refs¶
Enable tracing references for debugging purpose (disabled by default).
Effects:
Define the
Py_TRACE_REFS
macro.Add
sys.getobjects()
function.Add
PYTHONDUMPREFS
environment variable.
The
PYTHONDUMPREFS
environment variable can be used to dump objects and reference counts still alive at Python exit.Statically allocated objects are not traced.
Added in version 3.8.
Changed in version 3.13: This build is now ABI compatible with release build and debug build.
- --with-assertions¶
Build with C assertions enabled (default is no):
assert(...);
and_PyObject_ASSERT(...);
.If set, the
NDEBUG
macro is not defined in theOPT
compiler variable.See also the
--with-pydebug
option (debug build) which also enables assertions.Added in version 3.6.
- --with-valgrind¶
Enable Valgrind support (default is no).
- --with-dtrace¶
Enable DTrace support (default is no).
See Instrumenting CPython with DTrace and SystemTap.
Added in version 3.6.
- --with-address-sanitizer¶
Enable AddressSanitizer memory error detector,
asan
(default is no). To improve ASan detection capabilities you may also want to combine this with--without-pymalloc
to disable the specialized small-object allocator whose allocations are not tracked by ASan.Added in version 3.6.
- --with-memory-sanitizer¶
Enable MemorySanitizer allocation error detector,
msan
(default is no).Added in version 3.6.
- --with-undefined-behavior-sanitizer¶
Enable UndefinedBehaviorSanitizer undefined behaviour detector,
ubsan
(default is no).Added in version 3.6.
- --with-thread-sanitizer¶
Enable ThreadSanitizer data race detector,
tsan
(default is no).Added in version 3.13.
3.3.10. Linker options¶
Enable building a shared Python library:
libpython
(default is no).
- --without-static-libpython¶
Do not build
libpythonMAJOR.MINOR.a
and do not installpython.o
(built and enabled by default).Added in version 3.10.
3.3.11. Libraries options¶
- --with-libs='lib1 ...'¶
Link against additional libraries (default is no).
- --with-system-expat¶
Build the
pyexpat
module using an installedexpat
library (default is no).
- --with-system-libmpdec¶
Build the
_decimal
extension module using an installedmpdecimal
library, see thedecimal
module (default is yes).Added in version 3.3.
Changed in version 3.13: Default to using the installed
mpdecimal
library.Changed in version 3.15: A bundled copy of the library will no longer be selected implicitly if an installed
mpdecimal
library is not found. In Python 3.15 only, it can still be selected explicitly using--with-system-libmpdec=no
or--without-system-libmpdec
.Deprecated since version 3.13, will be removed in version 3.16: A copy of the
mpdecimal
library sources will no longer be distributed with Python 3.16.See also
- --with-readline=readline|editline¶
Designate a backend library for the
readline
module.readline: Use readline as the backend.
editline: Use editline as the backend.
Added in version 3.10.
- --without-readline¶
Don’t build the
readline
module (built by default).Don’t define the
HAVE_LIBREADLINE
macro.Added in version 3.10.
- --with-libm=STRING¶
Override
libm
math library to STRING (default is system-dependent).
- --with-libc=STRING¶
Override
libc
C library to STRING (default is system-dependent).
- --with-openssl=DIR¶
Root of the OpenSSL directory.
Added in version 3.7.
- --with-openssl-rpath=[no|auto|DIR]¶
Set runtime library directory (rpath) for OpenSSL libraries:
no
(default): don’t set rpath;auto
: auto-detect rpath from--with-openssl
andpkg-config
;DIR: set an explicit rpath.
Added in version 3.10.
3.3.12. Security Options¶
- --with-hash-algorithm=[fnv|siphash13|siphash24]¶
Select hash algorithm for use in
Python/pyhash.c
:siphash13
(default);siphash24
;fnv
.
Added in version 3.4.
Added in version 3.11:
siphash13
is added and it is the new default.
- --with-builtin-hashlib-hashes=md5,sha1,sha256,sha512,sha3,blake2¶
Built-in hash modules:
md5
;sha1
;sha256
;sha512
;sha3
(with shake);blake2
.
Added in version 3.9.
- --with-ssl-default-suites=[python|openssl|STRING]¶
Override the OpenSSL default cipher suites string:
python
(default): use Python’s preferred selection;openssl
: leave OpenSSL’s defaults untouched;STRING: use a custom string
See the
ssl
module.Added in version 3.7.
Changed in version 3.10: The settings
python
and STRING also set TLS 1.2 as minimum protocol version.
- --disable-safety¶
Disable compiler options that are recommended by OpenSSF for security reasons with no performance overhead. If this option is not enabled, CPython will be built based on safety compiler options with no slow down. When this option is enabled, CPython will not be built with the compiler options listed below.
The following compiler options are disabled with
--disable-safety
:-fstack-protector-strong: Enable run-time checks for stack-based buffer overflows.
-Wtrampolines: Enable warnings about trampolines that require executable stacks.
Added in version 3.14.
- --enable-slower-safety¶
Enable compiler options that are recommended by OpenSSF for security reasons which require overhead. If this option is not enabled, CPython will not be built based on safety compiler options which performance impact. When this option is enabled, CPython will be built with the compiler options listed below.
The following compiler options are enabled with
--enable-slower-safety
:-D_FORTIFY_SOURCE=3: Fortify sources with compile- and run-time checks for unsafe libc usage and buffer overflows.
Added in version 3.14.
3.3.13. macOS Options¶
See Mac/README.rst.
- --enable-universalsdk¶
- --enable-universalsdk=SDKDIR¶
Create a universal binary build. SDKDIR specifies which macOS SDK should be used to perform the build (default is no).
- --enable-framework¶
- --enable-framework=INSTALLDIR¶
Create a Python.framework rather than a traditional Unix install. Optional INSTALLDIR specifies the installation path (default is no).
- --with-universal-archs=ARCH¶
Specify the kind of universal binary that should be created. This option is only valid when
--enable-universalsdk
is set.Options:
universal2
(x86-64 and arm64);32-bit
(PPC and i386);64-bit
(PPC64 and x86-64);3-way
(i386, PPC and x86-64);intel
(i386 and x86-64);intel-32
(i386);intel-64
(x86-64);all
(PPC, i386, PPC64 and x86-64).
Note that values for this configuration item are not the same as the identifiers used for universal binary wheels on macOS. See the Python Packaging User Guide for details on the packaging platform compatibility tags used on macOS
- --with-framework-name=FRAMEWORK¶
Specify the name for the python framework on macOS only valid when
--enable-framework
is set (default:Python
).
- --with-app-store-compliance¶
- --with-app-store-compliance=PATCH-FILE¶
The Python standard library contains strings that are known to trigger automated inspection tool errors when submitted for distribution by the macOS and iOS App Stores. If enabled, this option will apply the list of patches that are known to correct app store compliance. A custom patch file can also be specified. This option is disabled by default.
Added in version 3.13.
3.3.14. iOS Options¶
See iOS/README.rst.
- --enable-framework=INSTALLDIR¶
Create a Python.framework. Unlike macOS, the INSTALLDIR argument specifying the installation path is mandatory.
- --with-framework-name=FRAMEWORK¶
Specify the name for the framework (default:
Python
).
3.3.15. Cross Compiling Options¶
Cross compiling, also known as cross building, can be used to build Python for another CPU architecture or platform. Cross compiling requires a Python interpreter for the build platform. The version of the build Python must match the version of the cross compiled host Python.
- --build=BUILD¶
configure for building on BUILD, usually guessed by config.guess.
- --host=HOST¶
cross-compile to build programs to run on HOST (target platform)
- --with-build-python=path/to/python¶
path to build
python
binary for cross compilingAdded in version 3.11.
- CONFIG_SITE=file¶
An environment variable that points to a file with configure overrides.
Example config.site file:
# config.site-aarch64 ac_cv_buggy_getaddrinfo=no ac_cv_file__dev_ptmx=yes ac_cv_file__dev_ptc=no
- HOSTRUNNER¶
Program to run CPython for the host platform for cross-compilation.
Added in version 3.11.
Cross compiling example:
CONFIG_SITE=config.site-aarch64 ../configure \
--build=x86_64-pc-linux-gnu \
--host=aarch64-unknown-linux-gnu \
--with-build-python=../x86_64/python
3.4. Python Build System¶
3.4.1. Main files of the build system¶
configure.ac
=>configure
;Makefile.pre.in
=>Makefile
(created byconfigure
);pyconfig.h
(created byconfigure
);Modules/Setup
: C extensions built by the Makefile usingModule/makesetup
shell script;
3.4.2. Main build steps¶
C files (
.c
) are built as object files (.o
).A static
libpython
library (.a
) is created from objects files.python.o
and the staticlibpython
library are linked into the finalpython
program.C extensions are built by the Makefile (see
Modules/Setup
).
3.4.3. Main Makefile targets¶
3.4.3.1. make¶
For the most part, when rebuilding after editing some code or
refreshing your checkout from upstream, all you need to do is execute
make
, which (per Make’s semantics) builds the default target, the
first one defined in the Makefile. By tradition (including in the
CPython project) this is usually the all
target. The
configure
script expands an autoconf
variable,
@DEF_MAKE_ALL_RULE@
to describe precisely which targets make
all
will build. The three choices are:
profile-opt
(configured with--enable-optimizations
)build_wasm
(chosen if the host platform matcheswasm32-wasi*
orwasm32-emscripten
)build_all
(configured without explicitly using either of the others)
Depending on the most recent source file changes, Make will rebuild
any targets (object files and executables) deemed out-of-date,
including running configure
again if necessary. Source/target
dependencies are many and maintained manually however, so Make
sometimes doesn’t have all the information necessary to correctly
detect all targets which need to be rebuilt. Depending on which
targets aren’t rebuilt, you might experience a number of problems. If
you have build or test problems which you can’t otherwise explain,
make clean && make
should work around most dependency problems, at
the expense of longer build times.
3.4.3.2. make platform¶
Build the python
program, but don’t build the standard library
extension modules. This generates a file named platform
which
contains a single line describing the details of the build platform,
e.g., macosx-14.3-arm64-3.12
or linux-x86_64-3.13
.
3.4.3.3. make profile-opt¶
Build Python using profile-guided optimization (PGO). You can use the
configure --enable-optimizations
option to make this the
default target of the make
command (make all
or just
make
).
3.4.3.4. make clean¶
Remove built files.
3.4.3.5. make distclean¶
In addition to the work done by make clean
, remove files
created by the configure script. configure
will have to be run
before building again. [6]
3.4.3.6. make install¶
Build the all
target and install Python.
3.4.3.7. make test¶
Build the all
target and run the Python test suite with the
--fast-ci
option without GUI tests. Variables:
TESTOPTS
: additional regrtest command-line options.TESTPYTHONOPTS
: additional Python command-line options.TESTTIMEOUT
: timeout in seconds (default: 10 minutes).
3.4.3.8. make ci¶
This is similar to make test
, but uses the -ugui
to also run GUI tests.
Added in version 3.14.
3.4.3.9. make buildbottest¶
This is similar to make test
, but uses the --slow-ci
option and default timeout of 20 minutes, instead of --fast-ci
option.
3.4.3.10. make regen-all¶
Regenerate (almost) all generated files. These include (but are not
limited to) bytecode cases, and parser generator file.
make regen-stdlib-module-names
and autoconf
must be run
separately for the remaining generated files.
3.4.4. C extensions¶
Some C extensions are built as built-in modules, like the sys
module.
They are built with the Py_BUILD_CORE_BUILTIN
macro defined.
Built-in modules have no __file__
attribute:
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> sys.__file__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'sys' has no attribute '__file__'
Other C extensions are built as dynamic libraries, like the _asyncio
module.
They are built with the Py_BUILD_CORE_MODULE
macro defined.
Example on Linux x86-64:
>>> import _asyncio
>>> _asyncio
<module '_asyncio' from '/usr/lib64/python3.9/lib-dynload/_asyncio.cpython-39-x86_64-linux-gnu.so'>
>>> _asyncio.__file__
'/usr/lib64/python3.9/lib-dynload/_asyncio.cpython-39-x86_64-linux-gnu.so'
Modules/Setup
is used to generate Makefile targets to build C extensions.
At the beginning of the files, C extensions are built as built-in modules.
Extensions defined after the *shared*
marker are built as dynamic libraries.
The PyAPI_FUNC()
, PyAPI_DATA()
and
PyMODINIT_FUNC
macros of Include/exports.h
are defined
differently depending if the Py_BUILD_CORE_MODULE
macro is defined:
Use
Py_EXPORTED_SYMBOL
if thePy_BUILD_CORE_MODULE
is definedUse
Py_IMPORTED_SYMBOL
otherwise.
If the Py_BUILD_CORE_BUILTIN
macro is used by mistake on a C extension
built as a shared library, its PyInit_xxx()
function is not exported,
causing an ImportError
on import.
3.5. Compiler and linker flags¶
Options set by the ./configure
script and environment variables and used by
Makefile
.
3.5.1. Preprocessor flags¶
- CONFIGURE_CPPFLAGS¶
Value of
CPPFLAGS
variable passed to the./configure
script.Added in version 3.6.
- CPPFLAGS¶
(Objective) C/C++ preprocessor flags, e.g.
-Iinclude_dir
if you have headers in a nonstandard directory include_dir.Both
CPPFLAGS
andLDFLAGS
need to contain the shell’s value to be able to build extension modules using the directories specified in the environment variables.
- BASECPPFLAGS¶
Added in version 3.4.
- PY_CPPFLAGS¶
Extra preprocessor flags added for building the interpreter object files.
Default:
$(BASECPPFLAGS) -I. -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) $(CPPFLAGS)
.Added in version 3.2.
3.5.2. Compiler flags¶
- CC¶
C compiler command.
Example:
gcc -pthread
.
- CXX¶
C++ compiler command.
Example:
g++ -pthread
.
- CFLAGS¶
C compiler flags.
- CFLAGS_NODIST¶
CFLAGS_NODIST
is used for building the interpreter and stdlib C extensions. Use it when a compiler flag should not be part ofCFLAGS
once Python is installed (gh-65320).In particular,
CFLAGS
should not contain:the compiler flag
-I
(for setting the search path for include files). The-I
flags are processed from left to right, and any flags inCFLAGS
would take precedence over user- and package-supplied-I
flags.hardening flags such as
-Werror
because distributions cannot control whether packages installed by users conform to such heightened standards.
Added in version 3.5.
- COMPILEALL_OPTS¶
Options passed to the
compileall
command line when building PYC files inmake install
. Default:-j0
.Added in version 3.12.
- EXTRA_CFLAGS¶
Extra C compiler flags.
- CONFIGURE_CFLAGS_NODIST¶
Value of
CFLAGS_NODIST
variable passed to the./configure
script.Added in version 3.5.
- BASECFLAGS¶
Base compiler flags.
- OPT¶
Optimization flags.
- CFLAGS_ALIASING¶
Strict or non-strict aliasing flags used to compile
Python/dtoa.c
.Added in version 3.7.
- CCSHARED¶
Compiler flags used to build a shared library.
For example,
-fPIC
is used on Linux and on BSD.
- CFLAGSFORSHARED¶
Extra C flags added for building the interpreter object files.
Default:
$(CCSHARED)
when--enable-shared
is used, or an empty string otherwise.
- PY_CFLAGS¶
Default:
$(BASECFLAGS) $(OPT) $(CONFIGURE_CFLAGS) $(CFLAGS) $(EXTRA_CFLAGS)
.
- PY_CFLAGS_NODIST¶
Default:
$(CONFIGURE_CFLAGS_NODIST) $(CFLAGS_NODIST) -I$(srcdir)/Include/internal
.Added in version 3.5.
- PY_STDMODULE_CFLAGS¶
C flags used for building the interpreter object files.
Default:
$(PY_CFLAGS) $(PY_CFLAGS_NODIST) $(PY_CPPFLAGS) $(CFLAGSFORSHARED)
.Added in version 3.7.
- PY_CORE_CFLAGS¶
Default:
$(PY_STDMODULE_CFLAGS) -DPy_BUILD_CORE
.Added in version 3.2.
- PY_BUILTIN_MODULE_CFLAGS¶
Compiler flags to build a standard library extension module as a built-in module, like the
posix
module.Default:
$(PY_STDMODULE_CFLAGS) -DPy_BUILD_CORE_BUILTIN
.Added in version 3.8.
- PURIFY¶
Purify command. Purify is a memory debugger program.
Default: empty string (not used).
3.5.3. Linker flags¶
- LINKCC¶
Linker command used to build programs like
python
and_testembed
.Default:
$(PURIFY) $(CC)
.
- CONFIGURE_LDFLAGS¶
Value of
LDFLAGS
variable passed to the./configure
script.Avoid assigning
CFLAGS
,LDFLAGS
, etc. so users can use them on the command line to append to these values without stomping the pre-set values.Added in version 3.2.
- LDFLAGS_NODIST¶
LDFLAGS_NODIST
is used in the same manner asCFLAGS_NODIST
. Use it when a linker flag should not be part ofLDFLAGS
once Python is installed (gh-65320).In particular,
LDFLAGS
should not contain:the compiler flag
-L
(for setting the search path for libraries). The-L
flags are processed from left to right, and any flags inLDFLAGS
would take precedence over user- and package-supplied-L
flags.
- CONFIGURE_LDFLAGS_NODIST¶
Value of
LDFLAGS_NODIST
variable passed to the./configure
script.Added in version 3.8.
- LDFLAGS¶
Linker flags, e.g.
-Llib_dir
if you have libraries in a nonstandard directory lib_dir.Both
CPPFLAGS
andLDFLAGS
need to contain the shell’s value to be able to build extension modules using the directories specified in the environment variables.
- LIBS¶
Linker flags to pass libraries to the linker when linking the Python executable.
Example:
-lrt
.
- LDSHARED¶
Command to build a shared library.
Default:
@LDSHARED@ $(PY_LDFLAGS)
.
- BLDSHARED¶
Command to build
libpython
shared library.Default:
@BLDSHARED@ $(PY_CORE_LDFLAGS)
.
- PY_LDFLAGS¶
Default:
$(CONFIGURE_LDFLAGS) $(LDFLAGS)
.
- PY_LDFLAGS_NODIST¶
Default:
$(CONFIGURE_LDFLAGS_NODIST) $(LDFLAGS_NODIST)
.Added in version 3.8.
- PY_CORE_LDFLAGS¶
Linker flags used for building the interpreter object files.
Added in version 3.8.
Footnotes
git clean -fdx
is an even more extreme way to “clean” your
checkout. It removes all files not known to Git.
When bug hunting using git bisect
, this is
recommended between probes
to guarantee a completely clean build. Use with care, as it
will delete all files not checked into Git, including your
new, uncommitted work.


Download Python 3.14 documentation

Download Python 3.14 documentation
Last updated on: Feb 18, 2026 (17:01 UTC).
Download an archive containing all the documentation for this version of Python:
| Format | Packed as .zip | Packed as .tar.bz2 |
|---|---|---|
| HTML | Download | Download |
| Plain text | Download | Download |
| Texinfo | Download | Download |
| EPUB | Download |
We no longer provide pre-built PDFs of the documentation.
To build a PDF archive, follow the instructions in the
Developer's Guide
and run make dist-pdf
in the Doc/
directory of a copy of the CPython repository.
See the directory listing for file sizes.
Problems
Open an issue if you have comments or suggestions for the Python documentation.