Common Object Structures

Common Object Structures¶
There are a large number of structures which are used in the definition of object types for Python. This section describes these structures and how they are used.
Base object types and macros¶
All Python objects ultimately share a small number of fields at the beginning
of the object’s representation in memory. These are represented by the
PyObject
and PyVarObject
types, which are defined, in turn,
by the expansions of some macros also used, whether directly or indirectly, in
the definition of all other Python objects. Additional macros can be found
under reference counting.
-
type PyObject¶
- Part of the Limited API. (Only some members are part of the stable ABI.)
All object types are extensions of this type. This is a type which contains the information Python needs to treat a pointer to an object as an object. In a normal “release” build, it contains only the object’s reference count and a pointer to the corresponding type object. Nothing is actually declared to be a
PyObject
, but every pointer to a Python object can be cast to a PyObject*.The members must not be accessed directly; instead use macros such as
Py_REFCNT
andPy_TYPE
.-
Py_ssize_t ob_refcnt¶
- Part of the Stable ABI.
The object’s reference count, as returned by
Py_REFCNT
. Do not use this field directly; instead use functions and macros such asPy_REFCNT
,Py_INCREF()
andPy_DecRef()
.The field type may be different from
Py_ssize_t
, depending on build configuration and platform.
-
PyTypeObject *ob_type¶
- Part of the Stable ABI.
The object’s type. Do not use this field directly; use
Py_TYPE
andPy_SET_TYPE()
instead.
-
Py_ssize_t ob_refcnt¶
-
type PyVarObject¶
- Part of the Limited API. (Only some members are part of the stable ABI.)
An extension of
PyObject
that adds theob_size
field. This is intended for objects that have some notion of length.As with
PyObject
, the members must not be accessed directly; instead use macros such asPy_SIZE
,Py_REFCNT
andPy_TYPE
.-
Py_ssize_t ob_size¶
- Part of the Stable ABI.
A size field, whose contents should be considered an object’s internal implementation detail.
Do not use this field directly; use
Py_SIZE
instead.Object creation functions such as
PyObject_NewVar()
will generally set this field to the requested size (number of items). After creation, arbitrary values can be stored inob_size
usingPy_SET_SIZE
.To get an object’s publicly exposed length, as returned by the Python function
len()
, usePyObject_Length()
instead.
-
Py_ssize_t ob_size¶
-
PyObject_HEAD¶
This is a macro used when declaring new types which represent objects without a varying length. The PyObject_HEAD macro expands to:
PyObject ob_base;
See documentation of
PyObject
above.
-
PyObject_VAR_HEAD¶
This is a macro used when declaring new types which represent objects with a length that varies from instance to instance. The PyObject_VAR_HEAD macro expands to:
PyVarObject ob_base;
See documentation of
PyVarObject
above.
-
PyTypeObject PyBaseObject_Type¶
- Part of the Stable ABI.
The base class of all other objects, the same as
object
in Python.
-
int Py_Is(PyObject *x, PyObject *y)¶
- Part of the Stable ABI since version 3.10.
Test if the x object is the y object, the same as
x is y
in Python.Added in version 3.10.
-
int Py_IsNone(PyObject *x)¶
- Part of the Stable ABI since version 3.10.
Test if an object is the
None
singleton, the same asx is None
in Python.Added in version 3.10.
-
int Py_IsTrue(PyObject *x)¶
- Part of the Stable ABI since version 3.10.
Test if an object is the
True
singleton, the same asx is True
in Python.Added in version 3.10.
-
int Py_IsFalse(PyObject *x)¶
- Part of the Stable ABI since version 3.10.
Test if an object is the
False
singleton, the same asx is False
in Python.Added in version 3.10.
-
PyTypeObject *Py_TYPE(PyObject *o)¶
- Return value: Borrowed reference. Part of the Stable ABI since version 3.14.
Get the type of the Python object o.
The returned reference is borrowed from o. Do not release it with
Py_DECREF()
or similar.
-
int Py_IS_TYPE(PyObject *o, PyTypeObject *type)¶
Return non-zero if the object o type is type. Return zero otherwise. Equivalent to:
Py_TYPE(o) == type
.Added in version 3.9.
-
void Py_SET_TYPE(PyObject *o, PyTypeObject *type)¶
Set the type of object o to type, without any checking or reference counting.
This is a very low-level operation. Consider instead setting the Python attribute
__class__
usingPyObject_SetAttrString()
or similar.Note that assigning an incompatible type can lead to undefined behavior.
If type is a heap type, the caller must create a new reference to it. Similarly, if the old type of o is a heap type, the caller must release a reference to that type.
Added in version 3.9.
-
Py_ssize_t Py_SIZE(PyVarObject *o)¶
Get the
ob_size
field of o.Changed in version 3.11:
Py_SIZE()
is changed to an inline static function. The parameter type is no longer const PyVarObject*.
-
void Py_SET_SIZE(PyVarObject *o, Py_ssize_t size)¶
Set the
ob_size
field of o to size.Added in version 3.9.
-
PyObject_HEAD_INIT(type)¶
This is a macro which expands to initialization values for a new
PyObject
type. This macro expands to:_PyObject_EXTRA_INIT 1, type,
-
PyVarObject_HEAD_INIT(type, size)¶
This is a macro which expands to initialization values for a new
PyVarObject
type, including theob_size
field. This macro expands to:_PyObject_EXTRA_INIT 1, type, size,
Implementing functions and methods¶
-
type PyCFunction¶
- Part of the Stable ABI.
Type of the functions used to implement most Python callables in C. Functions of this type take two PyObject* parameters and return one such value. If the return value is
NULL
, an exception shall have been set. If notNULL
, the return value is interpreted as the return value of the function as exposed in Python. The function must return a new reference.The function signature is:
PyObject *PyCFunction(PyObject *self, PyObject *args);
-
type PyCFunctionWithKeywords¶
- Part of the Stable ABI.
Type of the functions used to implement Python callables in C with signature METH_VARARGS | METH_KEYWORDS. The function signature is:
PyObject *PyCFunctionWithKeywords(PyObject *self, PyObject *args, PyObject *kwargs);
-
type PyCFunctionFast¶
- Part of the Stable ABI since version 3.13.
Type of the functions used to implement Python callables in C with signature
METH_FASTCALL
. The function signature is:PyObject *PyCFunctionFast(PyObject *self, PyObject *const *args, Py_ssize_t nargs);
-
type PyCFunctionFastWithKeywords¶
- Part of the Stable ABI since version 3.13.
Type of the functions used to implement Python callables in C with signature METH_FASTCALL | METH_KEYWORDS. The function signature is:
PyObject *PyCFunctionFastWithKeywords(PyObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames);
-
type PyCMethod¶
Type of the functions used to implement Python callables in C with signature METH_METHOD | METH_FASTCALL | METH_KEYWORDS. The function signature is:
PyObject *PyCMethod(PyObject *self, PyTypeObject *defining_class, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
Added in version 3.9.
-
type PyMethodDef¶
- Part of the Stable ABI (including all members).
Structure used to describe a method of an extension type. This structure has four fields:
-
const char *ml_name¶
Name of the method.
-
PyCFunction ml_meth¶
Pointer to the C implementation.
-
int ml_flags¶
Flags bits indicating how the call should be constructed.
-
const char *ml_doc¶
Points to the contents of the docstring.
-
const char *ml_name¶
The ml_meth
is a C function pointer.
The functions may be of different
types, but they always return PyObject*. If the function is not of
the PyCFunction
, the compiler will require a cast in the method table.
Even though PyCFunction
defines the first parameter as
PyObject*, it is common that the method implementation uses the
specific C type of the self object.
The ml_flags
field is a bitfield which can include
the following flags.
The individual flags indicate either a calling convention or a binding
convention.
There are these calling conventions:
-
METH_VARARGS¶
- Part of the Stable ABI.
This is the typical calling convention, where the methods have the type
PyCFunction
. The function expects two PyObject* values. The first one is the self object for methods; for module functions, it is the module object. The second parameter (often called args) is a tuple object representing all arguments. This parameter is typically processed usingPyArg_ParseTuple()
orPyArg_UnpackTuple()
.
-
METH_KEYWORDS¶
Can only be used in certain combinations with other flags: METH_VARARGS | METH_KEYWORDS, METH_FASTCALL | METH_KEYWORDS and METH_METHOD | METH_FASTCALL | METH_KEYWORDS.
- METH_VARARGS | METH_KEYWORDS
Methods with these flags must be of type
PyCFunctionWithKeywords
. The function expects three parameters: self, args, kwargs where kwargs is a dictionary of all the keyword arguments or possiblyNULL
if there are no keyword arguments. The parameters are typically processed usingPyArg_ParseTupleAndKeywords()
.
-
METH_FASTCALL¶
- Part of the Stable ABI since version 3.7.
Fast calling convention supporting only positional arguments. The methods have the type
PyCFunctionFast
. The first parameter is self, the second parameter is a C array of PyObject* values indicating the arguments and the third parameter is the number of arguments (the length of the array).Added in version 3.7.
Changed in version 3.10:
METH_FASTCALL
is now part of the stable ABI.
- METH_FASTCALL | METH_KEYWORDS
Extension of
METH_FASTCALL
supporting also keyword arguments, with methods of typePyCFunctionFastWithKeywords
. Keyword arguments are passed the same way as in the vectorcall protocol: there is an additional fourth PyObject* parameter which is a tuple representing the names of the keyword arguments (which are guaranteed to be strings) or possiblyNULL
if there are no keywords. The values of the keyword arguments are stored in the args array, after the positional arguments.Added in version 3.7.
-
METH_METHOD¶
- Part of the Stable ABI since version 3.7.
Can only be used in the combination with other flags: METH_METHOD | METH_FASTCALL | METH_KEYWORDS.
- METH_METHOD | METH_FASTCALL | METH_KEYWORDS
Extension of METH_FASTCALL | METH_KEYWORDS supporting the defining class, that is, the class that contains the method in question. The defining class might be a superclass of
Py_TYPE(self)
.The method needs to be of type
PyCMethod
, the same as forMETH_FASTCALL | METH_KEYWORDS
withdefining_class
argument added afterself
.Added in version 3.9.
-
METH_NOARGS¶
- Part of the Stable ABI.
Methods without parameters don’t need to check whether arguments are given if they are listed with the
METH_NOARGS
flag. They need to be of typePyCFunction
. The first parameter is typically named self and will hold a reference to the module or object instance. In all cases the second parameter will beNULL
.The function must have 2 parameters. Since the second parameter is unused,
Py_UNUSED
can be used to prevent a compiler warning.
-
METH_O¶
- Part of the Stable ABI.
Methods with a single object argument can be listed with the
METH_O
flag, instead of invokingPyArg_ParseTuple()
with a"O"
argument. They have the typePyCFunction
, with the self parameter, and a PyObject* parameter representing the single argument.
These two constants are not used to indicate the calling convention but the binding when used with methods of classes. These may not be used for functions defined for modules. At most one of these flags may be set for any given method.
-
METH_CLASS¶
- Part of the Stable ABI.
The method will be passed the type object as the first parameter rather than an instance of the type. This is used to create class methods, similar to what is created when using the
classmethod()
built-in function.
-
METH_STATIC¶
- Part of the Stable ABI.
The method will be passed
NULL
as the first parameter rather than an instance of the type. This is used to create static methods, similar to what is created when using thestaticmethod()
built-in function.
One other constant controls whether a method is loaded in place of another definition with the same method name.
-
METH_COEXIST¶
- Part of the Stable ABI.
The method will be loaded in place of existing definitions. Without METH_COEXIST, the default is to skip repeated definitions. Since slot wrappers are loaded before the method table, the existence of a sq_contains slot, for example, would generate a wrapped method named
__contains__()
and preclude the loading of a corresponding PyCFunction with the same name. With the flag defined, the PyCFunction will be loaded in place of the wrapper object and will co-exist with the slot. This is helpful because calls to PyCFunctions are optimized more than wrapper object calls.
-
PyTypeObject PyCMethod_Type¶
The type object corresponding to Python C method objects. This is available as
types.BuiltinMethodType
in the Python layer.
-
int PyCMethod_Check(PyObject *op)¶
Return true if op is an instance of the
PyCMethod_Type
type or a subtype of it. This function always succeeds.
-
int PyCMethod_CheckExact(PyObject *op)¶
This is the same as
PyCMethod_Check()
, but does not account for subtypes.
-
PyObject *PyCMethod_New(PyMethodDef *ml, PyObject *self, PyObject *module, PyTypeObject *cls)¶
- Return value: New reference. Part of the Stable ABI since version 3.9.
Turn ml into a Python callable object. The caller must ensure that ml outlives the callable. Typically, ml is defined as a static variable.
The self parameter will be passed as the self argument to the C function in
ml->ml_meth
when invoked. self can beNULL
.The callable object’s
__module__
attribute can be set from the given module argument. module should be a Python string, which will be used as name of the module the function is defined in. If unavailable, it can be set toNone
orNULL
.See also
The cls parameter will be passed as the defining_class argument to the C function. Must be set if
METH_METHOD
is set onml->ml_flags
.Added in version 3.9.
-
PyTypeObject PyCFunction_Type¶
- Part of the Stable ABI.
The type object corresponding to Python C function objects. This is available as
types.BuiltinFunctionType
in the Python layer.
-
int PyCFunction_Check(PyObject *op)¶
Return true if op is an instance of the
PyCFunction_Type
type or a subtype of it. This function always succeeds.
-
int PyCFunction_CheckExact(PyObject *op)¶
This is the same as
PyCFunction_Check()
, but does not account for subtypes.
-
PyObject *PyCFunction_NewEx(PyMethodDef *ml, PyObject *self, PyObject *module)¶
- Return value: New reference. Part of the Stable ABI.
Equivalent to
PyCMethod_New(ml, self, module, NULL)
.
-
PyObject *PyCFunction_New(PyMethodDef *ml, PyObject *self)¶
- Return value: New reference. Part of the Stable ABI since version 3.4.
Equivalent to
PyCMethod_New(ml, self, NULL, NULL)
.
-
int PyCFunction_GetFlags(PyObject *func)¶
- Part of the Stable ABI.
Get the function’s flags on func as they were passed to
ml_flags
.If func is not a C function object, this fails with an exception. func must not be
NULL
.This function returns the function’s flags on success, and
-1
with an exception set on failure.
-
int PyCFunction_GET_FLAGS(PyObject *func)¶
This is the same as
PyCFunction_GetFlags()
, but without error or type checking.
-
PyCFunction PyCFunction_GetFunction(PyObject *func)¶
- Part of the Stable ABI.
Get the function pointer on func as it was passed to
ml_meth
.If func is not a C function object, this fails with an exception. func must not be
NULL
.This function returns the function pointer on success, and
NULL
with an exception set on failure.
-
int PyCFunction_GET_FUNCTION(PyObject *func)¶
This is the same as
PyCFunction_GetFunction()
, but without error or type checking.
-
PyObject *PyCFunction_GetSelf(PyObject *func)¶
- Part of the Stable ABI.
Get the “self” object on func. This is the object that would be passed to the first argument of a
PyCFunction
. For C function objects created through aPyMethodDef
on aPyModuleDef
, this is the resulting module object.If func is not a C function object, this fails with an exception. func must not be
NULL
.This function returns a borrowed reference to the “self” object on success, and
NULL
with an exception set on failure.
-
PyObject *PyCFunction_GET_SELF(PyObject *func)¶
This is the same as
PyCFunction_GetSelf()
, but without error or type checking.
Accessing attributes of extension types¶
-
type PyMemberDef¶
- Part of the Stable ABI (including all members).
Structure which describes an attribute of a type which corresponds to a C struct member. When defining a class, put a NULL-terminated array of these structures in the
tp_members
slot.Its fields are, in order:
-
const char *name¶
Name of the member. A NULL value marks the end of a
PyMemberDef[]
array.The string should be static, no copy is made of it.
-
int type¶
The type of the member in the C struct. See Member types for the possible values.
-
Py_ssize_t offset¶
The offset in bytes that the member is located on the type’s object struct.
-
int flags¶
Zero or more of the Member flags, combined using bitwise OR.
-
const char *doc¶
The docstring, or NULL. The string should be static, no copy is made of it. Typically, it is defined using
PyDoc_STR
.
By default (when
flags
is0
), members allow both read and write access. Use thePy_READONLY
flag for read-only access. Certain types, likePy_T_STRING
, implyPy_READONLY
. OnlyPy_T_OBJECT_EX
(and legacyT_OBJECT
) members can be deleted.For heap-allocated types (created using
PyType_FromSpec()
or similar),PyMemberDef
may contain a definition for the special member"__vectorcalloffset__"
, corresponding totp_vectorcall_offset
in type objects. This member must be defined withPy_T_PYSSIZET
, and eitherPy_READONLY
orPy_READONLY | Py_RELATIVE_OFFSET
. For example:static PyMemberDef spam_type_members[] = { {"__vectorcalloffset__", Py_T_PYSSIZET, offsetof(Spam_object, vectorcall), Py_READONLY}, {NULL} /* Sentinel */ };
(You may need to
#include <stddef.h>
foroffsetof()
.)The legacy offsets
tp_dictoffset
andtp_weaklistoffset
can be defined similarly using"__dictoffset__"
and"__weaklistoffset__"
members, but extensions are strongly encouraged to usePy_TPFLAGS_MANAGED_DICT
andPy_TPFLAGS_MANAGED_WEAKREF
instead.Changed in version 3.12:
PyMemberDef
is always available. Previously, it required including"structmember.h"
.Changed in version 3.14:
Py_RELATIVE_OFFSET
is now allowed for"__vectorcalloffset__"
,"__dictoffset__"
and"__weaklistoffset__"
. -
const char *name¶
-
PyObject *PyMember_GetOne(const char *obj_addr, struct PyMemberDef *m)¶
- Part of the Stable ABI.
Get an attribute belonging to the object at address obj_addr. The attribute is described by
PyMemberDef
m. ReturnsNULL
on error.Changed in version 3.12:
PyMember_GetOne
is always available. Previously, it required including"structmember.h"
.
-
int PyMember_SetOne(char *obj_addr, struct PyMemberDef *m, PyObject *o)¶
- Part of the Stable ABI.
Set an attribute belonging to the object at address obj_addr to object o. The attribute to set is described by
PyMemberDef
m. Returns0
if successful and a negative value on failure.Changed in version 3.12:
PyMember_SetOne
is always available. Previously, it required including"structmember.h"
.
Member flags¶
The following flags can be used with PyMemberDef.flags
:
-
Py_READONLY¶
- Part of the Stable ABI since version 3.12.
Not writable.
-
Py_AUDIT_READ¶
- Part of the Stable ABI since version 3.12.
Emit an
object.__getattr__
audit event before reading.
-
Py_RELATIVE_OFFSET¶
- Part of the Stable ABI since version 3.12.
Indicates that the
offset
of thisPyMemberDef
entry indicates an offset from the subclass-specific data, rather than fromPyObject
.Can only be used as part of the
Py_tp_members
slot
when creating a class using negativebasicsize
. It is mandatory in that case. When settingtp_members
from the slot during class creation, Python clears the flag and setsPyMemberDef.offset
to the offset from thePyObject
struct.
Changed in version 3.10: The RESTRICTED
, READ_RESTRICTED
and
WRITE_RESTRICTED
macros available with
#include "structmember.h"
are deprecated.
READ_RESTRICTED
and RESTRICTED
are equivalent to
Py_AUDIT_READ
; WRITE_RESTRICTED
does nothing.
Changed in version 3.12: The READONLY
macro was renamed to Py_READONLY
.
The PY_AUDIT_READ
macro was renamed with the Py_
prefix.
The new names are now always available.
Previously, these required #include "structmember.h"
.
The header is still available and it provides the old names.
Member types¶
PyMemberDef.type
can be one of the following macros corresponding
to various C types.
When the member is accessed in Python, it will be converted to the
equivalent Python type.
When it is set from Python, it will be converted back to the C type.
If that is not possible, an exception such as TypeError
or
ValueError
is raised.
Unless marked (D), attributes defined this way cannot be deleted
using e.g. del
or delattr()
.
Macro name |
C type |
Python type |
|---|---|---|
|
char |
|
|
short |
|
|
int |
|
|
long |
|
|
long long |
|
|
unsigned char |
|
|
unsigned int |
|
|
unsigned short |
|
|
unsigned long |
|
|
unsigned long long |
|
|
||
|
float |
|
|
double |
|
|
char (written as 0 or 1) |
|
|
const char* (*) |
|
|
const char[] (*) |
|
|
char (0-127) |
|
|
|
(*): Zero-terminated, UTF8-encoded C string. With
Py_T_STRING
the C representation is a pointer; withPy_T_STRING_INPLACE
the string is stored directly in the structure.(**): String of length 1. Only ASCII is accepted.
(RO): Implies
Py_READONLY
.(D): Can be deleted, in which case the pointer is set to
NULL
. Reading aNULL
pointer raisesAttributeError
.
Added in version 3.12: In previous versions, the macros were only available with
#include "structmember.h"
and were named without the Py_
prefix
(e.g. as T_INT
).
The header is still available and contains the old names, along with
the following deprecated types:
-
T_OBJECT¶
Like
Py_T_OBJECT_EX
, butNULL
is converted toNone
. This results in surprising behavior in Python: deleting the attribute effectively sets it toNone
.
-
T_NONE¶
Always
None
. Must be used withPy_READONLY
.
Defining Getters and Setters¶
-
type PyGetSetDef¶
- Part of the Stable ABI (including all members).
Structure to define property-like access for a type. See also description of the
PyTypeObject.tp_getset
slot.-
const char *name¶
attribute name
-
setter set¶
Optional C function to set or delete the attribute. If
NULL
, the attribute is read-only.
-
const char *doc¶
optional docstring
-
void *closure¶
Optional user data pointer, providing additional data for getter and setter.
-
const char *name¶
-
typedef PyObject *(*getter)(PyObject*, void*)¶
- Part of the Stable ABI.
The
get
function takes one PyObject* parameter (the instance) and a user data pointer (the associatedclosure
):It should return a new reference on success or
NULL
with a set exception on failure.
-
typedef int (*setter)(PyObject*, PyObject*, void*)¶
- Part of the Stable ABI.
set
functions take two PyObject* parameters (the instance and the value to be set) and a user data pointer (the associatedclosure
):In case the attribute should be deleted the second parameter is
NULL
. Should return0
on success or-1
with a set exception on failure.


API and ABI Versioning

API and ABI Versioning¶
Build-time version constants¶
CPython exposes its version number in the following macros.
Note that these correspond to the version code is built with.
See Py_Version
for the version used at run time.
See C API Stability for a discussion of API and ABI stability across versions.
-
PY_MAJOR_VERSION¶
The
3
in3.4.1a2
.
-
PY_MINOR_VERSION¶
The
4
in3.4.1a2
.
-
PY_MICRO_VERSION¶
The
1
in3.4.1a2
.
-
PY_RELEASE_LEVEL¶
The
a
in3.4.1a2
. This can be0xA
for alpha,0xB
for beta,0xC
for release candidate or0xF
for final.
-
PY_RELEASE_SERIAL¶
The
2
in3.4.1a2
. Zero for final releases.
-
PY_VERSION_HEX¶
The Python version number encoded in a single integer. See
Py_PACK_FULL_VERSION()
for the encoding details.Use this for numeric comparisons, for example,
#if PY_VERSION_HEX >= ...
.
These macros are defined in Include/patchlevel.h.
Run-time version¶
-
const unsigned long Py_Version¶
- Part of the Stable ABI since version 3.11.
The Python runtime version number encoded in a single constant integer. See
Py_PACK_FULL_VERSION()
for the encoding details. This contains the Python version used at run time.Use this for numeric comparisons, for example,
if (Py_Version >= ...)
.Added in version 3.11.
Bit-packing macros¶
-
uint32_t Py_PACK_FULL_VERSION(int major, int minor, int micro, int release_level, int release_serial)¶
- Part of the Stable ABI since version 3.14.
Return the given version, encoded as a single 32-bit integer with the following structure:
Argument
No. of bits
Bit mask
Bit shift
Example values
3.4.1a2
3.10.0
major
8
0xFF000000
24
0x03
0x03
minor
8
0x00FF0000
16
0x04
0x0A
micro
8
0x0000FF00
8
0x01
0x00
release_level
4
0x000000F0
4
0xA
0xF
release_serial
4
0x0000000F
0
0x2
0x0
For example:
Version
Py_PACK_FULL_VERSION
argumentsEncoded version
3.4.1a2
(3, 4, 1, 0xA, 2)
0x030401a2
3.10.0
(3, 10, 0, 0xF, 0)
0x030a00f0
Out-of range bits in the arguments are ignored. That is, the macro can be defined as:
#ifndef Py_PACK_FULL_VERSION #define Py_PACK_FULL_VERSION(X, Y, Z, LEVEL, SERIAL) ( \ (((X) & 0xff) << 24) | \ (((Y) & 0xff) << 16) | \ (((Z) & 0xff) << 8) | \ (((LEVEL) & 0xf) << 4) | \ (((SERIAL) & 0xf) << 0)) #endif
Py_PACK_FULL_VERSION
is primarily a macro, intended for use in#if
directives, but it is also available as an exported function.Added in version 3.14.
-
uint32_t Py_PACK_VERSION(int major, int minor)¶
- Part of the Stable ABI since version 3.14.
Equivalent to
Py_PACK_FULL_VERSION(major, minor, 0, 0, 0)
. The result does not correspond to any Python release, but is useful in numeric comparisons.Added in version 3.14.


Lexical analysis

2. Lexical analysis¶
A Python program is read by a parser. Input to the parser is a stream of tokens, generated by the lexical analyzer (also known as the tokenizer). This chapter describes how the lexical analyzer produces these tokens.
The lexical analyzer determines the program text’s encoding
(UTF-8 by default), and decodes the text into
source characters.
If the text cannot be decoded, a SyntaxError
is raised.
Next, the lexical analyzer uses the source characters to generate a stream of tokens. The type of a generated token generally depends on the next source character to be processed. Similarly, other special behavior of the analyzer depends on the first source character that hasn’t yet been processed. The following table gives a quick summary of these source characters, with links to sections that contain more information.
Character |
Next token (or other relevant documentation) |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2.1. Line structure¶
A Python program is divided into a number of logical lines.
2.1.1. Logical lines¶
The end of a logical line is represented by the token NEWLINE
.
Statements cannot cross logical line boundaries except where NEWLINE
is allowed by the syntax (e.g., between statements in compound statements).
A logical line is constructed from one or more physical lines by following
the explicit or implicit
line joining rules.
2.1.2. Physical lines¶
A physical line is a sequence of characters terminated by one the following end-of-line sequences:
the Unix form using ASCII LF (linefeed),
the Windows form using the ASCII sequence CR LF (return followed by linefeed),
the ‘Classic Mac OS’ form using the ASCII CR (return) character.
Regardless of platform, each of these sequences is replaced by a single ASCII LF (linefeed) character. (This is done even inside string literals.) Each line can use any of the sequences; they do not need to be consistent within a file.
The end of input also serves as an implicit terminator for the final physical line.
Formally:
newline: <ASCII LF> | <ASCII CR> <ASCII LF> | <ASCII CR>
2.1.4. Encoding declarations¶
If a comment in the first or second line of the Python script matches the
regular expression coding[=:]\s*([-\w.]+)
, this comment is processed as an
encoding declaration; the first group of this expression names the encoding of
the source code file. The encoding declaration must appear on a line of its
own. If it is the second line, the first line must also be a comment-only line.
The recommended forms of an encoding expression are
# -*- coding: <encoding-name> -*-
which is recognized also by GNU Emacs, and
# vim:fileencoding=<encoding-name>
which is recognized by Bram Moolenaar’s VIM.
If no encoding declaration is found, the default encoding is UTF-8. If the
implicit or explicit encoding of a file is UTF-8, an initial UTF-8 byte-order
mark (b'\xef\xbb\xbf'
) is ignored rather than being a syntax error.
If an encoding is declared, the encoding name must be recognized by Python (see Standard Encodings). The encoding is used for all lexical analysis, including string literals, comments and identifiers.
All lexical analysis, including string literals, comments and identifiers, works on Unicode text decoded using the source encoding. Any Unicode code point, except the NUL control character, can appear in Python source.
source_character: <any Unicode code point, except NUL>
2.1.5. Explicit line joining¶
Two or more physical lines may be joined into logical lines using backslash
characters (\
), as follows: when a physical line ends in a backslash that is
not part of a string literal or comment, it is joined with the following forming
a single logical line, deleting the backslash and the following end-of-line
character. For example:
if 1900 < year < 2100 and 1 <= month <= 12 \
and 1 <= day <= 31 and 0 <= hour < 24 \
and 0 <= minute < 60 and 0 <= second < 60: # Looks like a valid date
return 1
A line ending in a backslash cannot carry a comment. A backslash does not continue a comment. A backslash does not continue a token except for string literals (i.e., tokens other than string literals cannot be split across physical lines using a backslash). A backslash is illegal elsewhere on a line outside a string literal.
2.1.6. Implicit line joining¶
Expressions in parentheses, square brackets or curly braces can be split over more than one physical line without using backslashes. For example:
month_names = ['Januari', 'Februari', 'Maart', # These are the
'April', 'Mei', 'Juni', # Dutch names
'Juli', 'Augustus', 'September', # for the months
'Oktober', 'November', 'December'] # of the year
Implicitly continued lines can carry comments. The indentation of the continuation lines is not important. Blank continuation lines are allowed. There is no NEWLINE token between implicit continuation lines. Implicitly continued lines can also occur within triple-quoted strings (see below); in that case they cannot carry comments.
2.1.7. Blank lines¶
A logical line that contains only spaces, tabs, formfeeds and possibly a
comment, is ignored (i.e., no NEWLINE
token is generated).
During interactive input of statements, handling of a blank line may differ
depending on the implementation of the read-eval-print loop.
In the standard interactive interpreter, an entirely blank logical line (that
is, one containing not even whitespace or a comment) terminates a multi-line
statement.
2.1.8. Indentation¶
Leading whitespace (spaces and tabs) at the beginning of a logical line is used to compute the indentation level of the line, which in turn is used to determine the grouping of statements.
Tabs are replaced (from left to right) by one to eight spaces such that the total number of characters up to and including the replacement is a multiple of eight (this is intended to be the same rule as used by Unix). The total number of spaces preceding the first non-blank character then determines the line’s indentation. Indentation cannot be split over multiple physical lines using backslashes; the whitespace up to the first backslash determines the indentation.
Indentation is rejected as inconsistent if a source file mixes tabs and spaces
in a way that makes the meaning dependent on the worth of a tab in spaces; a
TabError
is raised in that case.
Cross-platform compatibility note: because of the nature of text editors on non-UNIX platforms, it is unwise to use a mixture of spaces and tabs for the indentation in a single source file. It should also be noted that different platforms may explicitly limit the maximum indentation level.
A formfeed character may be present at the start of the line; it will be ignored for the indentation calculations above. Formfeed characters occurring elsewhere in the leading whitespace have an undefined effect (for instance, they may reset the space count to zero).
The indentation levels of consecutive lines are used to generate
INDENT
and DEDENT
tokens, using a stack,
as follows.
Before the first line of the file is read, a single zero is pushed on the stack;
this will never be popped off again. The numbers pushed on the stack will
always be strictly increasing from bottom to top. At the beginning of each
logical line, the line’s indentation level is compared to the top of the stack.
If it is equal, nothing happens. If it is larger, it is pushed on the stack, and
one INDENT
token is generated. If it is smaller, it must be one of the
numbers occurring on the stack; all numbers on the stack that are larger are
popped off, and for each number popped off a DEDENT
token is generated.
At the end of the file, a DEDENT
token is generated for each number
remaining on the stack that is larger than zero.
Here is an example of a correctly (though confusingly) indented piece of Python code:
def perm(l):
# Compute the list of all permutations of l
if len(l) <= 1:
return [l]
r = []
for i in range(len(l)):
s = l[:i] + l[i+1:]
p = perm(s)
for x in p:
r.append(l[i:i+1] + x)
return r
The following example shows various indentation errors:
def perm(l): # error: first line indented
for i in range(len(l)): # error: not indented
s = l[:i] + l[i+1:]
p = perm(l[:i] + l[i+1:]) # error: unexpected indent
for x in p:
r.append(l[i:i+1] + x)
return r # error: inconsistent dedent
(Actually, the first three errors are detected by the parser; only the last
error is found by the lexical analyzer — the indentation of return r
does
not match a level popped off the stack.)
2.1.9. Whitespace between tokens¶
Except at the beginning of a logical line or in string literals, the whitespace characters space, tab and formfeed can be used interchangeably to separate tokens:
whitespace: ' ' | tab | formfeed
Whitespace is needed between two tokens only if their concatenation
could otherwise be interpreted as a different token. For example, ab
is one
token, but a b
is two tokens. However, +a
and + a
both produce
two tokens, +
and a
, as +a
is not a valid token.
2.1.10. End marker¶
At the end of non-interactive input, the lexical analyzer generates an
ENDMARKER
token.
2.2. Other tokens¶
Besides NEWLINE
, INDENT
and DEDENT
,
the following categories of tokens exist:
identifiers and keywords (NAME
), literals (such as
NUMBER
and STRING
), and other symbols
(operators and delimiters, OP
).
Whitespace characters (other than logical line terminators, discussed earlier)
are not tokens, but serve to delimit tokens.
Where ambiguity exists, a token comprises the longest possible string that
forms a legal token, when read from left to right.
2.3. Names (identifiers and keywords)¶
NAME
tokens represent identifiers, keywords, and
soft keywords.
Names are composed of the following characters:
uppercase and lowercase letters (
A-Z
anda-z
),the underscore (
_
),digits (
0
through9
), which cannot appear as the first character, andnon-ASCII characters. Valid names may only contain “letter-like” and “digit-like” characters; see Non-ASCII characters in names for details.
Names must contain at least one character, but have no upper length limit. Case is significant.
Formally, names are described by the following lexical definitions:
NAME:name_start
name_continue
* name_start: "a"..."z" | "A"..."Z" | "_" | <non-ASCII character> name_continue: name_start | "0"..."9" identifier: <NAME
, except keywords>
Note that not all names matched by this grammar are valid; see Non-ASCII characters in names for details.
2.3.1. Keywords¶
The following names are used as reserved words, or keywords of the language, and cannot be used as ordinary identifiers. They must be spelled exactly as written here:
False await else import pass
None break except in raise
True class finally is return
and continue for lambda try
as def from nonlocal while
assert del global not with
async elif if or yield
2.3.2. Soft Keywords¶
Added in version 3.10.
Some names are only reserved under specific contexts. These are known as soft keywords:
These syntactically act as keywords in their specific contexts, but this distinction is done at the parser level, not when tokenizing.
As soft keywords, their use in the grammar is possible while still preserving compatibility with existing code that uses these names as identifier names.
Changed in version 3.12: type
is now a soft keyword.
2.3.3. Reserved classes of identifiers¶
Certain classes of identifiers (besides keywords) have special meanings. These classes are identified by the patterns of leading and trailing underscore characters:
_*
Not imported by
from module import *
._
In a
case
pattern within amatch
statement,_
is a soft keyword that denotes a wildcard.Separately, the interactive interpreter makes the result of the last evaluation available in the variable
_
. (It is stored in thebuiltins
module, alongside built-in functions likeprint
.)Elsewhere,
_
is a regular identifier. It is often used to name “special” items, but it is not special to Python itself.Note
The name
_
is often used in conjunction with internationalization; refer to the documentation for thegettext
module for more information on this convention.It is also commonly used for unused variables.
__*__
System-defined names, informally known as “dunder” names. These names are defined by the interpreter and its implementation (including the standard library). Current system names are discussed in the Special method names section and elsewhere. More will likely be defined in future versions of Python. Any use of
__*__
names, in any context, that does not follow explicitly documented use, is subject to breakage without warning.__*
Class-private names. Names in this category, when used within the context of a class definition, are re-written to use a mangled form to help avoid name clashes between “private” attributes of base and derived classes. See section Identifiers (Names).
2.3.4. Non-ASCII characters in names¶
Names that contain non-ASCII characters need additional normalization
and validation beyond the rules and grammar explained
above.
For example, ř_1
, 蛇
, or साँप
are valid names, but r〰2
,
€
, or 🐍
are not.
This section explains the exact rules.
All names are converted into the normalization form NFKC while parsing.
This means that, for example, some typographic variants of characters are
converted to their “basic” form. For example, ﬁⁿₐˡᵢᶻₐᵗᵢᵒₙ
normalizes to
finalization
, so Python treats them as the same name:
>>> ﬁⁿₐˡᵢᶻₐᵗᵢᵒₙ = 3
>>> finalization
3
Note
Normalization is done at the lexical level only.
Run-time functions that take names as strings generally do not normalize
their arguments.
For example, the variable defined above is accessible at run time in the
globals()
dictionary as globals()["finalization"]
but not
globals()["ﬁⁿₐˡᵢᶻₐᵗᵢᵒₙ"]
.
Similarly to how ASCII-only names must contain only letters, digits and
the underscore, and cannot start with a digit, a valid name must
start with a character in the “letter-like” set xid_start
,
and the remaining characters must be in the “letter- and digit-like” set
xid_continue
.
These sets based on the XID_Start and XID_Continue sets as defined by the
Unicode standard annex UAX-31.
Python’s xid_start
additionally includes the underscore (_
).
Note that Python does not necessarily conform to UAX-31.
A non-normative listing of characters in the XID_Start and XID_Continue
sets as defined by Unicode is available in the DerivedCoreProperties.txt
file in the Unicode Character Database.
For reference, the construction rules for the xid_*
sets are given below.
The set id_start
is defined as the union of:
Unicode category
<Lu>
- uppercase letters (includesA
toZ
)Unicode category
<Ll>
- lowercase letters (includesa
toz
)Unicode category
<Lt>
- titlecase lettersUnicode category
<Lm>
- modifier lettersUnicode category
<Lo>
- other lettersUnicode category
<Nl>
- letter numbers{
"_"
} - the underscore<Other_ID_Start>
- an explicit set of characters in PropList.txt to support backwards compatibility
The set xid_start
then closes this set under NFKC normalization, by
removing all characters whose normalization is not of the form
id_start id_continue*
.
The set id_continue
is defined as the union of:
id_start
(see above)Unicode category
<Nd>
- decimal numbers (includes0
to9
)Unicode category
<Pc>
- connector punctuationsUnicode category
<Mn>
- nonspacing marksUnicode category
<Mc>
- spacing combining marks<Other_ID_Continue>
- another explicit set of characters in PropList.txt to support backwards compatibility
Again, xid_continue
closes this set under NFKC normalization.
Unicode categories use the version of the Unicode Character Database as
included in the unicodedata
module.
2.4. Literals¶
Literals are notations for constant values of some built-in types.
In terms of lexical analysis, Python has string, bytes and numeric literals.
Other “literals” are lexically denoted using keywords
(None
, True
, False
) and the special
ellipsis token (...
).
2.5. String and Bytes literals¶
String literals are text enclosed in single quotes ('
) or double
quotes ("
). For example:
"spam"
'eggs'
The quote used to start the literal also terminates it, so a string literal can only contain the other quote (except with escape sequences, see below). For example:
'Say "Hello", please.'
"Don't do that!"
Except for this limitation, the choice of quote character ('
or "
)
does not affect how the literal is parsed.
Inside a string literal, the backslash (\
) character introduces an
escape sequence, which has special meaning depending on the character
after the backslash.
For example, \"
denotes the double quote character, and does not end
the string:
>>> print("Say \"Hello\" to everyone!")
Say "Hello" to everyone!
See escape sequences below for a full list of such sequences, and more details.
2.5.1. Triple-quoted strings¶
Strings can also be enclosed in matching groups of three single or double quotes. These are generally referred to as triple-quoted strings:
"""This is a triple-quoted string."""
In triple-quoted literals, unescaped quotes are allowed (and are
retained), except that three unescaped quotes in a row terminate the literal,
if they are of the same kind ('
or "
) used at the start:
"""This string has "quotes" inside."""
Unescaped newlines are also allowed and retained:
'''This triple-quoted string
continues on the next line.'''
2.5.2. String prefixes¶
String literals can have an optional prefix that influences how the content of the literal is parsed, for example:
b"data"
f'{result=}'
The allowed prefixes are:
r
: Raw stringf
: Formatted string literal (“f-string”)t
: Template string literal (“t-string”)u
: No effect (allowed for backwards compatibility)
See the linked sections for details on each type.
Prefixes are case-insensitive (for example, ‘B
’ works the same as ‘b
’).
The ‘r
’ prefix can be combined with ‘f
’, ‘t
’ or ‘b
’, so ‘fr
’,
‘rf
’, ‘tr
’, ‘rt
’, ‘br
’, and ‘rb
’ are also valid prefixes.
Added in version 3.3: The 'rb'
prefix of raw bytes literals has been added as a synonym
of 'br'
.
Support for the unicode legacy literal (u'value'
) was reintroduced
to simplify the maintenance of dual Python 2.x and 3.x codebases.
See PEP 414 for more information.
2.5.3. Formal grammar¶
String literals, except “f-strings” and “t-strings”, are described by the following lexical definitions.
These definitions use negative lookaheads (!
)
to indicate that an ending quote ends the literal.
STRING: [stringprefix
] (stringcontent
) stringprefix: <("r" | "u" | "b" | "br" | "rb"), case-insensitive> stringcontent: | "'''" ( !"'''"longstringitem
)* "'''" | '"""' ( !'"""'longstringitem
)* '"""' | "'" ( !"'"stringitem
)* "'" | '"' ( !'"'stringitem
)* '"' stringitem:stringchar
|stringescapeseq
stringchar: <anysource_character
, except backslash and newline> longstringitem:stringitem
| newline stringescapeseq: "\" <anysource_character
>
Note that as in all lexical definitions, whitespace is significant. In particular, the prefix (if any) must be immediately followed by the starting quote.
2.5.4. Escape sequences¶
Unless an ‘r
’ or ‘R
’ prefix is present, escape sequences in string and
bytes literals are interpreted according to rules similar to those used by
Standard C. The recognized escape sequences are:
Escape Sequence |
Meaning |
|---|---|
|
|
|
|
|
|
|
|
|
ASCII Bell (BEL) |
|
ASCII Backspace (BS) |
|
ASCII Formfeed (FF) |
|
ASCII Linefeed (LF) |
|
ASCII Carriage Return (CR) |
|
ASCII Horizontal Tab (TAB) |
|
ASCII Vertical Tab (VT) |
|
|
|
|
|
|
|
|
|
2.5.4.1. Ignored end of line¶
A backslash can be added at the end of a line to ignore the newline:
>>> 'This string will not include \
... backslashes or newline characters.'
'This string will not include backslashes or newline characters.'
The same result can be achieved using triple-quoted strings, or parentheses and string literal concatenation.
2.5.4.2. Escaped characters¶
To include a backslash in a non-raw Python string
literal, it must be doubled. The \\
escape sequence denotes a single
backslash character:
>>> print('C:\\Program Files')
C:\Program Files
Similarly, the \'
and \"
sequences denote the single and double
quote character, respectively:
>>> print('\' and \"')
' and "
2.5.4.3. Octal character¶
The sequence \ooo
denotes a character with the octal (base 8)
value ooo:
>>> '\120'
'P'
Up to three octal digits (0 through 7) are accepted.
In a bytes literal, character means a byte with the given value. In a string literal, it means a Unicode character with the given value.
Changed in version 3.11: Octal escapes with value larger than 0o377
(255) produce a
DeprecationWarning
.
Changed in version 3.12: Octal escapes with value larger than 0o377
(255) produce a
SyntaxWarning
.
In a future Python version they will raise a SyntaxError
.
2.5.4.4. Hexadecimal character¶
The sequence \xhh
denotes a character with the hex (base 16)
value hh:
>>> '\x50'
'P'
Unlike in Standard C, exactly two hex digits are required.
In a bytes literal, character means a byte with the given value. In a string literal, it means a Unicode character with the given value.
2.5.4.5. Named Unicode character¶
The sequence \N{name}
denotes a Unicode character
with the given name:
>>> '\N{LATIN CAPITAL LETTER P}'
'P'
>>> '\N{SNAKE}'
'🐍'
This sequence cannot appear in bytes literals.
Changed in version 3.3: Support for name aliases has been added.
2.5.4.6. Hexadecimal Unicode characters¶
These sequences \uxxxx
and \Uxxxxxxxx
denote the
Unicode character with the given hex (base 16) value.
Exactly four digits are required for \u
; exactly eight digits are
required for \U
.
The latter can encode any Unicode character.
>>> '\u1234'
'ሴ'
>>> '\U0001f40d'
'🐍'
These sequences cannot appear in bytes literals.
2.5.4.7. Unrecognized escape sequences¶
Unlike in Standard C, all unrecognized escape sequences are left in the string unchanged, that is, the backslash is left in the result:
>>> print('\q')
\q
>>> list('\q')
['\\', 'q']
Note that for bytes literals, the escape sequences only recognized in string
literals (\N...
, \u...
, \U...
) fall into the category of
unrecognized escapes.
Changed in version 3.6: Unrecognized escape sequences produce a DeprecationWarning
.
Changed in version 3.12: Unrecognized escape sequences produce a SyntaxWarning
.
In a future Python version they will raise a SyntaxError
.
2.5.5. Bytes literals¶
Bytes literals are always prefixed with ‘b
’ or ‘B
’; they produce an
instance of the bytes
type instead of the str
type.
They may only contain ASCII characters; bytes with a numeric value of 128
or greater must be expressed with escape sequences (typically
Hexadecimal character or Octal character):
>>> b'\x89PNG\r\n\x1a\n'
b'\x89PNG\r\n\x1a\n'
>>> list(b'\x89PNG\r\n\x1a\n')
[137, 80, 78, 71, 13, 10, 26, 10]
Similarly, a zero byte must be expressed using an escape sequence (typically
\0
or \x00
).
2.5.6. Raw string literals¶
Both string and bytes literals may optionally be prefixed with a letter ‘r
’
or ‘R
’; such constructs are called raw string literals
and raw bytes literals respectively and treat backslashes as
literal characters.
As a result, in raw string literals, escape sequences
are not treated specially:
>>> r'\d{4}-\d{2}-\d{2}'
'\\d{4}-\\d{2}-\\d{2}'
Even in a raw literal, quotes can be escaped with a backslash, but the
backslash remains in the result; for example, r"\""
is a valid string
literal consisting of two characters: a backslash and a double quote; r"\"
is not a valid string literal (even a raw string cannot end in an odd number of
backslashes). Specifically, a raw literal cannot end in a single backslash
(since the backslash would escape the following quote character). Note also
that a single backslash followed by a newline is interpreted as those two
characters as part of the literal, not as a line continuation.
2.5.7. f-strings¶
Added in version 3.6.
Changed in version 3.8: Added the debug specifier (=
)
Changed in version 3.12: Many restrictions on expressions within f-strings have been removed. Notably, nested strings, comments, and backslashes are now permitted.
A formatted string literal or f-string is a string literal
that is prefixed with ‘f
’ or ‘F
’.
Unlike other string literals, f-strings do not have a constant value.
They may contain replacement fields delimited by curly braces {}
.
Replacement fields contain expressions which are evaluated at run time.
For example:
>>> who = 'nobody'
>>> nationality = 'Spanish'
>>> f'{who.title()} expects the {nationality} Inquisition!'
'Nobody expects the Spanish Inquisition!'
Any doubled curly braces ({{
or }}
) outside replacement fields
are replaced with the corresponding single curly brace:
>>> print(f'{{...}}')
{...}
Other characters outside replacement fields are treated like in ordinary string literals. This means that escape sequences are decoded (except when a literal is also marked as a raw string), and newlines are possible in triple-quoted f-strings:
>>> name = 'Galahad'
>>> favorite_color = 'blue'
>>> print(f'{name}:\t{favorite_color}')
Galahad: blue
>>> print(rf"C:\Users\{name}")
C:\Users\Galahad
>>> print(f'''Three shall be the number of the counting
... and the number of the counting shall be three.''')
Three shall be the number of the counting
and the number of the counting shall be three.
Expressions in formatted string literals are treated like regular
Python expressions.
Each expression is evaluated in the context where the formatted string literal
appears, in order from left to right.
An empty expression is not allowed, and both lambda
and
assignment expressions :=
must be surrounded by explicit parentheses:
>>> f'{(half := 1/2)}, {half * 42}'
'0.5, 21.0'
Reusing the outer f-string quoting type inside a replacement field is permitted:
>>> a = dict(x=2)
>>> f"abc {a["x"]} def"
'abc 2 def'
Backslashes are also allowed in replacement fields and are evaluated the same way as in any other context:
>>> a = ["a", "b", "c"]
>>> print(f"List a contains:\n{"\n".join(a)}")
List a contains:
a
b
c
It is possible to nest f-strings:
>>> name = 'world'
>>> f'Repeated:{f' hello {name}' * 3}'
'Repeated: hello world hello world hello world'
Portable Python programs should not use more than 5 levels of nesting.
CPython implementation detail: CPython does not limit nesting of f-strings.
Replacement expressions can contain newlines in both single-quoted and
triple-quoted f-strings and they can contain comments.
Everything that comes after a #
inside a replacement field
is a comment (even closing braces and quotes).
This means that replacement fields with comments must be closed in a
different line:
>>> a = 2
>>> f"abc{a # This comment }" continues until the end of the line
... + 3}"
'abc5'
After the expression, replacement fields may optionally contain:
a debug specifier – an equal sign (
=
), optionally surrounded by whitespace on one or both sides;a conversion specifier –
!s
,!r
or!a
; and/ora format specifier prefixed with a colon (
:
).
See the Standard Library section on f-strings for details on how these fields are evaluated.
As that section explains, format specifiers are passed as the second argument
to the format()
function to format a replacement field value.
For example, they can be used to specify a field width and padding characters
using the Format Specification Mini-Language:
>>> number = 14.3
>>> f'{number:20.7f}'
' 14.3000000'
Top-level format specifiers may include nested replacement fields:
>>> field_size = 20
>>> precision = 7
>>> f'{number:{field_size}.{precision}f}'
' 14.3000000'
These nested fields may include their own conversion fields and format specifiers:
>>> number = 3
>>> f'{number:{field_size}}'
' 3'
>>> f'{number:{field_size:05}}'
'00000000000000000003'
However, these nested fields may not include more deeply nested replacement fields.
Formatted string literals cannot be used as docstrings, even if they do not include expressions:
>>> def foo():
... f"Not a docstring"
...
>>> print(foo.__doc__)
None
See also
PEP 498 – Literal String Interpolation
PEP 701 – Syntactic formalization of f-strings
str.format()
, which uses a related format string mechanism.
2.5.8. t-strings¶
Added in version 3.14.
A template string literal or t-string is a string literal
that is prefixed with ‘t
’ or ‘T
’.
These strings follow the same syntax rules as
formatted string literals.
For differences in evaluation rules, see the
Standard Library section on t-strings
2.5.9. Formal grammar for f-strings¶
F-strings are handled partly by the lexical analyzer, which produces the
tokens FSTRING_START
, FSTRING_MIDDLE
and FSTRING_END
, and partly by the parser, which handles
expressions in the replacement field.
The exact way the work is split is a CPython implementation detail.
Correspondingly, the f-string grammar is a mix of lexical and syntactic definitions.
Whitespace is significant in these situations:
There may be no whitespace in
FSTRING_START
(between the prefix and quote).Whitespace in
FSTRING_MIDDLE
is part of the literal string contents.In
fstring_replacement_field
, iff_debug_specifier
is present, all whitespace after the opening brace until thef_debug_specifier
, as well as whitespace immediately followingf_debug_specifier
, is retained as part of the expression.CPython implementation detail: The expression is not handled in the tokenization phase; it is retrieved from the source code using locations of the
{
token and the token after=
.
The FSTRING_MIDDLE
definition uses
negative lookaheads (!
)
to indicate special characters (backslash, newline, {
, }
) and
sequences (f_quote
).
fstring:FSTRING_START
fstring_middle
*FSTRING_END
FSTRING_START:fstringprefix
("'" | '"' | "'''" | '"""') FSTRING_END:f_quote
fstringprefix: <("f" | "fr" | "rf"), case-insensitive> f_debug_specifier: '=' f_quote: <the quote character(s) used in FSTRING_START> fstring_middle: |fstring_replacement_field
|FSTRING_MIDDLE
FSTRING_MIDDLE: | (!"\" !newline
!'{' !'}' !f_quote
)source_character
|stringescapeseq
| "{{" | "}}" | <newline, in triple-quoted f-strings only> fstring_replacement_field: | '{'f_expression
[f_debug_specifier
] [fstring_conversion
] [fstring_full_format_spec
] '}' fstring_conversion: | "!" ("s" | "r" | "a") fstring_full_format_spec: | ':'fstring_format_spec
* fstring_format_spec: |FSTRING_MIDDLE
|fstring_replacement_field
f_expression: | ','.(conditional_expression
| "*"or_expr
)+ [","] |yield_expression
Note
In the above grammar snippet, the f_quote
and FSTRING_MIDDLE
rules
are context-sensitive – they depend on the contents of FSTRING_START
of the nearest enclosing fstring
.
Constructing a more traditional formal grammar from this template is left as an exercise for the reader.
The grammar for t-strings is identical to the one for f-strings, with t instead of f at the beginning of rule and token names and in the prefix.
tstring: TSTRING_START tstring_middle* TSTRING_END <rest of the t-string grammar is omitted; see above>
2.6. Numeric literals¶
NUMBER
tokens represent numeric literals, of which there are
three types: integers, floating-point numbers, and imaginary numbers.
NUMBER:integer
|floatnumber
|imagnumber
The numeric value of a numeric literal is the same as if it were passed as a
string to the int
, float
or complex
class
constructor, respectively.
Note that not all valid inputs for those constructors are also valid literals.
Numeric literals do not include a sign; a phrase like -1
is
actually an expression composed of the unary operator ‘-
’ and the literal
1
.
2.6.1. Integer literals¶
Integer literals denote whole numbers. For example:
7
3
2147483647
There is no limit for the length of integer literals apart from what can be stored in available memory:
7922816251426433759354395033679228162514264337593543950336
Underscores can be used to group digits for enhanced readability, and are ignored for determining the numeric value of the literal. For example, the following literals are equivalent:
100_000_000_000
100000000000
1_00_00_00_00_000
Underscores can only occur between digits.
For example, _123
, 321_
, and 123__321
are not valid literals.
Integers can be specified in binary (base 2), octal (base 8), or hexadecimal
(base 16) using the prefixes 0b
, 0o
and 0x
, respectively.
Hexadecimal digits 10 through 15 are represented by letters A
-F
,
case-insensitive. For example:
0b100110111
0b_1110_0101
0o177
0o377
0xdeadbeef
0xDead_Beef
An underscore can follow the base specifier.
For example, 0x_1f
is a valid literal, but 0_x1f
and 0x__1f
are
not.
Leading zeros in a non-zero decimal number are not allowed.
For example, 0123
is not a valid literal.
This is for disambiguation with C-style octal literals, which Python used
before version 3.0.
Formally, integer literals are described by the following lexical definitions:
integer:decinteger
|bininteger
|octinteger
|hexinteger
|zerointeger
decinteger:nonzerodigit
(["_"]digit
)* bininteger: "0" ("b" | "B") (["_"]bindigit
)+ octinteger: "0" ("o" | "O") (["_"]octdigit
)+ hexinteger: "0" ("x" | "X") (["_"]hexdigit
)+ zerointeger: "0"+ (["_"] "0")* nonzerodigit: "1"..."9" digit: "0"..."9" bindigit: "0" | "1" octdigit: "0"..."7" hexdigit:digit
| "a"..."f" | "A"..."F"
Changed in version 3.6: Underscores are now allowed for grouping purposes in literals.
2.6.2. Floating-point literals¶
Floating-point (float) literals, such as 3.14
or 1.5
, denote
approximations of real numbers.
They consist of integer and fraction parts, each composed of decimal digits.
The parts are separated by a decimal point, .
:
2.71828
4.0
Unlike in integer literals, leading zeros are allowed.
For example, 077.010
is legal, and denotes the same number as 77.01
.
As in integer literals, single underscores may occur between digits to help readability:
96_485.332_123
3.14_15_93
Either of these parts, but not both, can be empty. For example:
10. # (equivalent to 10.0)
.001 # (equivalent to 0.001)
Optionally, the integer and fraction may be followed by an exponent:
the letter e
or E
, followed by an optional sign, +
or -
,
and a number in the same format as the integer and fraction parts.
The e
or E
represents “times ten raised to the power of”:
1.0e3 # (represents 1.0×10³, or 1000.0)
1.166e-5 # (represents 1.166×10⁻⁵, or 0.00001166)
6.02214076e+23 # (represents 6.02214076×10²³, or 602214076000000000000000.)
In floats with only integer and exponent parts, the decimal point may be omitted:
1e3 # (equivalent to 1.e3 and 1.0e3)
0e0 # (equivalent to 0.)
Formally, floating-point literals are described by the following lexical definitions:
floatnumber: |digitpart
"." [digitpart
] [exponent
] | "."digitpart
[exponent
] |digitpart
exponent
digitpart:digit
(["_"]digit
)* exponent: ("e" | "E") ["+" | "-"]digitpart
Changed in version 3.6: Underscores are now allowed for grouping purposes in literals.
2.6.3. Imaginary literals¶
Python has complex number objects, but no complex literals. Instead, imaginary literals denote complex numbers with a zero real part.
For example, in math, the complex number 3+4.2i is written
as the real number 3 added to the imaginary number 4.2i.
Python uses a similar syntax, except the imaginary unit is written as j
rather than i:
3+4.2j
This is an expression composed
of the integer literal 3
,
the operator ‘+
’,
and the imaginary literal 4.2j
.
Since these are three separate tokens, whitespace is allowed between them:
3 + 4.2j
No whitespace is allowed within each token.
In particular, the j
suffix, may not be separated from the number
before it.
The number before the j
has the same syntax as a floating-point literal.
Thus, the following are valid imaginary literals:
4.2j
3.14j
10.j
.001j
1e100j
3.14e-10j
3.14_15_93j
Unlike in a floating-point literal the decimal point can be omitted if the imaginary number only has an integer part. The number is still evaluated as a floating-point number, not an integer:
10j
0j
1000000000000000000000000j # equivalent to 1e+24j
The j
suffix is case-insensitive.
That means you can use J
instead:
3.14J # equivalent to 3.14j
Formally, imaginary literals are described by the following lexical definition:
imagnumber: (floatnumber
|digitpart
) ("j" | "J")
2.7. Operators and delimiters¶
The following grammar defines operator and delimiter tokens,
that is, the generic OP
token type.
A list of these tokens and their names
is also available in the token
module documentation.
OP: | assignment_operator | bitwise_operator | comparison_operator | enclosing_delimiter | other_delimiter | arithmetic_operator | "..." | other_op assignment_operator: "+=" | "-=" | "*=" | "**=" | "/=" | "//=" | "%=" | "&=" | "|=" | "^=" | "<<=" | ">>=" | "@=" | ":=" bitwise_operator: "&" | "|" | "^" | "~" | "<<" | ">>" comparison_operator: "<=" | ">=" | "<" | ">" | "==" | "!=" enclosing_delimiter: "(" | ")" | "[" | "]" | "{" | "}" other_delimiter: "," | ":" | "!" | ";" | "=" | "->" arithmetic_operator: "+" | "-" | "**" | "*" | "//" | "/" | "%" other_op: "." | "@"
Note
Generally, operators are used to combine expressions, while delimiters serve other purposes. However, there is no clear, formal distinction between the two categories.
Some tokens can serve as either operators or delimiters, depending on usage.
For example, *
is both the multiplication operator and a delimiter used
for sequence unpacking, and @
is both the matrix multiplication and
a delimiter that introduces decorators.
For some tokens, the distinction is unclear.
For example, some people consider .
, (
, and )
to be delimiters, while others
see the getattr()
operator and the function call operator(s).
Some of Python’s operators, like and
, or
, and not in
, use
keyword tokens rather than “symbols” (operator tokens).
A sequence of three consecutive periods (...
) has a special
meaning as an Ellipsis
literal.
2.1.3. Comments¶
A comment starts with a hash character (
#
) that is not part of a string literal, and ends at the end of the physical line. A comment signifies the end of the logical line unless the implicit line joining rules are invoked. Comments are ignored by the syntax.


Instrumenting CPython with DTrace and SystemTap¶
- author:
David Malcolm
- author:
Łukasz Langa
DTrace and SystemTap are monitoring tools, each providing a way to inspect what the processes on a computer system are doing. They both use domain-specific languages allowing a user to write scripts which:
filter which processes are to be observed
gather data from the processes of interest
generate reports on the data
As of Python 3.6, CPython can be built with embedded “markers”, also known as “probes”, that can be observed by a DTrace or SystemTap script, making it easier to monitor what the CPython processes on a system are doing.
CPython implementation detail: DTrace markers are implementation details of the CPython interpreter. No guarantees are made about probe compatibility between versions of CPython. DTrace scripts can stop working or work incorrectly without warning when changing CPython versions.
Enabling the static markers¶
macOS comes with built-in support for DTrace. On Linux, in order to build CPython with the embedded markers for SystemTap, the SystemTap development tools must be installed.
On a Linux machine, this can be done via:
$ yum install systemtap-sdt-devel
or:
$ sudo apt-get install systemtap-sdt-dev
CPython must then be configured with the --with-dtrace option
:
checking for --with-dtrace... yes
On macOS, you can list available DTrace probes by running a Python process in the background and listing all probes made available by the Python provider:
$ python3.6 -q &
$ sudo dtrace -l -P python$! # or: dtrace -l -m python3.6
ID PROVIDER MODULE FUNCTION NAME
29564 python18035 python3.6 _PyEval_EvalFrameDefault function-entry
29565 python18035 python3.6 dtrace_function_entry function-entry
29566 python18035 python3.6 _PyEval_EvalFrameDefault function-return
29567 python18035 python3.6 dtrace_function_return function-return
29568 python18035 python3.6 collect gc-done
29569 python18035 python3.6 collect gc-start
29570 python18035 python3.6 _PyEval_EvalFrameDefault line
29571 python18035 python3.6 maybe_dtrace_line line
On Linux, you can verify if the SystemTap static markers are present in the built binary by seeing if it contains a “.note.stapsdt” section.
$ readelf -S ./python | grep .note.stapsdt
[30] .note.stapsdt NOTE 0000000000000000 00308d78
If you’ve built Python as a shared library
(with the --enable-shared
configure option), you
need to look instead within the shared library. For example:
$ readelf -S libpython3.3dm.so.1.0 | grep .note.stapsdt
[29] .note.stapsdt NOTE 0000000000000000 00365b68
Sufficiently modern readelf can print the metadata:
$ readelf -n ./python
Displaying notes found at file offset 0x00000254 with length 0x00000020:
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 2.6.32
Displaying notes found at file offset 0x00000274 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: df924a2b08a7e89f6e11251d4602022977af2670
Displaying notes found at file offset 0x002d6c30 with length 0x00000144:
Owner Data size Description
stapsdt 0x00000031 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__start
Location: 0x00000000004371c3, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf6
Arguments: -4@%ebx
stapsdt 0x00000030 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__done
Location: 0x00000000004374e1, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf8
Arguments: -8@%rax
stapsdt 0x00000045 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: function__entry
Location: 0x000000000053db6c, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6be8
Arguments: 8@%rbp 8@%r12 -4@%eax
stapsdt 0x00000046 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: function__return
Location: 0x000000000053dba8, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bea
Arguments: 8@%rbp 8@%r12 -4@%eax
The above metadata contains information for SystemTap describing how it can patch strategically placed machine code instructions to enable the tracing hooks used by a SystemTap script.
Static DTrace probes¶
The following example DTrace script can be used to show the call/return hierarchy of a Python script, only tracing within the invocation of a function called “start”. In other words, import-time function invocations are not going to be listed:
self int indent;
python$target:::function-entry
/copyinstr(arg1) == "start"/
{
self->trace = 1;
}
python$target:::function-entry
/self->trace/
{
printf("%d\t%*s:", timestamp, 15, probename);
printf("%*s", self->indent, "");
printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
self->indent++;
}
python$target:::function-return
/self->trace/
{
self->indent--;
printf("%d\t%*s:", timestamp, 15, probename);
printf("%*s", self->indent, "");
printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
}
python$target:::function-return
/copyinstr(arg1) == "start"/
{
self->trace = 0;
}
It can be invoked like this:
$ sudo dtrace -q -s call_stack.d -c "python3.6 script.py"
The output looks like this:
156641360502280 function-entry:call_stack.py:start:23
156641360518804 function-entry: call_stack.py:function_1:1
156641360532797 function-entry: call_stack.py:function_3:9
156641360546807 function-return: call_stack.py:function_3:10
156641360563367 function-return: call_stack.py:function_1:2
156641360578365 function-entry: call_stack.py:function_2:5
156641360591757 function-entry: call_stack.py:function_1:1
156641360605556 function-entry: call_stack.py:function_3:9
156641360617482 function-return: call_stack.py:function_3:10
156641360629814 function-return: call_stack.py:function_1:2
156641360642285 function-return: call_stack.py:function_2:6
156641360656770 function-entry: call_stack.py:function_3:9
156641360669707 function-return: call_stack.py:function_3:10
156641360687853 function-entry: call_stack.py:function_4:13
156641360700719 function-return: call_stack.py:function_4:14
156641360719640 function-entry: call_stack.py:function_5:18
156641360732567 function-return: call_stack.py:function_5:21
156641360747370 function-return:call_stack.py:start:28
Static SystemTap markers¶
The low-level way to use the SystemTap integration is to use the static markers directly. This requires you to explicitly state the binary file containing them.
For example, this SystemTap script can be used to show the call/return hierarchy of a Python script:
probe process("python").mark("function__entry") {
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
printf("%s => %s in %s:%d\\n",
thread_indent(1), funcname, filename, lineno);
}
probe process("python").mark("function__return") {
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
printf("%s <= %s in %s:%d\\n",
thread_indent(-1), funcname, filename, lineno);
}
It can be invoked like this:
$ stap \
show-call-hierarchy.stp \
-c "./python test.py"
The output looks like this:
11408 python(8274): => __contains__ in Lib/_abcoll.py:362
11414 python(8274): => __getitem__ in Lib/os.py:425
11418 python(8274): => encode in Lib/os.py:490
11424 python(8274): <= encode in Lib/os.py:493
11428 python(8274): <= __getitem__ in Lib/os.py:426
11433 python(8274): <= __contains__ in Lib/_abcoll.py:366
where the columns are:
time in microseconds since start of script
name of executable
PID of process
and the remainder indicates the call/return hierarchy as the script executes.
For a --enable-shared
build of CPython, the markers are contained within the
libpython shared library, and the probe’s dotted path needs to reflect this. For
example, this line from the above example:
probe process("python").mark("function__entry") {
should instead read:
probe process("python").library("libpython3.6dm.so.1.0").mark("function__entry") {
(assuming a debug build of CPython 3.6)
Available static markers¶
- function__entry(str filename, str funcname, int lineno)
This marker indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions.
The filename, function name, and line number are provided back to the tracing script as positional arguments, which must be accessed using
$arg1
,$arg2
,$arg3
:$arg1
:(const char *)
filename, accessible usinguser_string($arg1)
$arg2
:(const char *)
function name, accessible usinguser_string($arg2)
$arg3
:int
line number
- function__return(str filename, str funcname, int lineno)
This marker is the converse of
function__entry()
, and indicates that execution of a Python function has ended (either viareturn
, or via an exception). It is only triggered for pure-Python (bytecode) functions.The arguments are the same as for
function__entry()
- line(str filename, str funcname, int lineno)
This marker indicates a Python line is about to be executed. It is the equivalent of line-by-line tracing with a Python profiler. It is not triggered within C functions.
The arguments are the same as for
function__entry()
.
- gc__start(int generation)
Fires when the Python interpreter starts a garbage collection cycle.
arg0
is the generation to scan, likegc.collect()
.
- gc__done(long collected)
Fires when the Python interpreter finishes a garbage collection cycle.
arg0
is the number of collected objects.
- import__find__load__start(str modulename)
Fires before
importlib
attempts to find and load the module.arg0
is the module name.Added in version 3.7.
- import__find__load__done(str modulename, int found)
Fires after
importlib
’s find_and_load function is called.arg0
is the module name,arg1
indicates if module was successfully loaded.Added in version 3.7.
- audit(str event, void *tuple)
Fires when
sys.audit()
orPySys_Audit()
is called.arg0
is the event name as C string,arg1
is aPyObject
pointer to a tuple object.Added in version 3.8.
SystemTap Tapsets¶
The higher-level way to use the SystemTap integration is to use a “tapset”: SystemTap’s equivalent of a library, which hides some of the lower-level details of the static markers.
Here is a tapset file, based on a non-shared build of CPython:
/*
Provide a higher-level wrapping around the function__entry and
function__return markers:
\*/
probe python.function.entry = process("python").mark("function__entry")
{
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
frameptr = $arg4
}
probe python.function.return = process("python").mark("function__return")
{
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
frameptr = $arg4
}
If this file is installed in SystemTap’s tapset directory (e.g.
/usr/share/systemtap/tapset
), then these additional probepoints become
available:
- python.function.entry(str filename, str funcname, int lineno, frameptr)
This probe point indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions.
- python.function.return(str filename, str funcname, int lineno, frameptr)
This probe point is the converse of
python.function.return
, and indicates that execution of a Python function has ended (either viareturn
, or via an exception). It is only triggered for pure-Python (bytecode) functions.
Examples¶
This SystemTap script uses the tapset above to more cleanly implement the example given above of tracing the Python function-call hierarchy, without needing to directly name the static markers:
probe python.function.entry
{
printf("%s => %s in %s:%d\n",
thread_indent(1), funcname, filename, lineno);
}
probe python.function.return
{
printf("%s <= %s in %s:%d\n",
thread_indent(-1), funcname, filename, lineno);
}
The following script uses the tapset above to provide a top-like view of all running CPython code, showing the top 20 most frequently entered bytecode frames, each second, across the whole system:
global fn_calls;
probe python.function.entry
{
fn_calls[pid(), filename, funcname, lineno] += 1;
}
probe timer.ms(1000) {
printf("\033[2J\033[1;1H") /* clear screen \*/
printf("%6s %80s %6s %30s %6s\n",
"PID", "FILENAME", "LINE", "FUNCTION", "CALLS")
foreach ([pid, filename, funcname, lineno] in fn_calls- limit 20) {
printf("%6d %80s %6d %30s %6d\n",
pid, filename, lineno, funcname,
fn_calls[pid, filename, funcname, lineno]);
}
delete fn_calls;
}


Debugging C API extensions and CPython Internals with GDB¶
This document explains how the Python GDB extension, python-gdb.py
, can
be used with the GDB debugger to debug CPython extensions and the
CPython interpreter itself.
When debugging low-level problems such as crashes or deadlocks, a low-level debugger, such as GDB, is useful to diagnose and correct the issue. By default, GDB (or any of its front-ends) doesn’t support high-level information specific to the CPython interpreter.
The python-gdb.py
extension adds CPython interpreter information to GDB.
The extension helps introspect the stack of currently executing Python functions.
Given a Python object represented by a PyObject* pointer,
the extension surfaces the type and value of the object.
Developers who are working on CPython extensions or tinkering with parts
of CPython that are written in C can use this document to learn how to use the
python-gdb.py
extension with GDB.
Note
This document assumes that you are familiar with the basics of GDB and the CPython C API. It consolidates guidance from the devguide and the Python wiki.
Prerequisites¶
You need to have:
GDB 7 or later. (For earlier versions of GDB, see
Misc/gdbinit
in the sources of Python 3.11 or earlier.)GDB-compatible debugging information for Python and any extension you are debugging.
The
python-gdb.py
extension.
The extension is built with Python, but might be distributed separately or not at all. Below, we include tips for a few common systems as examples. Note that even if the instructions match your system, they might be outdated.
Setup with Python built from source¶
When you build CPython from source, debugging information should be available,
and the build should add a python-gdb.py
file to the root directory of
your repository.
To activate support, you must add the directory containing python-gdb.py
to GDB’s “auto-load-safe-path”.
If you haven’t done this, recent versions of GDB will print out a warning
with instructions on how to do this.
Note
If you do not see instructions for your version of GDB, put this in your
configuration file (~/.gdbinit
or ~/.config/gdb/gdbinit
):
add-auto-load-safe-path /path/to/cpython
You can also add multiple paths, separated by :
.
Setup for Python from a Linux distro¶
Most Linux systems provide debug information for the system Python
in a package called python-debuginfo
, python-dbg
or similar.
For example:
Fedora:
sudo dnf install gdb sudo dnf debuginfo-install python3
Ubuntu:
sudo apt install gdb python3-dbg
On several recent Linux systems, GDB can download debugging symbols
automatically using debuginfod.
However, this will not install the python-gdb.py
extension;
you generally do need to install the debug info package separately.
Using the Debug build and Development mode¶
For easier debugging, you might want to:
Use a debug build of Python. (When building from source, use
configure --with-pydebug
. On Linux distros, install and run a package likepython-debug
orpython-dbg
, if available.)Use the runtime development mode (
-X dev
).
Both enable extra assertions and disable some optimizations. Sometimes this hides the bug you are trying to find, but in most cases they make the process easier.
Using the python-gdb
extension¶
When the extension is loaded, it provides two main features: pretty printers for Python values, and additional commands.
Pretty-printers¶
This is what a GDB backtrace looks like (truncated) when this extension is enabled:
#0 0x000000000041a6b1 in PyObject_Malloc (nbytes=Cannot access memory at address 0x7fffff7fefe8
) at Objects/obmalloc.c:748
#1 0x000000000041b7c0 in _PyObject_DebugMallocApi (id=111 'o', nbytes=24) at Objects/obmalloc.c:1445
#2 0x000000000041b717 in _PyObject_DebugMalloc (nbytes=24) at Objects/obmalloc.c:1412
#3 0x000000000044060a in _PyUnicode_New (length=11) at Objects/unicodeobject.c:346
#4 0x00000000004466aa in PyUnicodeUCS2_DecodeUTF8Stateful (s=0x5c2b8d "__lltrace__", size=11, errors=0x0, consumed=
0x0) at Objects/unicodeobject.c:2531
#5 0x0000000000446647 in PyUnicodeUCS2_DecodeUTF8 (s=0x5c2b8d "__lltrace__", size=11, errors=0x0)
at Objects/unicodeobject.c:2495
#6 0x0000000000440d1b in PyUnicodeUCS2_FromStringAndSize (u=0x5c2b8d "__lltrace__", size=11)
at Objects/unicodeobject.c:551
#7 0x0000000000440d94 in PyUnicodeUCS2_FromString (u=0x5c2b8d "__lltrace__") at Objects/unicodeobject.c:569
#8 0x0000000000584abd in PyDict_GetItemString (v=
{'Yuck': <type at remote 0xad4730>, '__builtins__': <module at remote 0x7ffff7fd5ee8>, '__file__': 'Lib/test/crashers/nasty_eq_vs_dict.py', '__package__': None, 'y': <Yuck(i=0) at remote 0xaacd80>, 'dict': {0: 0, 1: 1, 2: 2, 3: 3}, '__cached__': None, '__name__': '__main__', 'z': <Yuck(i=0) at remote 0xaace60>, '__doc__': None}, key=
0x5c2b8d "__lltrace__") at Objects/dictobject.c:2171
Notice how the dictionary argument to PyDict_GetItemString
is displayed
as its repr()
, rather than an opaque PyObject *
pointer.
The extension works by supplying a custom printing routine for values of type
PyObject *
. If you need to access lower-level details of an object, then
cast the value to a pointer of the appropriate type. For example:
(gdb) p globals
$1 = {'__builtins__': <module at remote 0x7ffff7fb1868>, '__name__':
'__main__', 'ctypes': <module at remote 0x7ffff7f14360>, '__doc__': None,
'__package__': None}
(gdb) p *(PyDictObject*)globals
$2 = {ob_refcnt = 3, ob_type = 0x3dbdf85820, ma_fill = 5, ma_used = 5,
ma_mask = 7, ma_table = 0x63d0f8, ma_lookup = 0x3dbdc7ea70
<lookdict_string>, ma_smalltable = {{me_hash = 7065186196740147912,
me_key = '__builtins__', me_value = <module at remote 0x7ffff7fb1868>},
{me_hash = -368181376027291943, me_key = '__name__',
me_value ='__main__'}, {me_hash = 0, me_key = 0x0, me_value = 0x0},
{me_hash = 0, me_key = 0x0, me_value = 0x0},
{me_hash = -9177857982131165996, me_key = 'ctypes',
me_value = <module at remote 0x7ffff7f14360>},
{me_hash = -8518757509529533123, me_key = '__doc__', me_value = None},
{me_hash = 0, me_key = 0x0, me_value = 0x0}, {
me_hash = 6614918939584953775, me_key = '__package__', me_value = None}}}
Note that the pretty-printers do not actually call repr()
.
For basic types, they try to match its result closely.
An area that can be confusing is that the custom printer for some types look a
lot like GDB’s built-in printer for standard types. For example, the
pretty-printer for a Python int
(PyLongObject*)
gives a representation that is not distinguishable from one of a
regular machine-level integer:
(gdb) p some_machine_integer
$3 = 42
(gdb) p some_python_integer
$4 = 42
The internal structure can be revealed with a cast to PyLongObject*:
(gdb) p *(PyLongObject*)some_python_integer
$5 = {ob_base = {ob_base = {ob_refcnt = 8, ob_type = 0x3dad39f5e0}, ob_size = 1},
ob_digit = {42}}
A similar confusion can arise with the str
type, where the output looks a
lot like gdb’s built-in printer for char *
:
(gdb) p ptr_to_python_str
$6 = '__builtins__'
The pretty-printer for str
instances defaults to using single-quotes (as
does Python’s repr
for strings) whereas the standard printer for char *
values uses double-quotes and contains a hexadecimal address:
(gdb) p ptr_to_char_star
$7 = 0x6d72c0 "hello world"
Again, the implementation details can be revealed with a cast to PyUnicodeObject*:
(gdb) p *(PyUnicodeObject*)$6
$8 = {ob_base = {ob_refcnt = 33, ob_type = 0x3dad3a95a0}, length = 12,
str = 0x7ffff2128500, hash = 7065186196740147912, state = 1, defenc = 0x0}
py-list
¶
The extension adds a
py-list
command, which lists the Python source code (if any) for the current frame in the selected thread. The current line is marked with a “>”:(gdb) py-list 901 if options.profile: 902 options.profile = False 903 profile_me() 904 return 905 >906 u = UI() 907 if not u.quit: 908 try: 909 gtk.main() 910 except KeyboardInterrupt: 911 # properly quit on a keyboard interrupt...Use
py-list START
to list at a different line number within the Python source, andpy-list START,END
to list a specific range of lines within the Python source.
py-up
and py-down
¶
The
py-up
andpy-down
commands are analogous to GDB’s regularup
anddown
commands, but try to move at the level of CPython frames, rather than C frames.GDB is not always able to read the relevant frame information, depending on the optimization level with which CPython was compiled. Internally, the commands look for C frames that are executing the default frame evaluation function (that is, the core bytecode interpreter loop within CPython) and look up the value of the related
PyFrameObject *
.They emit the frame number (at the C level) within the thread.
For example:
(gdb) py-up #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/ gnome_sudoku/main.py, line 906, in start_game () u = UI() (gdb) py-up #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/ gnome_sudoku/gnome_sudoku.py, line 22, in start_game(main=<module at remote 0xb771b7f4>) main.start_game() (gdb) py-up Unable to find an older python frameso we’re at the top of the Python stack.
The frame numbers correspond to those displayed by GDB’s standard
backtrace
command. The command skips C frames which are not executing Python code.Going back down:
(gdb) py-down #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game () u = UI() (gdb) py-down #34 (unable to read python frame information) (gdb) py-down #23 (unable to read python frame information) (gdb) py-down #19 (unable to read python frame information) (gdb) py-down #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': <float at remote 0x984b474>, 'gsd.hints': 0, 'timer.active_time': <float at remote 0x984b494>, 'timer.total_time': <float at remote 0x984b464>}], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated) swallower.run_dialog(self.dialog) (gdb) py-down #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>) gtk.main() (gdb) py-down #8 (unable to read python frame information) (gdb) py-down Unable to find a newer python frameand we’re at the bottom of the Python stack.
Note that in Python 3.12 and newer, the same C stack frame can be used for multiple Python stack frames. This means that
py-up
andpy-down
may move multiple Python frames at once. For example:(gdb) py-up #6 Frame 0x7ffff7fb62b0, for file /tmp/rec.py, line 5, in recursive_function (n=0) time.sleep(5) #6 Frame 0x7ffff7fb6240, for file /tmp/rec.py, line 7, in recursive_function (n=1) recursive_function(n-1) #6 Frame 0x7ffff7fb61d0, for file /tmp/rec.py, line 7, in recursive_function (n=2) recursive_function(n-1) #6 Frame 0x7ffff7fb6160, for file /tmp/rec.py, line 7, in recursive_function (n=3) recursive_function(n-1) #6 Frame 0x7ffff7fb60f0, for file /tmp/rec.py, line 7, in recursive_function (n=4) recursive_function(n-1) #6 Frame 0x7ffff7fb6080, for file /tmp/rec.py, line 7, in recursive_function (n=5) recursive_function(n-1) #6 Frame 0x7ffff7fb6020, for file /tmp/rec.py, line 9, in <module> () recursive_function(5) (gdb) py-up Unable to find an older python frame
py-bt
¶
The
py-bt
command attempts to display a Python-level backtrace of the current thread.For example:
(gdb) py-bt #8 (unable to read python frame information) #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>) gtk.main() #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': <float at remote 0x984b474>, 'gsd.hints': 0, 'timer.active_time': <float at remote 0x984b494>, 'timer.total_time': <float at remote 0x984b464>}], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated) swallower.run_dialog(self.dialog) #19 (unable to read python frame information) #23 (unable to read python frame information) #34 (unable to read python frame information) #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game () u = UI() #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/gnome_sudoku/gnome_sudoku.py, line 22, in start_game (main=<module at remote 0xb771b7f4>) main.start_game()The frame numbers correspond to those displayed by GDB’s standard
backtrace
command.
py-print
¶
The
py-print
command looks up a Python name and tries to print it. It looks in locals within the current thread, then globals, then finally builtins:(gdb) py-print self local 'self' = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4> (gdb) py-print __name__ global '__name__' = 'gnome_sudoku.dialog_swallower' (gdb) py-print len builtin 'len' = <built-in function len> (gdb) py-print scarlet_pimpernel 'scarlet_pimpernel' not foundIf the current C frame corresponds to multiple Python frames,
py-print
only considers the first one.
py-locals
¶
The
py-locals
command looks up all Python locals within the current Python frame in the selected thread, and prints their representations:(gdb) py-locals self = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4> d = <gtk.Dialog at remote 0x98faaa4>If the current C frame corresponds to multiple Python frames, locals from all of them will be shown:
(gdb) py-locals Locals for recursive_function n = 0 Locals for recursive_function n = 1 Locals for recursive_function n = 2 Locals for recursive_function n = 3 Locals for recursive_function n = 4 Locals for recursive_function n = 5 Locals for <module>
Use with GDB commands¶
The extension commands complement GDB’s built-in commands.
For example, you can use a frame numbers shown by py-bt
with the frame
command to go a specific frame within the selected thread, like this:
(gdb) py-bt
(output snipped)
#68 Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in <module> ()
main()
(gdb) frame 68
#68 0x00000000004cd1e6 in PyEval_EvalFrameEx (f=Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in <module> (), throwflag=0) at Python/ceval.c:2665
2665 x = call_function(&sp, oparg);
(gdb) py-list
1543 # Run the tests in a context manager that temporary changes the CWD to a
1544 # temporary and writable directory. If it's not possible to create or
1545 # change the CWD, the original CWD will be used. The original CWD is
1546 # available from test_support.SAVEDCWD.
1547 with test_support.temp_cwd(TESTCWD, quiet=True):
>1548 main()
The info threads
command will give you a list of the threads within the
process, and you can use the thread
command to select a different one:
(gdb) info threads
105 Thread 0x7fffefa18710 (LWP 10260) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
104 Thread 0x7fffdf5fe710 (LWP 10259) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
* 1 Thread 0x7ffff7fe2700 (LWP 10145) 0x00000038e46d73e3 in select () at ../sysdeps/unix/syscall-template.S:82
You can use thread apply all COMMAND
or (t a a COMMAND
for short) to run
a command on all threads. With py-bt
, this lets you see what every
thread is doing at the Python level:
(gdb) t a a py-bt
Thread 105 (Thread 0x7fffefa18710 (LWP 10260)):
#5 Frame 0x7fffd00019d0, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140737213728528), count=1, owner=140737213728528)
self.__block.acquire()
#8 Frame 0x7fffac001640, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858a90>, saved_state=(1, 140737213728528))
self._acquire_restore(saved_state)
#12 Frame 0x7fffb8001a10, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f ()
cond.wait()
#16 Frame 0x7fffb8001c40, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140737213728528)
f()
Thread 104 (Thread 0x7fffdf5fe710 (LWP 10259)):
#5 Frame 0x7fffe4001580, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140736940992272), count=1, owner=140736940992272)
self.__block.acquire()
#8 Frame 0x7fffc8002090, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858860>, saved_state=(1, 140736940992272))
self._acquire_restore(saved_state)
#12 Frame 0x7fffac001c90, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f ()
cond.wait()
#16 Frame 0x7fffac0011c0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140736940992272)
f()
Thread 1 (Thread 0x7ffff7fe2700 (LWP 10145)):
#5 Frame 0xcb5380, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 16, in _wait ()
time.sleep(0.01)
#8 Frame 0x7fffd00024a0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 378, in _check_notify (self=<ConditionTests(_testMethodName='test_notify', _resultForDoCleanups=<TestResult(_original_stdout=<cStringIO.StringO at remote 0xc191e0>, skipped=[], _mirrorOutput=False, testsRun=39, buffer=False, _original_stderr=<file at remote 0x7ffff7fc6340>, _stdout_buffer=<cStringIO.StringO at remote 0xc9c7f8>, _stderr_buffer=<cStringIO.StringO at remote 0xc9c790>, _moduleSetUpFailed=False, expectedFailures=[], errors=[], _previousTestClass=<type at remote 0x928310>, unexpectedSuccesses=[], failures=[], shouldStop=False, failfast=False) at remote 0xc185a0>, _threads=(0,), _cleanups=[], _type_equality_funcs={<type at remote 0x7eba00>: <instancemethod at remote 0xd750e0>, <type at remote 0x7e7820>: <instancemethod at remote 0xd75160>, <type at remote 0x7e30e0>: <instancemethod at remote 0xd75060>, <type at remote 0x7e7d20>: <instancemethod at remote 0xd751e0>, <type at remote 0x7f19e0...(truncated)
_wait()


Porting Extension Modules to Python 3

Porting Extension Modules to Python 3¶
We recommend the following resources for porting extension modules to Python 3:
The Migrating C extensions chapter from Supporting Python 3: An in-depth guide, a book on moving from Python 2 to Python 3 in general, guides the reader through porting an extension module.
The Porting guide from the py3c project provides opinionated suggestions with supporting code.
Recommended third party tools offer abstractions over the Python’s C API. Extensions generally need to be re-written to use one of them, but the library then handles differences between various Python versions and implementations.