Integer Objects

Integer Objects¶
All integers are implemented as “long” integer objects of arbitrary size.
On error, most PyLong_As*
APIs return (return type)-1
which cannot be
distinguished from a number. Use PyErr_Occurred()
to disambiguate.
-
type PyLongObject¶
- Part of the Limited API (as an opaque struct).
This subtype of
PyObject
represents a Python integer object.
-
PyTypeObject PyLong_Type¶
- Part of the Stable ABI.
This instance of
PyTypeObject
represents the Python integer type. This is the same object asint
in the Python layer.
-
int PyLong_Check(PyObject *p)¶
Return true if its argument is a
PyLongObject
or a subtype ofPyLongObject
. This function always succeeds.
-
int PyLong_CheckExact(PyObject *p)¶
Return true if its argument is a
PyLongObject
, but not a subtype ofPyLongObject
. This function always succeeds.
-
PyObject *PyLong_FromLong(long v)¶
- Return value: New reference. Part of the Stable ABI.
Return a new
PyLongObject
object from v, orNULL
on failure.CPython implementation detail: CPython keeps an array of integer objects for all integers between
-5
and256
. When you create an int in that range you actually just get back a reference to the existing object.
-
PyObject *PyLong_FromUnsignedLong(unsigned long v)¶
- Return value: New reference. Part of the Stable ABI.
Return a new
PyLongObject
object from a C unsigned long, orNULL
on failure.
-
PyObject *PyLong_FromSsize_t(Py_ssize_t v)¶
- Return value: New reference. Part of the Stable ABI.
Return a new
PyLongObject
object from a CPy_ssize_t
, orNULL
on failure.
-
PyObject *PyLong_FromSize_t(size_t v)¶
- Return value: New reference. Part of the Stable ABI.
Return a new
PyLongObject
object from a Csize_t
, orNULL
on failure.
-
PyObject *PyLong_FromLongLong(long long v)¶
- Return value: New reference. Part of the Stable ABI.
Return a new
PyLongObject
object from a C long long, orNULL
on failure.
-
PyObject *PyLong_FromInt32(int32_t value)¶
-
PyObject *PyLong_FromInt64(int64_t value)¶
- Part of the Stable ABI since version 3.14.
Return a new
PyLongObject
object from a signed C int32_t or int64_t, orNULL
with an exception set on failure.Added in version 3.14.
-
PyObject *PyLong_FromUnsignedLongLong(unsigned long long v)¶
- Return value: New reference. Part of the Stable ABI.
Return a new
PyLongObject
object from a C unsigned long long, orNULL
on failure.
-
PyObject *PyLong_FromUInt32(uint32_t value)¶
-
PyObject *PyLong_FromUInt64(uint64_t value)¶
- Part of the Stable ABI since version 3.14.
Return a new
PyLongObject
object from an unsigned C uint32_t or uint64_t, orNULL
with an exception set on failure.Added in version 3.14.
-
PyObject *PyLong_FromDouble(double v)¶
- Return value: New reference. Part of the Stable ABI.
Return a new
PyLongObject
object from the integer part of v, orNULL
on failure.
-
PyObject *PyLong_FromString(const char *str, char **pend, int base)¶
- Return value: New reference. Part of the Stable ABI.
Return a new
PyLongObject
based on the string value in str, which is interpreted according to the radix in base, orNULL
on failure. If pend is non-NULL
, *pend will point to the end of str on success or to the first character that could not be processed on error. If base is0
, str is interpreted using the Integer literals definition; in this case, leading zeros in a non-zero decimal number raises aValueError
. If base is not0
, it must be between2
and36
, inclusive. Leading and trailing whitespace and single underscores after a base specifier and between digits are ignored. If there are no digits or str is not NULL-terminated following the digits and trailing whitespace,ValueError
will be raised.See also
PyLong_AsNativeBytes()
andPyLong_FromNativeBytes()
functions can be used to convert aPyLongObject
to/from an array of bytes in base256
.
-
PyObject *PyLong_FromUnicodeObject(PyObject *u, int base)¶
- Return value: New reference.
Convert a sequence of Unicode digits in the string u to a Python integer value.
Added in version 3.3.
-
PyObject *PyLong_FromVoidPtr(void *p)¶
- Return value: New reference. Part of the Stable ABI.
Create a Python integer from the pointer p. The pointer value can be retrieved from the resulting value using
PyLong_AsVoidPtr()
.
-
PyObject *PyLong_FromNativeBytes(const void *buffer, size_t n_bytes, int flags)¶
- Part of the Stable ABI since version 3.14.
Create a Python integer from the value contained in the first n_bytes of buffer, interpreted as a two’s-complement signed number.
flags are as for
PyLong_AsNativeBytes()
. Passing-1
will select the native endian that CPython was compiled with and assume that the most-significant bit is a sign bit. PassingPy_ASNATIVEBYTES_UNSIGNED_BUFFER
will produce the same result as callingPyLong_FromUnsignedNativeBytes()
. Other flags are ignored.Added in version 3.13.
-
PyObject *PyLong_FromUnsignedNativeBytes(const void *buffer, size_t n_bytes, int flags)¶
- Part of the Stable ABI since version 3.14.
Create a Python integer from the value contained in the first n_bytes of buffer, interpreted as an unsigned number.
flags are as for
PyLong_AsNativeBytes()
. Passing-1
will select the native endian that CPython was compiled with and assume that the most-significant bit is not a sign bit. Flags other than endian are ignored.Added in version 3.13.
-
PyLong_FromPid(pid)¶
Macro for creating a Python integer from a process identifier.
This can be defined as an alias to
PyLong_FromLong()
orPyLong_FromLongLong()
, depending on the size of the system’s PID type.Added in version 3.2.
-
long PyLong_AsLong(PyObject *obj)¶
- Part of the Stable ABI.
Return a C long representation of obj. If obj is not an instance of
PyLongObject
, first call its__index__()
method (if present) to convert it to aPyLongObject
.Raise
OverflowError
if the value of obj is out of range for a long.Returns
-1
on error. UsePyErr_Occurred()
to disambiguate.Changed in version 3.8: Use
__index__()
if available.Changed in version 3.10: This function will no longer use
__int__()
.-
long PyLong_AS_LONG(PyObject *obj)¶
A soft deprecated alias. Exactly equivalent to the preferred
PyLong_AsLong
. In particular, it can fail withOverflowError
or another exception.Deprecated since version 3.14: The function is soft deprecated.
-
long PyLong_AS_LONG(PyObject *obj)¶
-
int PyLong_AsInt(PyObject *obj)¶
- Part of the Stable ABI since version 3.13.
Similar to
PyLong_AsLong()
, but store the result in a C int instead of a C long.Added in version 3.13.
-
long PyLong_AsLongAndOverflow(PyObject *obj, int *overflow)¶
- Part of the Stable ABI.
Return a C long representation of obj. If obj is not an instance of
PyLongObject
, first call its__index__()
method (if present) to convert it to aPyLongObject
.If the value of obj is greater than
LONG_MAX
or less thanLONG_MIN
, set *overflow to1
or-1
, respectively, and return-1
; otherwise, set *overflow to0
. If any other exception occurs set *overflow to0
and return-1
as usual.Returns
-1
on error. UsePyErr_Occurred()
to disambiguate.Changed in version 3.8: Use
__index__()
if available.Changed in version 3.10: This function will no longer use
__int__()
.
-
long long PyLong_AsLongLong(PyObject *obj)¶
- Part of the Stable ABI.
Return a C long long representation of obj. If obj is not an instance of
PyLongObject
, first call its__index__()
method (if present) to convert it to aPyLongObject
.Raise
OverflowError
if the value of obj is out of range for a long long.Returns
-1
on error. UsePyErr_Occurred()
to disambiguate.Changed in version 3.8: Use
__index__()
if available.Changed in version 3.10: This function will no longer use
__int__()
.
-
long long PyLong_AsLongLongAndOverflow(PyObject *obj, int *overflow)¶
- Part of the Stable ABI.
Return a C long long representation of obj. If obj is not an instance of
PyLongObject
, first call its__index__()
method (if present) to convert it to aPyLongObject
.If the value of obj is greater than
LLONG_MAX
or less thanLLONG_MIN
, set *overflow to1
or-1
, respectively, and return-1
; otherwise, set *overflow to0
. If any other exception occurs set *overflow to0
and return-1
as usual.Returns
-1
on error. UsePyErr_Occurred()
to disambiguate.Added in version 3.2.
Changed in version 3.8: Use
__index__()
if available.Changed in version 3.10: This function will no longer use
__int__()
.
-
Py_ssize_t PyLong_AsSsize_t(PyObject *pylong)¶
- Part of the Stable ABI.
Return a C
Py_ssize_t
representation of pylong. pylong must be an instance ofPyLongObject
.Raise
OverflowError
if the value of pylong is out of range for aPy_ssize_t
.Returns
-1
on error. UsePyErr_Occurred()
to disambiguate.
-
unsigned long PyLong_AsUnsignedLong(PyObject *pylong)¶
- Part of the Stable ABI.
Return a C unsigned long representation of pylong. pylong must be an instance of
PyLongObject
.Raise
OverflowError
if the value of pylong is out of range for a unsigned long.Returns
(unsigned long)-1
on error. UsePyErr_Occurred()
to disambiguate.
-
size_t PyLong_AsSize_t(PyObject *pylong)¶
- Part of the Stable ABI.
Return a C
size_t
representation of pylong. pylong must be an instance ofPyLongObject
.Raise
OverflowError
if the value of pylong is out of range for asize_t
.Returns
(size_t)-1
on error. UsePyErr_Occurred()
to disambiguate.
-
unsigned long long PyLong_AsUnsignedLongLong(PyObject *pylong)¶
- Part of the Stable ABI.
Return a C unsigned long long representation of pylong. pylong must be an instance of
PyLongObject
.Raise
OverflowError
if the value of pylong is out of range for an unsigned long long.Returns
(unsigned long long)-1
on error. UsePyErr_Occurred()
to disambiguate.Changed in version 3.1: A negative pylong now raises
OverflowError
, notTypeError
.
-
unsigned long PyLong_AsUnsignedLongMask(PyObject *obj)¶
- Part of the Stable ABI.
Return a C unsigned long representation of obj. If obj is not an instance of
PyLongObject
, first call its__index__()
method (if present) to convert it to aPyLongObject
.If the value of obj is out of range for an unsigned long, return the reduction of that value modulo
ULONG_MAX + 1
.Returns
(unsigned long)-1
on error. UsePyErr_Occurred()
to disambiguate.Changed in version 3.8: Use
__index__()
if available.Changed in version 3.10: This function will no longer use
__int__()
.
-
unsigned long long PyLong_AsUnsignedLongLongMask(PyObject *obj)¶
- Part of the Stable ABI.
Return a C unsigned long long representation of obj. If obj is not an instance of
PyLongObject
, first call its__index__()
method (if present) to convert it to aPyLongObject
.If the value of obj is out of range for an unsigned long long, return the reduction of that value modulo
ULLONG_MAX + 1
.Returns
(unsigned long long)-1
on error. UsePyErr_Occurred()
to disambiguate.Changed in version 3.8: Use
__index__()
if available.Changed in version 3.10: This function will no longer use
__int__()
.
-
int PyLong_AsInt32(PyObject *obj, int32_t *value)¶
-
int PyLong_AsInt64(PyObject *obj, int64_t *value)¶
- Part of the Stable ABI since version 3.14.
Set *value to a signed C int32_t or int64_t representation of obj.
If obj is not an instance of
PyLongObject
, first call its__index__()
method (if present) to convert it to aPyLongObject
.If the obj value is out of range, raise an
OverflowError
.Set *value and return
0
on success. Set an exception and return-1
on error.value must not be
NULL
.Added in version 3.14.
-
int PyLong_AsUInt32(PyObject *obj, uint32_t *value)¶
-
int PyLong_AsUInt64(PyObject *obj, uint64_t *value)¶
- Part of the Stable ABI since version 3.14.
Set *value to an unsigned C uint32_t or uint64_t representation of obj.
If obj is not an instance of
PyLongObject
, first call its__index__()
method (if present) to convert it to aPyLongObject
.If obj is negative, raise a
ValueError
.If the obj value is out of range, raise an
OverflowError
.
Set *value and return
0
on success. Set an exception and return-1
on error.value must not be
NULL
.Added in version 3.14.
-
double PyLong_AsDouble(PyObject *pylong)¶
- Part of the Stable ABI.
Return a C double representation of pylong. pylong must be an instance of
PyLongObject
.Raise
OverflowError
if the value of pylong is out of range for a double.Returns
-1.0
on error. UsePyErr_Occurred()
to disambiguate.
-
void *PyLong_AsVoidPtr(PyObject *pylong)¶
- Part of the Stable ABI.
Convert a Python integer pylong to a C void pointer. If pylong cannot be converted, an
OverflowError
will be raised. This is only assured to produce a usable void pointer for values created withPyLong_FromVoidPtr()
.Returns
NULL
on error. UsePyErr_Occurred()
to disambiguate.
-
Py_ssize_t PyLong_AsNativeBytes(PyObject *pylong, void *buffer, Py_ssize_t n_bytes, int flags)¶
- Part of the Stable ABI since version 3.14.
Copy the Python integer value pylong to a native buffer of size n_bytes. The flags can be set to
-1
to behave similarly to a C cast, or to values documented below to control the behavior.Returns
-1
with an exception raised on error. This may happen if pylong cannot be interpreted as an integer, or if pylong was negative and thePy_ASNATIVEBYTES_REJECT_NEGATIVE
flag was set.Otherwise, returns the number of bytes required to store the value. If this is equal to or less than n_bytes, the entire value was copied. All n_bytes of the buffer are written: remaining bytes filled by copies of the sign bit.
If the returned value is greater than n_bytes, the value was truncated: as many of the lowest bits of the value as could fit are written, and the higher bits are ignored. This matches the typical behavior of a C-style downcast.
Note
Overflow is not considered an error. If the returned value is larger than n_bytes, most significant bits were discarded.
0
will never be returned.Values are always copied as two’s-complement.
Usage example:
int32_t value; Py_ssize_t bytes = PyLong_AsNativeBytes(pylong, &value, sizeof(value), -1); if (bytes < 0) { // Failed. A Python exception was set with the reason. return NULL; } else if (bytes <= (Py_ssize_t)sizeof(value)) { // Success! } else { // Overflow occurred, but 'value' contains the truncated // lowest bits of pylong. }
Passing zero to n_bytes will return the size of a buffer that would be large enough to hold the value. This may be larger than technically necessary, but not unreasonably so. If n_bytes=0, buffer may be
NULL
.Note
Passing n_bytes=0 to this function is not an accurate way to determine the bit length of the value.
To get at the entire Python value of an unknown size, the function can be called twice: first to determine the buffer size, then to fill it:
// Ask how much space we need. Py_ssize_t expected = PyLong_AsNativeBytes(pylong, NULL, 0, -1); if (expected < 0) { // Failed. A Python exception was set with the reason. return NULL; } assert(expected != 0); // Impossible per the API definition. uint8_t *bignum = malloc(expected); if (!bignum) { PyErr_SetString(PyExc_MemoryError, "bignum malloc failed."); return NULL; } // Safely get the entire value. Py_ssize_t bytes = PyLong_AsNativeBytes(pylong, bignum, expected, -1); if (bytes < 0) { // Exception has been set. free(bignum); return NULL; } else if (bytes > expected) { // This should not be possible. PyErr_SetString(PyExc_RuntimeError, "Unexpected bignum truncation after a size check."); free(bignum); return NULL; } // The expected success given the above pre-check. // ... use bignum ... free(bignum);
flags is either
-1
(Py_ASNATIVEBYTES_DEFAULTS
) to select defaults that behave most like a C cast, or a combination of the other flags in the table below. Note that-1
cannot be combined with other flags.Currently,
-1
corresponds toPy_ASNATIVEBYTES_NATIVE_ENDIAN | Py_ASNATIVEBYTES_UNSIGNED_BUFFER
.Flag
Value
-
Py_ASNATIVEBYTES_DEFAULTS¶
- Part of the Stable ABI since version 3.14.
-1
-
Py_ASNATIVEBYTES_BIG_ENDIAN¶
- Part of the Stable ABI since version 3.14.
0
-
Py_ASNATIVEBYTES_LITTLE_ENDIAN¶
- Part of the Stable ABI since version 3.14.
1
-
Py_ASNATIVEBYTES_NATIVE_ENDIAN¶
- Part of the Stable ABI since version 3.14.
3
-
Py_ASNATIVEBYTES_UNSIGNED_BUFFER¶
- Part of the Stable ABI since version 3.14.
4
-
Py_ASNATIVEBYTES_REJECT_NEGATIVE¶
- Part of the Stable ABI since version 3.14.
8
-
Py_ASNATIVEBYTES_ALLOW_INDEX¶
- Part of the Stable ABI since version 3.14.
16
Specifying
Py_ASNATIVEBYTES_NATIVE_ENDIAN
will override any other endian flags. Passing2
is reserved.By default, sufficient buffer will be requested to include a sign bit. For example, when converting 128 with n_bytes=1, the function will return 2 (or more) in order to store a zero sign bit.
If
Py_ASNATIVEBYTES_UNSIGNED_BUFFER
is specified, a zero sign bit will be omitted from size calculations. This allows, for example, 128 to fit in a single-byte buffer. If the destination buffer is later treated as signed, a positive input value may become negative. Note that the flag does not affect handling of negative values: for those, space for a sign bit is always requested.Specifying
Py_ASNATIVEBYTES_REJECT_NEGATIVE
causes an exception to be set if pylong is negative. Without this flag, negative values will be copied provided there is enough space for at least one sign bit, regardless of whetherPy_ASNATIVEBYTES_UNSIGNED_BUFFER
was specified.If
Py_ASNATIVEBYTES_ALLOW_INDEX
is specified and a non-integer value is passed, its__index__()
method will be called first. This may result in Python code executing and other threads being allowed to run, which could cause changes to other objects or values in use. When flags is-1
, this option is not set, and non-integer values will raiseTypeError
.Note
With the default flags (
-1
, or UNSIGNED_BUFFER without REJECT_NEGATIVE), multiple Python integers can map to a single value without overflow. For example, both255
and-1
fit a single-byte buffer and set all its bits. This matches typical C cast behavior.Added in version 3.13.
-
Py_ASNATIVEBYTES_DEFAULTS¶
-
PyLong_AsPid(pid)¶
Macro for converting a Python integer into a process identifier.
This can be defined as an alias to
PyLong_AsLong()
,PyLong_FromLongLong()
, orPyLong_AsInt()
, depending on the size of the system’s PID type.Added in version 3.2.
-
int PyLong_GetSign(PyObject *obj, int *sign)¶
Get the sign of the integer object obj.
On success, set *sign to the integer sign (0, -1 or +1 for zero, negative or positive integer, respectively) and return 0.
On failure, return -1 with an exception set. This function always succeeds if obj is a
PyLongObject
or its subtype.Added in version 3.14.
-
int PyLong_IsPositive(PyObject *obj)¶
Check if the integer object obj is positive (
obj > 0
).If obj is an instance of
PyLongObject
or its subtype, return1
when it’s positive and0
otherwise. Else set an exception and return-1
.Added in version 3.14.
-
int PyLong_IsNegative(PyObject *obj)¶
Check if the integer object obj is negative (
obj < 0
).If obj is an instance of
PyLongObject
or its subtype, return1
when it’s negative and0
otherwise. Else set an exception and return-1
.Added in version 3.14.
-
int PyLong_IsZero(PyObject *obj)¶
Check if the integer object obj is zero.
If obj is an instance of
PyLongObject
or its subtype, return1
when it’s zero and0
otherwise. Else set an exception and return-1
.Added in version 3.14.
-
PyObject *PyLong_GetInfo(void)¶
- Part of the Stable ABI.
On success, return a read only named tuple, that holds information about Python’s internal representation of integers. See
sys.int_info
for description of individual fields.On failure, return
NULL
with an exception set.Added in version 3.1.
-
int PyUnstable_Long_IsCompact(const PyLongObject *op)¶
- This is Unstable API. It may change without warning in minor releases.
Return 1 if op is compact, 0 otherwise.
This function makes it possible for performance-critical code to implement a “fast path” for small integers. For compact values use
PyUnstable_Long_CompactValue()
; for others fall back to aPyLong_As*
function orPyLong_AsNativeBytes()
.The speedup is expected to be negligible for most users.
Exactly what values are considered compact is an implementation detail and is subject to change.
Added in version 3.12.
-
Py_ssize_t PyUnstable_Long_CompactValue(const PyLongObject *op)¶
- This is Unstable API. It may change without warning in minor releases.
If op is compact, as determined by
PyUnstable_Long_IsCompact()
, return its value.Otherwise, the return value is undefined.
Added in version 3.12.
Export API¶
Added in version 3.14.
-
struct PyLongLayout¶
Layout of an array of “digits” (“limbs” in the GMP terminology), used to represent absolute value for arbitrary precision integers.
Use
PyLong_GetNativeLayout()
to get the native layout of Pythonint
objects, used internally for integers with “big enough” absolute value.See also
sys.int_info
which exposes similar information in Python.-
uint8_t bits_per_digit¶
Bits per digit. For example, a 15 bit digit means that bits 0-14 contain meaningful information.
-
uint8_t digit_size¶
Digit size in bytes. For example, a 15 bit digit will require at least 2 bytes.
-
int8_t digits_order¶
Digits order:
1
for most significant digit first-1
for least significant digit first
-
int8_t digit_endianness¶
Digit endianness:
1
for most significant byte first (big endian)-1
for least significant byte first (little endian)
-
uint8_t bits_per_digit¶
-
const PyLongLayout *PyLong_GetNativeLayout(void)¶
Get the native layout of Python
int
objects.See the
PyLongLayout
structure.The function must not be called before Python initialization nor after Python finalization. The returned layout is valid until Python is finalized. The layout is the same for all Python sub-interpreters in a process, and so it can be cached.
-
struct PyLongExport¶
Export of a Python
int
object.There are two cases:
-
Py_ssize_t ndigits¶
Number of digits in
digits
array. Only valid ifdigits
is notNULL
.
-
const void *digits¶
Read-only array of unsigned digits. Can be
NULL
.
-
Py_ssize_t ndigits¶
-
int PyLong_Export(PyObject *obj, PyLongExport *export_long)¶
Export a Python
int
object.export_long must point to a
PyLongExport
structure allocated by the caller. It must not beNULL
.On success, fill in *export_long and return
0
. On error, set an exception and return-1
.PyLong_FreeExport()
must be called when the export is no longer needed.CPython implementation detail: This function always succeeds if obj is a Python
int
object or a subclass.
-
void PyLong_FreeExport(PyLongExport *export_long)¶
Release the export export_long created by
PyLong_Export()
.CPython implementation detail: Calling
PyLong_FreeExport()
is optional if export_long->digits isNULL
.
PyLongWriter API¶
The PyLongWriter
API can be used to import an integer.
Added in version 3.14.
-
struct PyLongWriter¶
A Python
int
writer instance.The instance must be destroyed by
PyLongWriter_Finish()
orPyLongWriter_Discard()
.
-
PyLongWriter *PyLongWriter_Create(int negative, Py_ssize_t ndigits, void **digits)¶
Create a
PyLongWriter
.On success, allocate *digits and return a writer. On error, set an exception and return
NULL
.negative is
1
if the number is negative, or0
otherwise.ndigits is the number of digits in the digits array. It must be greater than 0.
digits must not be NULL.
After a successful call to this function, the caller should fill in the array of digits digits and then call
PyLongWriter_Finish()
to get a Pythonint
. The layout of digits is described byPyLong_GetNativeLayout()
.Digits must be in the range [
0
;(1 << bits_per_digit) - 1
] (where thebits_per_digit
is the number of bits per digit). Any unused most significant digits must be set to0
.Alternately, call
PyLongWriter_Discard()
to destroy the writer instance without creating anint
object.
-
PyObject *PyLongWriter_Finish(PyLongWriter *writer)¶
- Return value: New reference.
Finish a
PyLongWriter
created byPyLongWriter_Create()
.On success, return a Python
int
object. On error, set an exception and returnNULL
.The function takes care of normalizing the digits and converts the object to a compact integer if needed.
The writer instance and the digits array are invalid after the call.
-
void PyLongWriter_Discard(PyLongWriter *writer)¶
Discard a
PyLongWriter
created byPyLongWriter_Create()
.If writer is
NULL
, no operation is performed.The writer instance and the digits array are invalid after the call.
Deprecated API¶
These macros are soft deprecated. They describe parameters
of the internal representation of PyLongObject
instances.
Use PyLong_GetNativeLayout()
instead, along with PyLong_Export()
to read integer data or PyLongWriter
to write it.
These currently use the same layout, but are designed to continue working correctly
even if CPython’s internal integer representation changes.
-
PyLong_SHIFT¶
This is equivalent to
bits_per_digit
in the output ofPyLong_GetNativeLayout()
.
-
PyLong_BASE¶
This is currently equivalent to 1 << PyLong_SHIFT.
-
PyLong_MASK¶
This is currently equivalent to (1 << PyLong_SHIFT) - 1


Type Objects

Type Objects¶
-
type PyTypeObject¶
- Part of the Limited API (as an opaque struct).
The C structure of the objects used to describe built-in types.
-
PyTypeObject PyType_Type¶
- Part of the Stable ABI.
This is the type object for type objects; it is the same object as
type
in the Python layer.
-
int PyType_Check(PyObject *o)¶
Return non-zero if the object o is a type object, including instances of types derived from the standard type object. Return 0 in all other cases. This function always succeeds.
-
int PyType_CheckExact(PyObject *o)¶
Return non-zero if the object o is a type object, but not a subtype of the standard type object. Return 0 in all other cases. This function always succeeds.
-
unsigned int PyType_ClearCache()¶
- Part of the Stable ABI.
Clear the internal lookup cache. Return the current version tag.
-
unsigned long PyType_GetFlags(PyTypeObject *type)¶
- Part of the Stable ABI.
Return the
tp_flags
member of type. This function is primarily meant for use withPy_LIMITED_API
; the individual flag bits are guaranteed to be stable across Python releases, but access totp_flags
itself is not part of the limited API.Added in version 3.2.
Changed in version 3.4: The return type is now
unsigned long
rather thanlong
.
-
PyObject *PyType_GetDict(PyTypeObject *type)¶
Return the type object’s internal namespace, which is otherwise only exposed via a read-only proxy (
cls.__dict__
). This is a replacement for accessingtp_dict
directly. The returned dictionary must be treated as read-only.This function is meant for specific embedding and language-binding cases, where direct access to the dict is necessary and indirect access (e.g. via the proxy or
PyObject_GetAttr()
) isn’t adequate.Extension modules should continue to use
tp_dict
, directly or indirectly, when setting up their own types.Added in version 3.12.
-
void PyType_Modified(PyTypeObject *type)¶
- Part of the Stable ABI.
Invalidate the internal lookup cache for the type and all of its subtypes. This function must be called after any manual modification of the attributes or base classes of the type.
-
int PyType_AddWatcher(PyType_WatchCallback callback)¶
Register callback as a type watcher. Return a non-negative integer ID which must be passed to future calls to
PyType_Watch()
. In case of error (e.g. no more watcher IDs available), return-1
and set an exception.In free-threaded builds,
PyType_AddWatcher()
is not thread-safe, so it must be called at start up (before spawning the first thread).Added in version 3.12.
-
int PyType_ClearWatcher(int watcher_id)¶
Clear watcher identified by watcher_id (previously returned from
PyType_AddWatcher()
). Return0
on success,-1
on error (e.g. if watcher_id was never registered.)An extension should never call
PyType_ClearWatcher
with a watcher_id that was not returned to it by a previous call toPyType_AddWatcher()
.Added in version 3.12.
-
int PyType_Watch(int watcher_id, PyObject *type)¶
Mark type as watched. The callback granted watcher_id by
PyType_AddWatcher()
will be called wheneverPyType_Modified()
reports a change to type. (The callback may be called only once for a series of consecutive modifications to type, if_PyType_Lookup()
is not called on type between the modifications; this is an implementation detail and subject to change.)An extension should never call
PyType_Watch
with a watcher_id that was not returned to it by a previous call toPyType_AddWatcher()
.Added in version 3.12.
-
int PyType_Unwatch(int watcher_id, PyObject *type)¶
Mark type as not watched. This undoes a previous call to
PyType_Watch()
. type must not beNULL
.An extension should never call this function with a watcher_id that was not returned to it by a previous call to
PyType_AddWatcher()
.On success, this function returns
0
. On failure, this function returns-1
with an exception set.Added in version 3.12.
-
typedef int (*PyType_WatchCallback)(PyObject *type)¶
Type of a type-watcher callback function.
The callback must not modify type or cause
PyType_Modified()
to be called on type or any type in its MRO; violating this rule could cause infinite recursion.Added in version 3.12.
-
int PyType_HasFeature(PyTypeObject *o, int feature)¶
Return non-zero if the type object o sets the feature feature. Type features are denoted by single bit flags.
-
int PyType_FastSubclass(PyTypeObject *type, int flag)¶
Return non-zero if the type object type sets the subclass flag flag. Subclass flags are denoted by
Py_TPFLAGS_*_SUBCLASS
. This function is used by many_Check
functions for common types.See also
PyObject_TypeCheck()
, which is used as a slower alternative in_Check
functions for types that don’t come with subclass flags.
-
int PyType_IS_GC(PyTypeObject *o)¶
Return true if the type object includes support for the cycle detector; this tests the type flag
Py_TPFLAGS_HAVE_GC
.
-
int PyType_IsSubtype(PyTypeObject *a, PyTypeObject *b)¶
- Part of the Stable ABI.
Return true if a is a subtype of b.
This function only checks for actual subtypes, which means that
__subclasscheck__()
is not called on b. CallPyObject_IsSubclass()
to do the same check thatissubclass()
would do.
-
PyObject *PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)¶
- Return value: New reference. Part of the Stable ABI.
Generic handler for the
tp_alloc
slot of a type object. Uses Python’s default memory allocation mechanism to allocate memory for a new instance, zeros the memory, then initializes the memory as if by callingPyObject_Init()
orPyObject_InitVar()
.Do not call this directly to allocate memory for an object; call the type’s
tp_alloc
slot instead.For types that support garbage collection (i.e., the
Py_TPFLAGS_HAVE_GC
flag is set), this function behaves likePyObject_GC_New
orPyObject_GC_NewVar
(except the memory is guaranteed to be zeroed before initialization), and should be paired withPyObject_GC_Del()
intp_free
. Otherwise, it behaves likePyObject_New
orPyObject_NewVar
(except the memory is guaranteed to be zeroed before initialization) and should be paired withPyObject_Free()
intp_free
.
-
PyObject *PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds)¶
- Return value: New reference. Part of the Stable ABI.
Generic handler for the
tp_new
slot of a type object. Creates a new instance using the type’stp_alloc
slot and returns the resulting object.
-
int PyType_Ready(PyTypeObject *type)¶
- Part of the Stable ABI.
Finalize a type object. This should be called on all type objects to finish their initialization. This function is responsible for adding inherited slots from a type’s base class. Return
0
on success, or return-1
and sets an exception on error.Note
If some of the base classes implements the GC protocol and the provided type does not include the
Py_TPFLAGS_HAVE_GC
in its flags, then the GC protocol will be automatically implemented from its parents. On the contrary, if the type being created does includePy_TPFLAGS_HAVE_GC
in its flags then it must implement the GC protocol itself by at least implementing thetp_traverse
handle.
-
PyObject *PyType_GetName(PyTypeObject *type)¶
- Return value: New reference. Part of the Stable ABI since version 3.11.
Return the type’s name. Equivalent to getting the type’s
__name__
attribute.Added in version 3.11.
-
PyObject *PyType_GetQualName(PyTypeObject *type)¶
- Return value: New reference. Part of the Stable ABI since version 3.11.
Return the type’s qualified name. Equivalent to getting the type’s
__qualname__
attribute.Added in version 3.11.
-
PyObject *PyType_GetFullyQualifiedName(PyTypeObject *type)¶
- Part of the Stable ABI since version 3.13.
Return the type’s fully qualified name. Equivalent to
f"{type.__module__}.{type.__qualname__}"
, ortype.__qualname__
iftype.__module__
is not a string or is equal to"builtins"
.Added in version 3.13.
-
PyObject *PyType_GetModuleName(PyTypeObject *type)¶
- Part of the Stable ABI since version 3.13.
Return the type’s module name. Equivalent to getting the
type.__module__
attribute.Added in version 3.13.
-
void *PyType_GetSlot(PyTypeObject *type, int slot)¶
- Part of the Stable ABI since version 3.4.
Return the function pointer stored in the given slot. If the result is
NULL
, this indicates that either the slot isNULL
, or that the function was called with invalid parameters. Callers will typically cast the result pointer into the appropriate function type.See
PyType_Slot.slot
for possible values of the slot argument.Added in version 3.4.
Changed in version 3.10:
PyType_GetSlot()
can now accept all types. Previously, it was limited to heap types.
-
PyObject *PyType_GetModule(PyTypeObject *type)¶
- Part of the Stable ABI since version 3.10.
Return the module object associated with the given type when the type was created using
PyType_FromModuleAndSpec()
.If no module is associated with the given type, sets
TypeError
and returnsNULL
.This function is usually used to get the module in which a method is defined. Note that in such a method,
PyType_GetModule(Py_TYPE(self))
may not return the intended result.Py_TYPE(self)
may be a subclass of the intended class, and subclasses are not necessarily defined in the same module as their superclass. SeePyCMethod
to get the class that defines the method. SeePyType_GetModuleByDef()
for cases whenPyCMethod
cannot be used.Added in version 3.9.
-
void *PyType_GetModuleState(PyTypeObject *type)¶
- Part of the Stable ABI since version 3.10.
Return the state of the module object associated with the given type. This is a shortcut for calling
PyModule_GetState()
on the result ofPyType_GetModule()
.If no module is associated with the given type, sets
TypeError
and returnsNULL
.If the type has an associated module but its state is
NULL
, returnsNULL
without setting an exception.Added in version 3.9.
-
PyObject *PyType_GetModuleByDef(PyTypeObject *type, struct PyModuleDef *def)¶
- Return value: Borrowed reference. Part of the Stable ABI since version 3.13.
Find the first superclass whose module was created from the given
PyModuleDef
def, and return that module.If no module is found, raises a
TypeError
and returnsNULL
.This function is intended to be used together with
PyModule_GetState()
to get module state from slot methods (such astp_init
ornb_add
) and other places where a method’s defining class cannot be passed using thePyCMethod
calling convention.The returned reference is borrowed from type, and will be valid as long as you hold a reference to type. Do not release it with
Py_DECREF()
or similar.Added in version 3.11.
-
int PyType_GetBaseByToken(PyTypeObject *type, void *token, PyTypeObject **result)¶
- Part of the Stable ABI since version 3.14.
Find the first superclass in type’s method resolution order whose
Py_tp_token
token is equal to the given one.If found, set *result to a new strong reference to it and return
1
.If not found, set *result to
NULL
and return0
.On error, set *result to
NULL
and return-1
with an exception set.
The result argument may be
NULL
, in which case *result is not set. Use this if you need only the return value.The token argument may not be
NULL
.Added in version 3.14.
-
int PyUnstable_Type_AssignVersionTag(PyTypeObject *type)¶
- This is Unstable API. It may change without warning in minor releases.
Attempt to assign a version tag to the given type.
Returns 1 if the type already had a valid version tag or a new one was assigned, or 0 if a new tag could not be assigned.
Added in version 3.12.
-
int PyType_SUPPORTS_WEAKREFS(PyTypeObject *type)¶
Return true if instances of type support creating weak references, false otherwise. This function always succeeds. type must not be
NULL
.See also
Creating Heap-Allocated Types¶
The following functions and structs are used to create heap types.
-
PyObject *PyType_FromMetaclass(PyTypeObject *metaclass, PyObject *module, PyType_Spec *spec, PyObject *bases)¶
- Part of the Stable ABI since version 3.12.
Create and return a heap type from the spec (see
Py_TPFLAGS_HEAPTYPE
).The metaclass metaclass is used to construct the resulting type object. When metaclass is
NULL
, the metaclass is derived from bases (or Py_tp_base[s] slots if bases isNULL
, see below).Metaclasses that override
tp_new
are not supported, except iftp_new
isNULL
.The bases argument can be used to specify base classes; it can either be only one class or a tuple of classes. If bases is
NULL
, thePy_tp_bases
slot is used instead. If that also isNULL
, thePy_tp_base
slot is used instead. If that also isNULL
, the new type derives fromobject
.The module argument can be used to record the module in which the new class is defined. It must be a module object or
NULL
. If notNULL
, the module is associated with the new type and can later be retrieved withPyType_GetModule()
. The associated module is not inherited by subclasses; it must be specified for each class individually.This function calls
PyType_Ready()
on the new type.Note that this function does not fully match the behavior of calling
type()
or using theclass
statement. With user-provided base types or metaclasses, prefer callingtype
(or the metaclass) overPyType_From*
functions. Specifically:__new__()
is not called on the new class (and it must be set totype.__new__
).__init__()
is not called on the new class.__init_subclass__()
is not called on any bases.__set_name__()
is not called on new descriptors.
Added in version 3.12.
-
PyObject *PyType_FromModuleAndSpec(PyObject *module, PyType_Spec *spec, PyObject *bases)¶
- Return value: New reference. Part of the Stable ABI since version 3.10.
Equivalent to
PyType_FromMetaclass(NULL, module, spec, bases)
.Added in version 3.9.
Changed in version 3.10: The function now accepts a single class as the bases argument and
NULL
as thetp_doc
slot.Changed in version 3.12: The function now finds and uses a metaclass corresponding to the provided base classes. Previously, only
type
instances were returned.The
tp_new
of the metaclass is ignored. which may result in incomplete initialization. Creating classes whose metaclass overridestp_new
is deprecated.Changed in version 3.14: Creating classes whose metaclass overrides
tp_new
is no longer allowed.
-
PyObject *PyType_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)¶
- Return value: New reference. Part of the Stable ABI since version 3.3.
Equivalent to
PyType_FromMetaclass(NULL, NULL, spec, bases)
.Added in version 3.3.
Changed in version 3.12: The function now finds and uses a metaclass corresponding to the provided base classes. Previously, only
type
instances were returned.The
tp_new
of the metaclass is ignored. which may result in incomplete initialization. Creating classes whose metaclass overridestp_new
is deprecated.Changed in version 3.14: Creating classes whose metaclass overrides
tp_new
is no longer allowed.
-
PyObject *PyType_FromSpec(PyType_Spec *spec)¶
- Return value: New reference. Part of the Stable ABI.
Equivalent to
PyType_FromMetaclass(NULL, NULL, spec, NULL)
.Changed in version 3.12: The function now finds and uses a metaclass corresponding to the base classes provided in Py_tp_base[s] slots. Previously, only
type
instances were returned.The
tp_new
of the metaclass is ignored. which may result in incomplete initialization. Creating classes whose metaclass overridestp_new
is deprecated.Changed in version 3.14: Creating classes whose metaclass overrides
tp_new
is no longer allowed.
-
int PyType_Freeze(PyTypeObject *type)¶
- Part of the Stable ABI since version 3.14.
Make a type immutable: set the
Py_TPFLAGS_IMMUTABLETYPE
flag.All base classes of type must be immutable.
On success, return
0
. On error, set an exception and return-1
.The type must not be used before it’s made immutable. For example, type instances must not be created before the type is made immutable.
Added in version 3.14.
-
type PyType_Spec¶
- Part of the Stable ABI (including all members).
Structure defining a type’s behavior.
-
const char *name¶
Name of the type, used to set
PyTypeObject.tp_name
.
-
int basicsize¶
If positive, specifies the size of the instance in bytes. It is used to set
PyTypeObject.tp_basicsize
.If zero, specifies that
tp_basicsize
should be inherited.If negative, the absolute value specifies how much space instances of the class need in addition to the superclass. Use
PyObject_GetTypeData()
to get a pointer to subclass-specific memory reserved this way. For negativebasicsize
, Python will insert padding when needed to meettp_basicsize
’s alignment requirements.Changed in version 3.12: Previously, this field could not be negative.
-
int itemsize¶
Size of one element of a variable-size type, in bytes. Used to set
PyTypeObject.tp_itemsize
. Seetp_itemsize
documentation for caveats.If zero,
tp_itemsize
is inherited. Extending arbitrary variable-sized classes is dangerous, since some types use a fixed offset for variable-sized memory, which can then overlap fixed-sized memory used by a subclass. To help prevent mistakes, inheritingitemsize
is only possible in the following situations:The base is not variable-sized (its
tp_itemsize
).The requested
PyType_Spec.basicsize
is positive, suggesting that the memory layout of the base class is known.The requested
PyType_Spec.basicsize
is zero, suggesting that the subclass does not access the instance’s memory directly.With the
Py_TPFLAGS_ITEMS_AT_END
flag.
-
unsigned int flags¶
Type flags, used to set
PyTypeObject.tp_flags
.If the
Py_TPFLAGS_HEAPTYPE
flag is not set,PyType_FromSpecWithBases()
sets it automatically.
-
PyType_Slot *slots¶
Array of
PyType_Slot
structures. Terminated by the special slot value{0, NULL}
.Each slot ID should be specified at most once.
-
const char *name¶
-
type PyType_Slot¶
- Part of the Stable ABI (including all members).
Structure defining optional functionality of a type, containing a slot ID and a value pointer.
-
int slot¶
A slot ID.
Slot IDs are named like the field names of the structures
PyTypeObject
,PyNumberMethods
,PySequenceMethods
,PyMappingMethods
andPyAsyncMethods
with an addedPy_
prefix. For example, use:Py_nb_add
to setPyNumberMethods.nb_add
An additional slot is supported that does not correspond to a
PyTypeObject
struct field:The following “offset” fields cannot be set using
PyType_Slot
:tp_weaklistoffset
(usePy_TPFLAGS_MANAGED_WEAKREF
instead if possible)tp_dictoffset
(usePy_TPFLAGS_MANAGED_DICT
instead if possible)tp_vectorcall_offset
(use"__vectorcalloffset__"
in PyMemberDef)
If it is not possible to switch to a
MANAGED
flag (for example, for vectorcall or to support Python older than 3.12), specify the offset inPy_tp_members
. See PyMemberDef documentation for details.The following internal fields cannot be set at all when creating a heap type:
tp_dict
,tp_mro
,tp_cache
,tp_subclasses
, andtp_weaklist
.
Setting
Py_tp_bases
orPy_tp_base
may be problematic on some platforms. To avoid issues, use the bases argument ofPyType_FromSpecWithBases()
instead.Changed in version 3.9: Slots in
PyBufferProcs
may be set in the unlimited API.Changed in version 3.11:
bf_getbuffer
andbf_releasebuffer
are now available under the limited API.Changed in version 3.14: The field
tp_vectorcall
can now be set usingPy_tp_vectorcall
. See the field’s documentation for details.
-
void *pfunc¶
The desired value of the slot. In most cases, this is a pointer to a function.
pfunc values may not be
NULL
, except for the following slots:Py_tp_token
(for clarity, preferPy_TP_USE_SPEC
rather thanNULL
)
-
int slot¶
-
Py_tp_token¶
- Part of the Stable ABI since version 3.14.
A
slot
that records a static memory layout ID for a class.If the
PyType_Spec
of the class is statically allocated, the token can be set to the spec using the special valuePy_TP_USE_SPEC
:static PyType_Slot foo_slots[] = { {Py_tp_token, Py_TP_USE_SPEC},
It can also be set to an arbitrary pointer, but you must ensure that:
The pointer outlives the class, so it’s not reused for something else while the class exists.
It “belongs” to the extension module where the class lives, so it will not clash with other extensions.
Use
PyType_GetBaseByToken()
to check if a class’s superclass has a given token – that is, check whether the memory layout is compatible.To get the token for a given class (without considering superclasses), use
PyType_GetSlot()
withPy_tp_token
.Added in version 3.14.
-
Py_TP_USE_SPEC¶
- Part of the Stable ABI since version 3.14.
Used as a value with
Py_tp_token
to set the token to the class’sPyType_Spec
. Expands toNULL
.Added in version 3.14.


Memory Management

Memory Management¶
Overview¶
Memory management in Python involves a private heap containing all Python objects and data structures. The management of this private heap is ensured internally by the Python memory manager. The Python memory manager has different components which deal with various dynamic storage management aspects, like sharing, segmentation, preallocation or caching.
At the lowest level, a raw memory allocator ensures that there is enough room in the private heap for storing all Python-related data by interacting with the memory manager of the operating system. On top of the raw memory allocator, several object-specific allocators operate on the same heap and implement distinct memory management policies adapted to the peculiarities of every object type. For example, integer objects are managed differently within the heap than strings, tuples or dictionaries because integers imply different storage requirements and speed/space tradeoffs. The Python memory manager thus delegates some of the work to the object-specific allocators, but ensures that the latter operate within the bounds of the private heap.
It is important to understand that the management of the Python heap is performed by the interpreter itself and that the user has no control over it, even if they regularly manipulate object pointers to memory blocks inside that heap. The allocation of heap space for Python objects and other internal buffers is performed on demand by the Python memory manager through the Python/C API functions listed in this document.
To avoid memory corruption, extension writers should never try to operate on
Python objects with the functions exported by the C library: malloc()
,
calloc()
, realloc()
and free()
. This will result in mixed
calls between the C allocator and the Python memory manager with fatal
consequences, because they implement different algorithms and operate on
different heaps. However, one may safely allocate and release memory blocks
with the C library allocator for individual purposes, as shown in the following
example:
PyObject *res;
char *buf = (char *) malloc(BUFSIZ); /* for I/O */
if (buf == NULL)
return PyErr_NoMemory();
...Do some I/O operation involving buf...
res = PyBytes_FromString(buf);
free(buf); /* malloc'ed */
return res;
In this example, the memory request for the I/O buffer is handled by the C library allocator. The Python memory manager is involved only in the allocation of the bytes object returned as a result.
In most situations, however, it is recommended to allocate memory from the Python heap specifically because the latter is under control of the Python memory manager. For example, this is required when the interpreter is extended with new object types written in C. Another reason for using the Python heap is the desire to inform the Python memory manager about the memory needs of the extension module. Even when the requested memory is used exclusively for internal, highly specific purposes, delegating all memory requests to the Python memory manager causes the interpreter to have a more accurate image of its memory footprint as a whole. Consequently, under certain circumstances, the Python memory manager may or may not trigger appropriate actions, like garbage collection, memory compaction or other preventive procedures. Note that by using the C library allocator as shown in the previous example, the allocated memory for the I/O buffer escapes completely the Python memory manager.
See also
The PYTHONMALLOC
environment variable can be used to configure
the memory allocators used by Python.
The PYTHONMALLOCSTATS
environment variable can be used to print
statistics of the pymalloc memory allocator every time a
new pymalloc object arena is created, and on shutdown.
Allocator Domains¶
All allocating functions belong to one of three different “domains” (see also
PyMemAllocatorDomain
). These domains represent different allocation
strategies and are optimized for different purposes. The specific details on
how every domain allocates memory or what internal functions each domain calls
is considered an implementation detail, but for debugging purposes a simplified
table can be found at Default Memory Allocators.
The APIs used to allocate and free a block of memory must be from the same domain.
For example, PyMem_Free()
must be used to free memory allocated using PyMem_Malloc()
.
The three allocation domains are:
Raw domain: intended for allocating memory for general-purpose memory buffers where the allocation must go to the system allocator or where the allocator can operate without an attached thread state. The memory is requested directly from the system. See Raw Memory Interface.
“Mem” domain: intended for allocating memory for Python buffers and general-purpose memory buffers where the allocation must be performed with an attached thread state. The memory is taken from the Python private heap. See Memory Interface.
Object domain: intended for allocating memory for Python objects. The memory is taken from the Python private heap. See Object allocators.
Note
The free-threaded build requires that only Python objects are allocated using the “object” domain and that all Python objects are allocated using that domain. This differs from the prior Python versions, where this was only a best practice and not a hard requirement.
For example, buffers (non-Python objects) should be allocated using PyMem_Malloc()
,
PyMem_RawMalloc()
, or malloc()
, but not PyObject_Malloc()
.
Raw Memory Interface¶
The following function sets are wrappers to the system allocator. These functions are thread-safe, so a thread state does not need to be attached.
The default raw memory allocator uses
the following functions: malloc()
, calloc()
, realloc()
and free()
; call malloc(1)
(or calloc(1, 1)
) when requesting
zero bytes.
Added in version 3.4.
-
void *PyMem_RawMalloc(size_t n)¶
- Part of the Stable ABI since version 3.13.
Allocates n bytes and returns a pointer of type void* to the allocated memory, or
NULL
if the request fails.Requesting zero bytes returns a distinct non-
NULL
pointer if possible, as ifPyMem_RawMalloc(1)
had been called instead. The memory will not have been initialized in any way.
-
void *PyMem_RawCalloc(size_t nelem, size_t elsize)¶
- Part of the Stable ABI since version 3.13.
Allocates nelem elements each whose size in bytes is elsize and returns a pointer of type void* to the allocated memory, or
NULL
if the request fails. The memory is initialized to zeros.Requesting zero elements or elements of size zero bytes returns a distinct non-
NULL
pointer if possible, as ifPyMem_RawCalloc(1, 1)
had been called instead.Added in version 3.5.
-
void *PyMem_RawRealloc(void *p, size_t n)¶
- Part of the Stable ABI since version 3.13.
Resizes the memory block pointed to by p to n bytes. The contents will be unchanged to the minimum of the old and the new sizes.
If p is
NULL
, the call is equivalent toPyMem_RawMalloc(n)
; else if n is equal to zero, the memory block is resized but is not freed, and the returned pointer is non-NULL
.Unless p is
NULL
, it must have been returned by a previous call toPyMem_RawMalloc()
,PyMem_RawRealloc()
orPyMem_RawCalloc()
.If the request fails,
PyMem_RawRealloc()
returnsNULL
and p remains a valid pointer to the previous memory area.
-
void PyMem_RawFree(void *p)¶
- Part of the Stable ABI since version 3.13.
Frees the memory block pointed to by p, which must have been returned by a previous call to
PyMem_RawMalloc()
,PyMem_RawRealloc()
orPyMem_RawCalloc()
. Otherwise, or ifPyMem_RawFree(p)
has been called before, undefined behavior occurs.If p is
NULL
, no operation is performed.
Memory Interface¶
The following function sets, modeled after the ANSI C standard, but specifying behavior when requesting zero bytes, are available for allocating and releasing memory from the Python heap.
The default memory allocator uses the pymalloc memory allocator.
Warning
There must be an attached thread state when using these functions.
Changed in version 3.6: The default allocator is now pymalloc instead of system malloc()
.
-
void *PyMem_Malloc(size_t n)¶
- Part of the Stable ABI.
Allocates n bytes and returns a pointer of type void* to the allocated memory, or
NULL
if the request fails.Requesting zero bytes returns a distinct non-
NULL
pointer if possible, as ifPyMem_Malloc(1)
had been called instead. The memory will not have been initialized in any way.
-
void *PyMem_Calloc(size_t nelem, size_t elsize)¶
- Part of the Stable ABI since version 3.7.
Allocates nelem elements each whose size in bytes is elsize and returns a pointer of type void* to the allocated memory, or
NULL
if the request fails. The memory is initialized to zeros.Requesting zero elements or elements of size zero bytes returns a distinct non-
NULL
pointer if possible, as ifPyMem_Calloc(1, 1)
had been called instead.Added in version 3.5.
-
void *PyMem_Realloc(void *p, size_t n)¶
- Part of the Stable ABI.
Resizes the memory block pointed to by p to n bytes. The contents will be unchanged to the minimum of the old and the new sizes.
If p is
NULL
, the call is equivalent toPyMem_Malloc(n)
; else if n is equal to zero, the memory block is resized but is not freed, and the returned pointer is non-NULL
.Unless p is
NULL
, it must have been returned by a previous call toPyMem_Malloc()
,PyMem_Realloc()
orPyMem_Calloc()
.If the request fails,
PyMem_Realloc()
returnsNULL
and p remains a valid pointer to the previous memory area.
-
void PyMem_Free(void *p)¶
- Part of the Stable ABI.
Frees the memory block pointed to by p, which must have been returned by a previous call to
PyMem_Malloc()
,PyMem_Realloc()
orPyMem_Calloc()
. Otherwise, or ifPyMem_Free(p)
has been called before, undefined behavior occurs.If p is
NULL
, no operation is performed.
The following type-oriented macros are provided for convenience. Note that TYPE refers to any C type.
-
PyMem_New(TYPE, n)¶
Same as
PyMem_Malloc()
, but allocates(n * sizeof(TYPE))
bytes of memory. Returns a pointer cast toTYPE*
. The memory will not have been initialized in any way.
-
PyMem_Resize(p, TYPE, n)¶
Same as
PyMem_Realloc()
, but the memory block is resized to(n * sizeof(TYPE))
bytes. Returns a pointer cast toTYPE*
. On return, p will be a pointer to the new memory area, orNULL
in the event of failure.This is a C preprocessor macro; p is always reassigned. Save the original value of p to avoid losing memory when handling errors.
-
void PyMem_Del(void *p)¶
Same as
PyMem_Free()
.
Deprecated aliases¶
These are soft deprecated aliases to existing functions and macros. They exist solely for backwards compatibility.
Deprecated alias |
Corresponding function or macro |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
Changed in version 3.4: The macros are now aliases of the corresponding functions and macros. Previously, their behavior was the same, but their use did not necessarily preserve binary compatibility across Python versions.
Deprecated since version 2.0.
Object allocators¶
The following function sets, modeled after the ANSI C standard, but specifying behavior when requesting zero bytes, are available for allocating and releasing memory from the Python heap.
Note
There is no guarantee that the memory returned by these allocators can be successfully cast to a Python object when intercepting the allocating functions in this domain by the methods described in the Customize Memory Allocators section.
The default object allocator uses the pymalloc memory allocator.
Warning
There must be an attached thread state when using these functions.
-
void *PyObject_Malloc(size_t n)¶
- Part of the Stable ABI.
Allocates n bytes and returns a pointer of type void* to the allocated memory, or
NULL
if the request fails.Requesting zero bytes returns a distinct non-
NULL
pointer if possible, as ifPyObject_Malloc(1)
had been called instead. The memory will not have been initialized in any way.
-
void *PyObject_Calloc(size_t nelem, size_t elsize)¶
- Part of the Stable ABI since version 3.7.
Allocates nelem elements each whose size in bytes is elsize and returns a pointer of type void* to the allocated memory, or
NULL
if the request fails. The memory is initialized to zeros.Requesting zero elements or elements of size zero bytes returns a distinct non-
NULL
pointer if possible, as ifPyObject_Calloc(1, 1)
had been called instead.Added in version 3.5.
-
void *PyObject_Realloc(void *p, size_t n)¶
- Part of the Stable ABI.
Resizes the memory block pointed to by p to n bytes. The contents will be unchanged to the minimum of the old and the new sizes.
If p is
NULL
, the call is equivalent toPyObject_Malloc(n)
; else if n is equal to zero, the memory block is resized but is not freed, and the returned pointer is non-NULL
.Unless p is
NULL
, it must have been returned by a previous call toPyObject_Malloc()
,PyObject_Realloc()
orPyObject_Calloc()
.If the request fails,
PyObject_Realloc()
returnsNULL
and p remains a valid pointer to the previous memory area.
-
void PyObject_Free(void *p)¶
- Part of the Stable ABI.
Frees the memory block pointed to by p, which must have been returned by a previous call to
PyObject_Malloc()
,PyObject_Realloc()
orPyObject_Calloc()
. Otherwise, or ifPyObject_Free(p)
has been called before, undefined behavior occurs.If p is
NULL
, no operation is performed.Do not call this directly to free an object’s memory; call the type’s
tp_free
slot instead.Do not use this for memory allocated by
PyObject_GC_New
orPyObject_GC_NewVar
; usePyObject_GC_Del()
instead.See also
PyObject_GC_Del()
is the equivalent of this function for memory allocated by types that support garbage collection.
Default Memory Allocators¶
Default memory allocators:
Configuration |
Name |
PyMem_RawMalloc |
PyMem_Malloc |
PyObject_Malloc |
|---|---|---|---|---|
Release build |
|
|
|
|
Debug build |
|
|
|
|
Release build, without pymalloc |
|
|
|
|
Debug build, without pymalloc |
|
|
|
|
Legend:
Name: value for
PYTHONMALLOC
environment variable.malloc
: system allocators from the standard C library, C functions:malloc()
,calloc()
,realloc()
andfree()
.pymalloc
: pymalloc memory allocator.mimalloc
: mimalloc memory allocator. The pymalloc allocator will be used if mimalloc support isn’t available.“+ debug”: with debug hooks on the Python memory allocators.
“Debug build”: Python build in debug mode.
Customize Memory Allocators¶
Added in version 3.4.
-
type PyMemAllocatorEx¶
Structure used to describe a memory block allocator. The structure has the following fields:
Field
Meaning
void *ctx
user context passed as first argument
void* malloc(void *ctx, size_t size)
allocate a memory block
void* calloc(void *ctx, size_t nelem, size_t elsize)
allocate a memory block initialized with zeros
void* realloc(void *ctx, void *ptr, size_t new_size)
allocate or resize a memory block
void free(void *ctx, void *ptr)
free a memory block
Changed in version 3.5: The
PyMemAllocator
structure was renamed toPyMemAllocatorEx
and a newcalloc
field was added.
-
type PyMemAllocatorDomain¶
Enum used to identify an allocator domain. Domains:
-
PYMEM_DOMAIN_RAW¶
Functions:
-
PYMEM_DOMAIN_MEM¶
Functions:
-
PYMEM_DOMAIN_OBJ¶
Functions:
-
PYMEM_DOMAIN_RAW¶
-
void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)¶
Get the memory block allocator of the specified domain.
-
void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)¶
Set the memory block allocator of the specified domain.
The new allocator must return a distinct non-
NULL
pointer when requesting zero bytes.For the
PYMEM_DOMAIN_RAW
domain, the allocator must be thread-safe: a thread state is not attached when the allocator is called.For the remaining domains, the allocator must also be thread-safe: the allocator may be called in different interpreters that do not share a GIL.
If the new allocator is not a hook (does not call the previous allocator), the
PyMem_SetupDebugHooks()
function must be called to reinstall the debug hooks on top on the new allocator.See also
PyPreConfig.allocator
and Preinitialize Python with PyPreConfig.Warning
PyMem_SetAllocator()
does have the following contract:It can be called after
Py_PreInitialize()
and beforePy_InitializeFromConfig()
to install a custom memory allocator. There are no restrictions over the installed allocator other than the ones imposed by the domain (for instance, the Raw Domain allows the allocator to be called without an attached thread state). See the section on allocator domains for more information.If called after Python has finish initializing (after
Py_InitializeFromConfig()
has been called) the allocator must wrap the existing allocator. Substituting the current allocator for some other arbitrary one is not supported.
Changed in version 3.12: All allocators must be thread-safe.
-
void PyMem_SetupDebugHooks(void)¶
Setup debug hooks in the Python memory allocators to detect memory errors.
Debug hooks on the Python memory allocators¶
When Python is built in debug mode, the
PyMem_SetupDebugHooks()
function is called at the Python
preinitialization to setup debug hooks on Python memory allocators
to detect memory errors.
The PYTHONMALLOC
environment variable can be used to install debug
hooks on a Python compiled in release mode (ex: PYTHONMALLOC=debug
).
The PyMem_SetupDebugHooks()
function can be used to set debug hooks
after calling PyMem_SetAllocator()
.
These debug hooks fill dynamically allocated memory blocks with special,
recognizable bit patterns. Newly allocated memory is filled with the byte
0xCD
(PYMEM_CLEANBYTE
), freed memory is filled with the byte 0xDD
(PYMEM_DEADBYTE
). Memory blocks are surrounded by “forbidden bytes”
filled with the byte 0xFD
(PYMEM_FORBIDDENBYTE
). Strings of these bytes
are unlikely to be valid addresses, floats, or ASCII strings.
Runtime checks:
Detect API violations. For example, detect if
PyObject_Free()
is called on a memory block allocated byPyMem_Malloc()
.Detect write before the start of the buffer (buffer underflow).
Detect write after the end of the buffer (buffer overflow).
Check that there is an attached thread state when allocator functions of
PYMEM_DOMAIN_OBJ
(ex:PyObject_Malloc()
) andPYMEM_DOMAIN_MEM
(ex:PyMem_Malloc()
) domains are called.
On error, the debug hooks use the tracemalloc
module to get the
traceback where a memory block was allocated. The traceback is only displayed
if tracemalloc
is tracing Python memory allocations and the memory block
was traced.
Let S = sizeof(size_t)
. 2*S
bytes are added at each end of each block
of N bytes requested. The memory layout is like so, where p represents the
address returned by a malloc-like or realloc-like function (p[i:j]
means
the slice of bytes from *(p+i)
inclusive up to *(p+j)
exclusive; note
that the treatment of negative indices differs from a Python slice):
p[-2*S:-S]
Number of bytes originally asked for. This is a size_t, big-endian (easier to read in a memory dump).
p[-S]
API identifier (ASCII character):
'r'
forPYMEM_DOMAIN_RAW
.'m'
forPYMEM_DOMAIN_MEM
.'o'
forPYMEM_DOMAIN_OBJ
.
p[-S+1:0]
Copies of PYMEM_FORBIDDENBYTE. Used to catch under- writes and reads.
p[0:N]
The requested memory, filled with copies of PYMEM_CLEANBYTE, used to catch reference to uninitialized memory. When a realloc-like function is called requesting a larger memory block, the new excess bytes are also filled with PYMEM_CLEANBYTE. When a free-like function is called, these are overwritten with PYMEM_DEADBYTE, to catch reference to freed memory. When a realloc- like function is called requesting a smaller memory block, the excess old bytes are also filled with PYMEM_DEADBYTE.
p[N:N+S]
Copies of PYMEM_FORBIDDENBYTE. Used to catch over- writes and reads.
p[N+S:N+2*S]
Only used if the
PYMEM_DEBUG_SERIALNO
macro is defined (not defined by default).A serial number, incremented by 1 on each call to a malloc-like or realloc-like function. Big-endian
size_t
. If “bad memory” is detected later, the serial number gives an excellent way to set a breakpoint on the next run, to capture the instant at which this block was passed out. The static function bumpserialno() in obmalloc.c is the only place the serial number is incremented, and exists so you can set such a breakpoint easily.
A realloc-like or free-like function first checks that the PYMEM_FORBIDDENBYTE bytes at each end are intact. If they’ve been altered, diagnostic output is written to stderr, and the program is aborted via Py_FatalError(). The other main failure mode is provoking a memory error when a program reads up one of the special bit patterns and tries to use it as an address. If you get in a debugger then and look at the object, you’re likely to see that it’s entirely filled with PYMEM_DEADBYTE (meaning freed memory is getting used) or PYMEM_CLEANBYTE (meaning uninitialized memory is getting used).
Changed in version 3.6: The PyMem_SetupDebugHooks()
function now also works on Python
compiled in release mode. On error, the debug hooks now use
tracemalloc
to get the traceback where a memory block was allocated.
The debug hooks now also check if there is an attached thread state when
functions of PYMEM_DOMAIN_OBJ
and PYMEM_DOMAIN_MEM
domains are
called.
Changed in version 3.8: Byte patterns 0xCB
(PYMEM_CLEANBYTE
), 0xDB
(PYMEM_DEADBYTE
)
and 0xFB
(PYMEM_FORBIDDENBYTE
) have been replaced with 0xCD
,
0xDD
and 0xFD
to use the same values than Windows CRT debug
malloc()
and free()
.
The pymalloc allocator¶
Python has a pymalloc allocator optimized for small objects (smaller or equal
to 512 bytes) with a short lifetime. It uses memory mappings called “arenas”
with a fixed size of either 256 KiB on 32-bit platforms or 1 MiB on 64-bit
platforms. It falls back to PyMem_RawMalloc()
and
PyMem_RawRealloc()
for allocations larger than 512 bytes.
pymalloc is the default allocator of the
PYMEM_DOMAIN_MEM
(ex: PyMem_Malloc()
) and
PYMEM_DOMAIN_OBJ
(ex: PyObject_Malloc()
) domains.
The arena allocator uses the following functions:
VirtualAlloc()
andVirtualFree()
on Windows,mmap()
andmunmap()
if available,malloc()
andfree()
otherwise.
This allocator is disabled if Python is configured with the
--without-pymalloc
option. It can also be disabled at runtime using
the PYTHONMALLOC
environment variable (ex: PYTHONMALLOC=malloc
).
Typically, it makes sense to disable the pymalloc allocator when building
Python with AddressSanitizer (--with-address-sanitizer
) which helps
uncover low level bugs within the C code.
Customize pymalloc Arena Allocator¶
Added in version 3.4.
-
type PyObjectArenaAllocator¶
Structure used to describe an arena allocator. The structure has three fields:
Field
Meaning
void *ctx
user context passed as first argument
void* alloc(void *ctx, size_t size)
allocate an arena of size bytes
void free(void *ctx, void *ptr, size_t size)
free an arena
-
void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)¶
Get the arena allocator.
-
void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)¶
Set the arena allocator.
The mimalloc allocator¶
Added in version 3.13.
Python supports the mimalloc allocator when the underlying platform support is available. mimalloc “is a general purpose allocator with excellent performance characteristics. Initially developed by Daan Leijen for the runtime systems of the Koka and Lean languages.”
tracemalloc C API¶
Added in version 3.7.
-
int PyTraceMalloc_Track(unsigned int domain, uintptr_t ptr, size_t size)¶
Track an allocated memory block in the
tracemalloc
module.Return
0
on success, return-1
on error (failed to allocate memory to store the trace). Return-2
if tracemalloc is disabled.If memory block is already tracked, update the existing trace.
-
int PyTraceMalloc_Untrack(unsigned int domain, uintptr_t ptr)¶
Untrack an allocated memory block in the
tracemalloc
module. Do nothing if the block was not tracked.Return
-2
if tracemalloc is disabled, otherwise return0
.
Examples¶
Here is the example from section Overview, rewritten so that the I/O buffer is allocated from the Python heap by using the first function set:
PyObject *res;
char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */
if (buf == NULL)
return PyErr_NoMemory();
/* ...Do some I/O operation involving buf... */
res = PyBytes_FromString(buf);
PyMem_Free(buf); /* allocated with PyMem_Malloc */
return res;
The same code using the type-oriented function set:
PyObject *res;
char *buf = PyMem_New(char, BUFSIZ); /* for I/O */
if (buf == NULL)
return PyErr_NoMemory();
/* ...Do some I/O operation involving buf... */
res = PyBytes_FromString(buf);
PyMem_Free(buf); /* allocated with PyMem_New */
return res;
Note that in the two examples above, the buffer is always manipulated via functions belonging to the same set. Indeed, it is required to use the same memory API family for a given memory block, so that the risk of mixing different allocators is reduced to a minimum. The following code sequence contains two errors, one of which is labeled as fatal because it mixes two different allocators operating on different heaps.
char *buf1 = PyMem_New(char, BUFSIZ);
char *buf2 = (char *) malloc(BUFSIZ);
char *buf3 = (char *) PyMem_Malloc(BUFSIZ);
...
PyMem_Del(buf3); /* Wrong -- should be PyMem_Free() */
free(buf2); /* Right -- allocated via malloc() */
free(buf1); /* Fatal -- should be PyMem_Free() */
In addition to the functions aimed at handling raw memory blocks from the Python
heap, objects in Python are allocated and released with PyObject_New
,
PyObject_NewVar
and PyObject_Free()
.
These will be explained in the next chapter on defining and implementing new object types in C.