Errors and Exceptions

8. Errors and Exceptions¶
Until now error messages haven’t been more than mentioned, but if you have tried out the examples you have probably seen some. There are (at least) two distinguishable kinds of errors: syntax errors and exceptions.
8.1. Syntax Errors¶
Syntax errors, also known as parsing errors, are perhaps the most common kind of complaint you get while you are still learning Python:
>>> while True print('Hello world')
File "<stdin>", line 1
while True print('Hello world')
^^^^^
SyntaxError: invalid syntax
The parser repeats the offending line and displays little arrows pointing
at the place where the error was detected. Note that this is not always the
place that needs to be fixed. In the example, the error is detected at the
function print()
, since a colon (':'
) is missing just before it.
The file name (<stdin>
in our example) and line number are printed so you
know where to look in case the input came from a file.
8.2. Exceptions¶
Even if a statement or expression is syntactically correct, it may cause an error when an attempt is made to execute it. Errors detected during execution are called exceptions and are not unconditionally fatal: you will soon learn how to handle them in Python programs. Most exceptions are not handled by programs, however, and result in error messages as shown here:
>>> 10 * (1/0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
10 * (1/0)
~^~
ZeroDivisionError: division by zero
>>> 4 + spam*3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
4 + spam*3
^^^^
NameError: name 'spam' is not defined
>>> '2' + 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
'2' + 2
~~~~^~~
TypeError: can only concatenate str (not "int") to str
The last line of the error message indicates what happened. Exceptions come in
different types, and the type is printed as part of the message: the types in
the example are ZeroDivisionError
, NameError
and TypeError
.
The string printed as the exception type is the name of the built-in exception
that occurred. This is true for all built-in exceptions, but need not be true
for user-defined exceptions (although it is a useful convention). Standard
exception names are built-in identifiers (not reserved keywords).
The rest of the line provides detail based on the type of exception and what caused it.
The preceding part of the error message shows the context where the exception occurred, in the form of a stack traceback. In general it contains a stack traceback listing source lines; however, it will not display lines read from standard input.
Built-in Exceptions lists the built-in exceptions and their meanings.
8.3. Handling Exceptions¶
It is possible to write programs that handle selected exceptions. Look at the
following example, which asks the user for input until a valid integer has been
entered, but allows the user to interrupt the program (using Control-C or
whatever the operating system supports); note that a user-generated interruption
is signalled by raising the KeyboardInterrupt
exception.
>>> while True:
... try:
... x = int(input("Please enter a number: "))
... break
... except ValueError:
... print("Oops! That was no valid number. Try again...")
...
The try
statement works as follows.
First, the try clause (the statement(s) between the
try
andexcept
keywords) is executed.If no exception occurs, the except clause is skipped and execution of the
try
statement is finished.If an exception occurs during execution of the
try
clause, the rest of the clause is skipped. Then, if its type matches the exception named after theexcept
keyword, the except clause is executed, and then execution continues after the try/except block.If an exception occurs which does not match the exception named in the except clause, it is passed on to outer
try
statements; if no handler is found, it is an unhandled exception and execution stops with an error message.
A try
statement may have more than one except clause, to specify
handlers for different exceptions. At most one handler will be executed.
Handlers only handle exceptions that occur in the corresponding try clause,
not in other handlers of the same try
statement. An except clause
may name multiple exceptions as a parenthesized tuple, for example:
... except (RuntimeError, TypeError, NameError):
... pass
A class in an except
clause matches exceptions which are instances of the
class itself or one of its derived classes (but not the other way around — an
except clause listing a derived class does not match instances of its base classes).
For example, the following code will print B, C, D in that order:
class B(Exception):
pass
class C(B):
pass
class D(C):
pass
for cls in [B, C, D]:
try:
raise cls()
except D:
print("D")
except C:
print("C")
except B:
print("B")
Note that if the except clauses were reversed (with except B
first), it
would have printed B, B, B — the first matching except clause is triggered.
When an exception occurs, it may have associated values, also known as the exception’s arguments. The presence and types of the arguments depend on the exception type.
The except clause may specify a variable after the exception name. The
variable is bound to the exception instance which typically has an args
attribute that stores the arguments. For convenience, builtin exception
types define __str__()
to print all the arguments without explicitly
accessing .args
.
>>> try:
... raise Exception('spam', 'eggs')
... except Exception as inst:
... print(type(inst)) # the exception type
... print(inst.args) # arguments stored in .args
... print(inst) # __str__ allows args to be printed directly,
... # but may be overridden in exception subclasses
... x, y = inst.args # unpack args
... print('x =', x)
... print('y =', y)
...
<class 'Exception'>
('spam', 'eggs')
('spam', 'eggs')
x = spam
y = eggs
The exception’s __str__()
output is printed as the last part (‘detail’)
of the message for unhandled exceptions.
BaseException
is the common base class of all exceptions. One of its
subclasses, Exception
, is the base class of all the non-fatal exceptions.
Exceptions which are not subclasses of Exception
are not typically
handled, because they are used to indicate that the program should terminate.
They include SystemExit
which is raised by sys.exit()
and
KeyboardInterrupt
which is raised when a user wishes to interrupt
the program.
Exception
can be used as a wildcard that catches (almost) everything.
However, it is good practice to be as specific as possible with the types
of exceptions that we intend to handle, and to allow any unexpected
exceptions to propagate on.
The most common pattern for handling Exception
is to print or log
the exception and then re-raise it (allowing a caller to handle the
exception as well):
import sys
try:
f = open('myfile.txt')
s = f.readline()
i = int(s.strip())
except OSError as err:
print("OS error:", err)
except ValueError:
print("Could not convert data to an integer.")
except Exception as err:
print(f"Unexpected {err=}, {type(err)=}")
raise
The try
… except
statement has an optional else
clause, which, when present, must follow all except clauses. It is useful
for code that must be executed if the try clause does not raise an exception.
For example:
for arg in sys.argv[1:]:
try:
f = open(arg, 'r')
except OSError:
print('cannot open', arg)
else:
print(arg, 'has', len(f.readlines()), 'lines')
f.close()
The use of the else
clause is better than adding additional code to
the try
clause because it avoids accidentally catching an exception
that wasn’t raised by the code being protected by the try
…
except
statement.
Exception handlers do not handle only exceptions that occur immediately in the try clause, but also those that occur inside functions that are called (even indirectly) in the try clause. For example:
>>> def this_fails():
... x = 1/0
...
>>> try:
... this_fails()
... except ZeroDivisionError as err:
... print('Handling run-time error:', err)
...
Handling run-time error: division by zero
8.4. Raising Exceptions¶
The raise
statement allows the programmer to force a specified
exception to occur. For example:
>>> raise NameError('HiThere')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
raise NameError('HiThere')
NameError: HiThere
The sole argument to raise
indicates the exception to be raised.
This must be either an exception instance or an exception class (a class that
derives from BaseException
, such as Exception
or one of its
subclasses). If an exception class is passed, it will be implicitly
instantiated by calling its constructor with no arguments:
raise ValueError # shorthand for 'raise ValueError()'
If you need to determine whether an exception was raised but don’t intend to
handle it, a simpler form of the raise
statement allows you to
re-raise the exception:
>>> try:
... raise NameError('HiThere')
... except NameError:
... print('An exception flew by!')
... raise
...
An exception flew by!
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
raise NameError('HiThere')
NameError: HiThere
8.5. Exception Chaining¶
If an unhandled exception occurs inside an except
section, it will
have the exception being handled attached to it and included in the error
message:
>>> try:
... open("database.sqlite")
... except OSError:
... raise RuntimeError("unable to handle error")
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
open("database.sqlite")
~~~~^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'database.sqlite'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
raise RuntimeError("unable to handle error")
RuntimeError: unable to handle error
To indicate that an exception is a direct consequence of another, the
raise
statement allows an optional from
clause:
# exc must be exception instance or None.
raise RuntimeError from exc
This can be useful when you are transforming exceptions. For example:
>>> def func():
... raise ConnectionError
...
>>> try:
... func()
... except ConnectionError as exc:
... raise RuntimeError('Failed to open database') from exc
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
func()
~~~~^^
File "<stdin>", line 2, in func
ConnectionError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
raise RuntimeError('Failed to open database') from exc
RuntimeError: Failed to open database
It also allows disabling automatic exception chaining using the from None
idiom:
>>> try:
... open('database.sqlite')
... except OSError:
... raise RuntimeError from None
...
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
raise RuntimeError from None
RuntimeError
For more information about chaining mechanics, see Built-in Exceptions.
8.6. User-defined Exceptions¶
Programs may name their own exceptions by creating a new exception class (see
Classes for more about Python classes). Exceptions should typically
be derived from the Exception
class, either directly or indirectly.
Exception classes can be defined which do anything any other class can do, but are usually kept simple, often only offering a number of attributes that allow information about the error to be extracted by handlers for the exception.
Most exceptions are defined with names that end in “Error”, similar to the naming of the standard exceptions.
Many standard modules define their own exceptions to report errors that may occur in functions they define.
8.7. Defining Clean-up Actions¶
The try
statement has another optional clause which is intended to
define clean-up actions that must be executed under all circumstances. For
example:
>>> try:
... raise KeyboardInterrupt
... finally:
... print('Goodbye, world!')
...
Goodbye, world!
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
raise KeyboardInterrupt
KeyboardInterrupt
If a finally
clause is present, the finally
clause will execute as the last task before the try
statement completes. The finally
clause runs whether or
not the try
statement produces an exception. The following
points discuss more complex cases when an exception occurs:
If an exception occurs during execution of the
try
clause, the exception may be handled by anexcept
clause. If the exception is not handled by anexcept
clause, the exception is re-raised after thefinally
clause has been executed.An exception could occur during execution of an
except
orelse
clause. Again, the exception is re-raised after thefinally
clause has been executed.If the
finally
clause executes abreak
,continue
orreturn
statement, exceptions are not re-raised. This can be confusing and is therefore discouraged. From version 3.14 the compiler emits aSyntaxWarning
for it (see PEP 765).If the
try
statement reaches abreak
,continue
orreturn
statement, thefinally
clause will execute just prior to thebreak
,continue
orreturn
statement’s execution.If a
finally
clause includes areturn
statement, the returned value will be the one from thefinally
clause’sreturn
statement, not the value from thetry
clause’sreturn
statement. This can be confusing and is therefore discouraged. From version 3.14 the compiler emits aSyntaxWarning
for it (see PEP 765).
For example:
>>> def bool_return():
... try:
... return True
... finally:
... return False
...
>>> bool_return()
False
A more complicated example:
>>> def divide(x, y):
... try:
... result = x / y
... except ZeroDivisionError:
... print("division by zero!")
... else:
... print("result is", result)
... finally:
... print("executing finally clause")
...
>>> divide(2, 1)
result is 2.0
executing finally clause
>>> divide(2, 0)
division by zero!
executing finally clause
>>> divide("2", "1")
executing finally clause
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
divide("2", "1")
~~~~~~^^^^^^^^^^
File "<stdin>", line 3, in divide
result = x / y
~~^~~
TypeError: unsupported operand type(s) for /: 'str' and 'str'
As you can see, the finally
clause is executed in any event. The
TypeError
raised by dividing two strings is not handled by the
except
clause and therefore re-raised after the finally
clause has been executed.
In real world applications, the finally
clause is useful for
releasing external resources (such as files or network connections), regardless
of whether the use of the resource was successful.
8.8. Predefined Clean-up Actions¶
Some objects define standard clean-up actions to be undertaken when the object is no longer needed, regardless of whether or not the operation using the object succeeded or failed. Look at the following example, which tries to open a file and print its contents to the screen.
for line in open("myfile.txt"):
print(line, end="")
The problem with this code is that it leaves the file open for an indeterminate
amount of time after this part of the code has finished executing.
This is not an issue in simple scripts, but can be a problem for larger
applications. The with
statement allows objects like files to be
used in a way that ensures they are always cleaned up promptly and correctly.
with open("myfile.txt") as f:
for line in f:
print(line, end="")
After the statement is executed, the file f is always closed, even if a problem was encountered while processing the lines. Objects which, like files, provide predefined clean-up actions will indicate this in their documentation.
8.10. Enriching Exceptions with Notes¶
When an exception is created in order to be raised, it is usually initialized
with information that describes the error that has occurred. There are cases
where it is useful to add information after the exception was caught. For this
purpose, exceptions have a method add_note(note)
that accepts a string and
adds it to the exception’s notes list. The standard traceback rendering
includes all notes, in the order they were added, after the exception.
>>> try:
... raise TypeError('bad type')
... except Exception as e:
... e.add_note('Add some information')
... e.add_note('Add some more information')
... raise
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
raise TypeError('bad type')
TypeError: bad type
Add some information
Add some more information
>>>
For example, when collecting exceptions into an exception group, we may want to add context information for the individual errors. In the following each exception in the group has a note indicating when this error has occurred.
>>> def f():
... raise OSError('operation failed')
...
>>> excs = []
>>> for i in range(3):
... try:
... f()
... except Exception as e:
... e.add_note(f'Happened in Iteration {i+1}')
... excs.append(e)
...
>>> raise ExceptionGroup('We have some problems', excs)
+ Exception Group Traceback (most recent call last):
| File "<stdin>", line 1, in <module>
| raise ExceptionGroup('We have some problems', excs)
| ExceptionGroup: We have some problems (3 sub-exceptions)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "<stdin>", line 3, in <module>
| f()
| ~^^
| File "<stdin>", line 2, in f
| raise OSError('operation failed')
| OSError: operation failed
| Happened in Iteration 1
+---------------- 2 ----------------
| Traceback (most recent call last):
| File "<stdin>", line 3, in <module>
| f()
| ~^^
| File "<stdin>", line 2, in f
| raise OSError('operation failed')
| OSError: operation failed
| Happened in Iteration 2
+---------------- 3 ----------------
| Traceback (most recent call last):
| File "<stdin>", line 3, in <module>
| f()
| ~^^
| File "<stdin>", line 2, in f
| raise OSError('operation failed')
| OSError: operation failed
| Happened in Iteration 3
+------------------------------------
>>>

File
, line
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
File
, line
, in
File
, line
, in
:
File
, line
, in
File
, line
, in
File
, line
, in
File
, line
, in
:
File
, line
, in
:


Python HOWTOs

Python HOWTOs¶
Python HOWTOs are documents that cover a specific topic in-depth. Modeled on the Linux Documentation Project’s HOWTO collection, this collection is an effort to foster documentation that’s more detailed than the Python Library Reference.
General:
Advanced development:
Debugging and profiling:


Monitoring C API

Monitoring C API¶
Added in version 3.13.
An extension may need to interact with the event monitoring system. Subscribing
to events and registering callbacks can be done via the Python API exposed in
sys.monitoring
.
Generating Execution Events¶
The functions below make it possible for an extension to fire monitoring
events as it emulates the execution of Python code. Each of these functions
accepts a PyMonitoringState
struct which contains concise information
about the activation state of events, as well as the event arguments, which
include a PyObject*
representing the code object, the instruction offset
and sometimes additional, event-specific arguments (see sys.monitoring
for details about the signatures of the different event callbacks).
The codelike
argument should be an instance of types.CodeType
or of a type that emulates it.
The VM disables tracing when firing an event, so there is no need for user code to do that.
Monitoring functions should not be called with an exception set, except those listed below as working with the current exception.
-
type PyMonitoringState¶
Representation of the state of an event type. It is allocated by the user while its contents are maintained by the monitoring API functions described below.
All of the functions below return 0 on success and -1 (with an exception set) on error.
See sys.monitoring
for descriptions of the events.
-
int PyMonitoring_FirePyStartEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset)¶
Fire a
PY_START
event.
-
int PyMonitoring_FirePyResumeEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset)¶
Fire a
PY_RESUME
event.
-
int PyMonitoring_FirePyReturnEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset, PyObject *retval)¶
Fire a
PY_RETURN
event.
-
int PyMonitoring_FirePyYieldEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset, PyObject *retval)¶
Fire a
PY_YIELD
event.
-
int PyMonitoring_FireCallEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset, PyObject *callable, PyObject *arg0)¶
Fire a
CALL
event.
-
int PyMonitoring_FireLineEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset, int lineno)¶
Fire a
LINE
event.
-
int PyMonitoring_FireJumpEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset, PyObject *target_offset)¶
Fire a
JUMP
event.
-
int PyMonitoring_FireBranchLeftEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset, PyObject *target_offset)¶
Fire a
BRANCH_LEFT
event.
-
int PyMonitoring_FireBranchRightEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset, PyObject *target_offset)¶
Fire a
BRANCH_RIGHT
event.
-
int PyMonitoring_FireCReturnEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset, PyObject *retval)¶
Fire a
C_RETURN
event.
-
int PyMonitoring_FirePyThrowEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset)¶
Fire a
PY_THROW
event with the current exception (as returned byPyErr_GetRaisedException()
).
-
int PyMonitoring_FireRaiseEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset)¶
Fire a
RAISE
event with the current exception (as returned byPyErr_GetRaisedException()
).
-
int PyMonitoring_FireCRaiseEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset)¶
Fire a
C_RAISE
event with the current exception (as returned byPyErr_GetRaisedException()
).
-
int PyMonitoring_FireReraiseEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset)¶
Fire a
RERAISE
event with the current exception (as returned byPyErr_GetRaisedException()
).
-
int PyMonitoring_FireExceptionHandledEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset)¶
Fire an
EXCEPTION_HANDLED
event with the current exception (as returned byPyErr_GetRaisedException()
).
-
int PyMonitoring_FirePyUnwindEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset)¶
Fire a
PY_UNWIND
event with the current exception (as returned byPyErr_GetRaisedException()
).
-
int PyMonitoring_FireStopIterationEvent(PyMonitoringState *state, PyObject *codelike, int32_t offset, PyObject *value)¶
Fire a
STOP_ITERATION
event. Ifvalue
is an instance ofStopIteration
, it is used. Otherwise, a newStopIteration
instance is created withvalue
as its argument.
Managing the Monitoring State¶
Monitoring states can be managed with the help of monitoring scopes. A scope would typically correspond to a Python function.
-
int PyMonitoring_EnterScope(PyMonitoringState *state_array, uint64_t *version, const uint8_t *event_types, Py_ssize_t length)¶
Enter a monitored scope.
event_types
is an array of the event IDs for events that may be fired from the scope. For example, the ID of aPY_START
event is the valuePY_MONITORING_EVENT_PY_START
, which is numerically equal to the base-2 logarithm ofsys.monitoring.events.PY_START
.state_array
is an array with a monitoring state entry for each event inevent_types
, it is allocated by the user but populated byPyMonitoring_EnterScope()
with information about the activation state of the event. The size ofevent_types
(and hence also ofstate_array
) is given inlength
.The
version
argument is a pointer to a value which should be allocated by the user together withstate_array
and initialized to 0, and then set only byPyMonitoring_EnterScope()
itself. It allows this function to determine whether event states have changed since the previous call, and to return quickly if they have not.The scopes referred to here are lexical scopes: a function, class or method.
PyMonitoring_EnterScope()
should be called whenever the lexical scope is entered. Scopes can be reentered, reusing the same state_array and version, in situations like when emulating a recursive Python function. When a code-like’s execution is paused, such as when emulating a generator, the scope needs to be exited and re-entered.The macros for event_types are:
Macro
Event
-
PY_MONITORING_EVENT_BRANCH_LEFT¶
-
PY_MONITORING_EVENT_BRANCH_RIGHT¶
-
PY_MONITORING_EVENT_CALL¶
-
PY_MONITORING_EVENT_C_RAISE¶
-
PY_MONITORING_EVENT_C_RETURN¶
-
PY_MONITORING_EVENT_EXCEPTION_HANDLED¶
-
PY_MONITORING_EVENT_INSTRUCTION¶
-
PY_MONITORING_EVENT_JUMP¶
-
PY_MONITORING_EVENT_LINE¶
-
PY_MONITORING_EVENT_PY_RESUME¶
-
PY_MONITORING_EVENT_PY_RETURN¶
-
PY_MONITORING_EVENT_PY_START¶
-
PY_MONITORING_EVENT_PY_THROW¶
-
PY_MONITORING_EVENT_PY_UNWIND¶
-
PY_MONITORING_EVENT_PY_YIELD¶
-
PY_MONITORING_EVENT_RAISE¶
-
PY_MONITORING_EVENT_RERAISE¶
-
PY_MONITORING_EVENT_STOP_ITERATION¶
-
PY_MONITORING_EVENT_BRANCH_LEFT¶
-
int PyMonitoring_ExitScope(void)¶
Exit the last scope that was entered with
PyMonitoring_EnterScope()
.
-
int PY_MONITORING_IS_INSTRUMENTED_EVENT(uint8_t ev)¶
Return true if the event corresponding to the event ID ev is a local event.
Added in version 3.13.
Deprecated since version 3.14: This function is soft deprecated.


“Why is Python Installed on my Computer?” FAQ¶
What is Python?¶
Python is a programming language. It’s used for many different applications. It’s used in some high schools and colleges as an introductory programming language because Python is easy to learn, but it’s also used by professional software developers at places such as Google, NASA, and Lucasfilm Ltd.
If you wish to learn more about Python, start with the Beginner’s Guide to Python.
Why is Python installed on my machine?¶
If you find Python installed on your system but don’t remember installing it, there are several possible ways it could have gotten there.
Perhaps another user on the computer wanted to learn programming and installed it; you’ll have to figure out who’s been using the machine and might have installed it.
A third-party application installed on the machine might have been written in Python and included a Python installation. There are many such applications, from GUI programs to network servers and administrative scripts.
Some Windows machines also have Python installed. At this writing we’re aware of computers from Hewlett-Packard and Compaq that include Python. Apparently some of HP/Compaq’s administrative tools are written in Python.
Many Unix-compatible operating systems, such as macOS and some Linux distributions, have Python installed by default; it’s included in the base installation.
Can I delete Python?¶
That depends on where Python came from.
If someone installed it deliberately, you can remove it without hurting anything. On Windows, use the Add/Remove Programs icon in the Control Panel.
If Python was installed by a third-party application, you can also remove it, but that application will no longer work. You should use that application’s uninstaller rather than removing Python directly.
If Python came with your operating system, removing it is not recommended. If you remove it, whatever tools were written in Python will no longer run, and some of them might be important to you. Reinstalling the whole system would then be required to fix things again.


Graphic User Interface FAQ¶
General GUI Questions¶
What GUI toolkits exist for Python?¶
Standard builds of Python include an object-oriented interface to the Tcl/Tk widget set, called tkinter. This is probably the easiest to install (since it comes included with most binary distributions of Python) and use. For more info about Tk, including pointers to the source, see the Tcl/Tk home page. Tcl/Tk is fully portable to the macOS, Windows, and Unix platforms.
Depending on what platform(s) you are aiming at, there are also several alternatives. A list of cross-platform and platform-specific GUI frameworks can be found on the python wiki.
Tkinter questions¶
How do I freeze Tkinter applications?¶
Freeze is a tool to create stand-alone applications. When freezing Tkinter applications, the applications will not be truly stand-alone, as the application will still need the Tcl and Tk libraries.
One solution is to ship the application with the Tcl and Tk libraries, and point
to them at run-time using the TCL_LIBRARY
and TK_LIBRARY
environment variables.
Various third-party freeze libraries such as py2exe and cx_Freeze have handling for Tkinter applications built-in.
Can I have Tk events handled while waiting for I/O?¶
On platforms other than Windows, yes, and you don’t even
need threads! But you’ll have to restructure your I/O
code a bit. Tk has the equivalent of Xt’s XtAddInput()
call, which allows you
to register a callback function which will be called from the Tk mainloop when
I/O is possible on a file descriptor. See File Handlers.
I can’t get key bindings to work in Tkinter: why?¶
An often-heard complaint is that event handlers bound
to events with the bind()
method
don’t get handled even when the appropriate key is pressed.
The most common cause is that the widget to which the binding applies doesn’t have “keyboard focus”. Check out the Tk documentation for the focus command. Usually a widget is given the keyboard focus by clicking in it (but not for labels; see the takefocus option).


Python on Windows FAQ¶
How do I run a Python program under Windows?¶
This is not necessarily a straightforward question. If you are already familiar with running programs from the Windows command line then everything will seem obvious; otherwise, you might need a little more guidance.
Unless you use some sort of integrated development environment, you will end up
typing Windows commands into what is referred to as a
“Command prompt window”. Usually you can create such a window from your
search bar by searching for cmd
. You should be able to recognize
when you have started such a window because you will see a Windows “command
prompt”, which usually looks like this:
C:\>
The letter may be different, and there might be other things after it, so you might just as easily see something like:
D:\YourName\Projects\Python>
depending on how your computer has been set up and what else you have recently done with it. Once you have started such a window, you are well on the way to running Python programs.
You need to realize that your Python scripts have to be processed by another program called the Python interpreter. The interpreter reads your script, compiles it into bytecodes, and then executes the bytecodes to run your program. So, how do you arrange for the interpreter to handle your Python?
First, you need to make sure that your command window recognises the word
“py” as an instruction to start the interpreter. If you have opened a
command window, you should try entering the command py
and hitting
return:
C:\Users\YourName> py
You should then see something like:
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
You have started the interpreter in “interactive mode”. That means you can enter Python statements or expressions interactively and have them executed or evaluated while you wait. This is one of Python’s strongest features. Check it by entering a few expressions of your choice and seeing the results:
>>> print("Hello")
Hello
>>> "Hello" * 3
'HelloHelloHello'
Many people use the interactive mode as a convenient yet highly programmable
calculator. When you want to end your interactive Python session,
call the exit()
function or hold the Ctrl key down
while you enter a Z, then hit the “Enter” key to get
back to your Windows command prompt.
You may also find that you have a Start-menu entry such as >>>
prompt in a new window. If so, the window will disappear
after you call the exit()
function or enter the Ctrl-Z
character; Windows is running a single “python”
command in the window, and closes it when you terminate the interpreter.
Now that we know the py
command is recognized, you can give your
Python script to it. You’ll have to give either an absolute or a
relative path to the Python script. Let’s say your Python script is
located in your desktop and is named hello.py
, and your command
prompt is nicely opened in your home directory so you’re seeing something
similar to:
C:\Users\YourName>
So now you’ll ask the py
command to give your script to Python by
typing py
followed by your script path:
C:\Users\YourName> py Desktop\hello.py
hello
How do I make Python scripts executable?¶
On Windows, the standard Python installer already associates the .py
extension with a file type (Python.File) and gives that file type an open
command that runs the interpreter (D:\Program Files\Python\python.exe "%1"
%*
). This is enough to make scripts executable from the command prompt as
‘foo.py’. If you’d rather be able to execute the script by simple typing ‘foo’
with no extension you need to add .py to the PATHEXT environment variable.
Why does Python sometimes take so long to start?¶
Usually Python starts very quickly on Windows, but occasionally there are bug reports that Python suddenly begins to take a long time to start up. This is made even more puzzling because Python will work fine on other Windows systems which appear to be configured identically.
The problem may be caused by a misconfiguration of virus checking software on the problem machine. Some virus scanners have been known to introduce startup overhead of two orders of magnitude when the scanner is configured to monitor all reads from the filesystem. Try checking the configuration of virus scanning software on your systems to ensure that they are indeed configured identically. McAfee, when configured to scan all file system read activity, is a particular offender.
How do I make an executable from a Python script?¶
See How can I create a stand-alone binary from a Python script? for a list of tools that can be used to make executables.
Is a *.pyd
file the same as a DLL?¶
Yes, .pyd files are dll’s, but there are a few differences. If you have a DLL
named foo.pyd
, then it must have a function PyInit_foo()
. You can then
write Python “import foo”, and Python will search for foo.pyd (as well as
foo.py, foo.pyc) and if it finds it, will attempt to call PyInit_foo()
to
initialize it. You do not link your .exe with foo.lib, as that would cause
Windows to require the DLL to be present.
Note that the search path for foo.pyd is PYTHONPATH, not the same as the path
that Windows uses to search for foo.dll. Also, foo.pyd need not be present to
run your program, whereas if you linked your program with a dll, the dll is
required. Of course, foo.pyd is required if you want to say import foo
. In
a DLL, linkage is declared in the source code with __declspec(dllexport)
.
In a .pyd, linkage is defined in a list of available functions.
How can I embed Python into a Windows application?¶
Embedding the Python interpreter in a Windows app can be summarized as follows:
Do not build Python into your .exe file directly. On Windows, Python must be a DLL to handle importing modules that are themselves DLL’s. (This is the first key undocumented fact.) Instead, link to
pythonNN.dll
; it is typically installed inC:\Windows\System
. NN is the Python version, a number such as “33” for Python 3.3.You can link to Python in two different ways. Load-time linking means linking against
pythonNN.lib
, while run-time linking means linking againstpythonNN.dll
. (General note:pythonNN.lib
is the so-called “import lib” corresponding topythonNN.dll
. It merely defines symbols for the linker.)Run-time linking greatly simplifies link options; everything happens at run time. Your code must load
pythonNN.dll
using the WindowsLoadLibraryEx()
routine. The code must also use access routines and data inpythonNN.dll
(that is, Python’s C API’s) using pointers obtained by the WindowsGetProcAddress()
routine. Macros can make using these pointers transparent to any C code that calls routines in Python’s C API.If you use SWIG, it is easy to create a Python “extension module” that will make the app’s data and methods available to Python. SWIG will handle just about all the grungy details for you. The result is C code that you link into your .exe file (!) You do not have to create a DLL file, and this also simplifies linking.
SWIG will create an init function (a C function) whose name depends on the name of the extension module. For example, if the name of the module is leo, the init function will be called initleo(). If you use SWIG shadow classes, as you should, the init function will be called initleoc(). This initializes a mostly hidden helper class used by the shadow class.
The reason you can link the C code in step 2 into your .exe file is that calling the initialization function is equivalent to importing the module into Python! (This is the second key undocumented fact.)
In short, you can use the following code to initialize the Python interpreter with your extension module.
#include <Python.h> ... Py_Initialize(); // Initialize Python. initmyAppc(); // Initialize (import) the helper class. PyRun_SimpleString("import myApp"); // Import the shadow class.
There are two problems with Python’s C API which will become apparent if you use a compiler other than MSVC, the compiler used to build pythonNN.dll.
Problem 1: The so-called “Very High Level” functions that take
FILE *
arguments will not work in a multi-compiler environment because each compiler’s notion of astruct FILE
will be different. From an implementation standpoint these are very low level functions.Problem 2: SWIG generates the following code when generating wrappers to void functions:
Py_INCREF(Py_None); _resultobj = Py_None; return _resultobj;
Alas, Py_None is a macro that expands to a reference to a complex data structure called _Py_NoneStruct inside pythonNN.dll. Again, this code will fail in a mult-compiler environment. Replace such code by:
return Py_BuildValue("");
It may be possible to use SWIG’s
%typemap
command to make the change automatically, though I have not been able to get this to work (I’m a complete SWIG newbie).Using a Python shell script to put up a Python interpreter window from inside your Windows app is not a good idea; the resulting window will be independent of your app’s windowing system. Rather, you (or the wxPythonWindow class) should create a “native” interpreter window. It is easy to connect that window to the Python interpreter. You can redirect Python’s i/o to _any_ object that supports read and write, so all you need is a Python object (defined in your extension module) that contains read() and write() methods.
How do I keep editors from inserting tabs into my Python source?¶
The FAQ does not recommend using tabs, and the Python style guide, PEP 8, recommends 4 spaces for distributed Python code; this is also the Emacs python-mode default.
Under any editor, mixing tabs and spaces is a bad idea. MSVC is no different in this respect, and is easily configured to use spaces: Take
, and for file type “Default” set “Tab size” and “Indent size” to 4, and select the “Insert spaces” radio button.Python raises IndentationError
or TabError
if mixed tabs
and spaces are causing problems in leading whitespace.
You may also run the tabnanny
module to check a directory tree
in batch mode.
How do I check for a keypress without blocking?¶
Use the msvcrt
module. This is a standard Windows-specific extension module.
It defines a function kbhit()
which checks whether a keyboard hit is
present, and getch()
which gets one character without echoing it.
How do I solve the missing api-ms-win-crt-runtime-l1-1-0.dll error?¶
This can occur on Python 3.5 and later when using Windows 8.1 or earlier without all updates having been installed. First ensure your operating system is supported and is up to date, and if that does not resolve the issue, visit the Microsoft support page for guidance on manually installing the C Runtime update.


Extending/Embedding FAQ¶
Can I create my own functions in C?¶
Yes, you can create built-in modules containing functions, variables, exceptions and even new types in C. This is explained in the document Extending and Embedding the Python Interpreter.
Most intermediate or advanced Python books will also cover this topic.
Can I create my own functions in C++?¶
Yes, using the C compatibility features found in C++. Place extern "C" {
... }
around the Python include files and put extern "C"
before each
function that is going to be called by the Python interpreter. Global or static
C++ objects with constructors are probably not a good idea.
Writing C is hard; are there any alternatives?¶
There are a number of alternatives to writing your own C extensions, depending on what you’re trying to do. Recommended third party tools offer both simpler and more sophisticated approaches to creating C and C++ extensions for Python.
How can I execute arbitrary Python statements from C?¶
The highest-level function to do this is PyRun_SimpleString()
which takes
a single string argument to be executed in the context of the module
__main__
and returns 0
for success and -1
when an exception occurred
(including SyntaxError
). If you want more control, use
PyRun_String()
; see the source for PyRun_SimpleString()
in
Python/pythonrun.c
.
How can I evaluate an arbitrary Python expression from C?¶
Call the function PyRun_String()
from the previous question with the
start symbol Py_eval_input
; it parses an expression, evaluates it and
returns its value.
How do I extract C values from a Python object?¶
That depends on the object’s type. If it’s a tuple, PyTuple_Size()
returns its length and PyTuple_GetItem()
returns the item at a specified
index. Lists have similar functions, PyList_Size()
and
PyList_GetItem()
.
For bytes, PyBytes_Size()
returns its length and
PyBytes_AsStringAndSize()
provides a pointer to its value and its
length. Note that Python bytes objects may contain null bytes so C’s
strlen()
should not be used.
To test the type of an object, first make sure it isn’t NULL
, and then use
PyBytes_Check()
, PyTuple_Check()
, PyList_Check()
, etc.
There is also a high-level API to Python objects which is provided by the
so-called ‘abstract’ interface – read Include/abstract.h
for further
details. It allows interfacing with any kind of Python sequence using calls
like PySequence_Length()
, PySequence_GetItem()
, etc. as well
as many other useful protocols such as numbers (PyNumber_Index()
et
al.) and mappings in the PyMapping APIs.
How do I use Py_BuildValue() to create a tuple of arbitrary length?¶
You can’t. Use PyTuple_Pack()
instead.
How do I call an object’s method from C?¶
The PyObject_CallMethod()
function can be used to call an arbitrary
method of an object. The parameters are the object, the name of the method to
call, a format string like that used with Py_BuildValue()
, and the
argument values:
PyObject *
PyObject_CallMethod(PyObject *object, const char *method_name,
const char *arg_format, ...);
This works for any object that has methods – whether built-in or user-defined.
You are responsible for eventually Py_DECREF()
‘ing the return value.
To call, e.g., a file object’s “seek” method with arguments 10, 0 (assuming the file object pointer is “f”):
res = PyObject_CallMethod(f, "seek", "(ii)", 10, 0);
if (res == NULL) {
... an exception occurred ...
}
else {
Py_DECREF(res);
}
Note that since PyObject_CallObject()
always wants a tuple for the
argument list, to call a function without arguments, pass “()” for the format,
and to call a function with one argument, surround the argument in parentheses,
e.g. “(i)”.
How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)?¶
In Python code, define an object that supports the write()
method. Assign
this object to sys.stdout
and sys.stderr
. Call print_error, or
just allow the standard traceback mechanism to work. Then, the output will go
wherever your write()
method sends it.
The easiest way to do this is to use the io.StringIO
class:
>>> import io, sys
>>> sys.stdout = io.StringIO()
>>> print('foo')
>>> print('hello world!')
>>> sys.stderr.write(sys.stdout.getvalue())
foo
hello world!
A custom object to do the same would look like this:
>>> import io, sys
>>> class StdoutCatcher(io.TextIOBase):
... def __init__(self):
... self.data = []
... def write(self, stuff):
... self.data.append(stuff)
...
>>> import sys
>>> sys.stdout = StdoutCatcher()
>>> print('foo')
>>> print('hello world!')
>>> sys.stderr.write(''.join(sys.stdout.data))
foo
hello world!
How do I access a module written in Python from C?¶
You can get a pointer to the module object as follows:
module = PyImport_ImportModule("<modulename>");
If the module hasn’t been imported yet (i.e. it is not yet present in
sys.modules
), this initializes the module; otherwise it simply returns
the value of sys.modules["<modulename>"]
. Note that it doesn’t enter the
module into any namespace – it only ensures it has been initialized and is
stored in sys.modules
.
You can then access the module’s attributes (i.e. any name defined in the module) as follows:
attr = PyObject_GetAttrString(module, "<attrname>");
Calling PyObject_SetAttrString()
to assign to variables in the module
also works.
How do I interface to C++ objects from Python?¶
Depending on your requirements, there are many approaches. To do this manually, begin by reading the “Extending and Embedding” document. Realize that for the Python run-time system, there isn’t a whole lot of difference between C and C++ – so the strategy of building a new Python type around a C structure (pointer) type will also work for C++ objects.
For C++ libraries, see Writing C is hard; are there any alternatives?.
I added a module using the Setup file and the make fails; why?¶
Setup must end in a newline, if there is no newline there, the build process fails. (Fixing this requires some ugly shell script hackery, and this bug is so minor that it doesn’t seem worth the effort.)
How do I debug an extension?¶
When using GDB with dynamically loaded extensions, you can’t set a breakpoint in your extension until your extension is loaded.
In your .gdbinit
file (or interactively), add the command:
br _PyImport_LoadDynamicModule
Then, when you run GDB:
$ gdb /local/bin/python
gdb) run myscript.py
gdb) continue # repeat until your extension is loaded
gdb) finish # so that your extension is loaded
gdb) br myfunction.c:50
gdb) continue
I want to compile a Python module on my Linux system, but some files are missing. Why?¶
Most packaged versions of Python omit some files required for compiling Python extensions.
For Red Hat, install the python3-devel RPM to get the necessary files.
For Debian, run apt-get install python3-dev
.
How do I tell “incomplete input” from “invalid input”?¶
Sometimes you want to emulate the Python interactive interpreter’s behavior, where it gives you a continuation prompt when the input is incomplete (e.g. you typed the start of an “if” statement or you didn’t close your parentheses or triple string quotes), but it gives you a syntax error message immediately when the input is invalid.
In Python you can use the codeop
module, which approximates the parser’s
behavior sufficiently. IDLE uses this, for example.
The easiest way to do it in C is to call PyRun_InteractiveLoop()
(perhaps
in a separate thread) and let the Python interpreter handle the input for
you. You can also set the PyOS_ReadlineFunctionPointer()
to point at your
custom input function. See Modules/readline.c
and Parser/myreadline.c
for more hints.
How do I find undefined g++ symbols __builtin_new or __pure_virtual?¶
To dynamically load g++ extension modules, you must recompile Python, relink it
using g++ (change LINKCC in the Python Modules Makefile), and link your
extension module using g++ (e.g., g++ -shared -o mymodule.so mymodule.o
).
Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)?¶
Yes, you can inherit from built-in classes such as int
, list
,
dict
, etc.
The Boost Python Library (BPL, https://www.boost.org/libs/python/doc/index.html) provides a way of doing this from C++ (i.e. you can inherit from an extension class written in C++ using the BPL).


Library and Extension FAQ¶
General Library Questions¶
How do I find a module or application to perform task X?¶
Check the Library Reference to see if there’s a relevant standard library module. (Eventually you’ll learn what’s in the standard library and will be able to skip this step.)
For third-party packages, search the Python Package Index or try Google or another web search engine. Searching for “Python” plus a keyword or two for your topic of interest will usually find something helpful.
Where is the math.py (socket.py, regex.py, etc.) source file?¶
If you can’t find a source file for a module it may be a built-in or
dynamically loaded module implemented in C, C++ or other compiled language.
In this case you may not have the source file or it may be something like
mathmodule.c
, somewhere in a C source directory (not on the Python Path).
There are (at least) three kinds of modules in Python:
modules written in Python (.py);
modules written in C and dynamically loaded (.dll, .pyd, .so, .sl, etc);
modules written in C and linked with the interpreter; to get a list of these, type:
import sys print(sys.builtin_module_names)
How do I make a Python script executable on Unix?¶
You need to do two things: the script file’s mode must be executable and the
first line must begin with #!
followed by the path of the Python
interpreter.
The first is done by executing chmod +x scriptfile
or perhaps chmod 755
scriptfile
.
The second can be done in a number of ways. The most straightforward way is to write
#!/usr/local/bin/python
as the very first line of your file, using the pathname for where the Python interpreter is installed on your platform.
If you would like the script to be independent of where the Python interpreter
lives, you can use the env program. Almost all Unix variants support
the following, assuming the Python interpreter is in a directory on the user’s
PATH
:
#!/usr/bin/env python
Don’t do this for CGI scripts. The PATH
variable for CGI scripts is
often very minimal, so you need to use the actual absolute pathname of the
interpreter.
Occasionally, a user’s environment is so full that the /usr/bin/env program fails; or there’s no env program at all. In that case, you can try the following hack (due to Alex Rezinsky):
#! /bin/sh
""":"
exec python $0 ${1+"$@"}
"""
The minor disadvantage is that this defines the script’s __doc__ string. However, you can fix that by adding
__doc__ = """...Whatever..."""
Is there a curses/termcap package for Python?¶
For Unix variants: The standard Python source distribution comes with a curses module in the Modules subdirectory, though it’s not compiled by default. (Note that this is not available in the Windows distribution – there is no curses module for Windows.)
The curses
module supports basic curses features as well as many additional
functions from ncurses and SYSV curses such as colour, alternative character set
support, pads, and mouse support. This means the module isn’t compatible with
operating systems that only have BSD curses, but there don’t seem to be any
currently maintained OSes that fall into this category.
Is there an equivalent to C’s onexit() in Python?¶
The atexit
module provides a register function that is similar to C’s
onexit()
.
Why don’t my signal handlers work?¶
The most common problem is that the signal handler is declared with the wrong argument list. It is called as
handler(signum, frame)
so it should be declared with two parameters:
def handler(signum, frame):
...
Common tasks¶
How do I test a Python program or component?¶
Python comes with two testing frameworks. The doctest
module finds
examples in the docstrings for a module and runs them, comparing the output with
the expected output given in the docstring.
The unittest
module is a fancier testing framework modelled on Java and
Smalltalk testing frameworks.
To make testing easier, you should use good modular design in your program. Your program should have almost all functionality encapsulated in either functions or class methods – and this sometimes has the surprising and delightful effect of making the program run faster (because local variable accesses are faster than global accesses). Furthermore the program should avoid depending on mutating global variables, since this makes testing much more difficult to do.
The “global main logic” of your program may be as simple as
if __name__ == "__main__":
main_logic()
at the bottom of the main module of your program.
Once your program is organized as a tractable collection of function and class behaviours, you should write test functions that exercise the behaviours. A test suite that automates a sequence of tests can be associated with each module. This sounds like a lot of work, but since Python is so terse and flexible it’s surprisingly easy. You can make coding much more pleasant and fun by writing your test functions in parallel with the “production code”, since this makes it easy to find bugs and even design flaws earlier.
“Support modules” that are not intended to be the main module of a program may include a self-test of the module.
if __name__ == "__main__":
self_test()
Even programs that interact with complex external interfaces may be tested when the external interfaces are unavailable by using “fake” interfaces implemented in Python.
How do I create documentation from doc strings?¶
The pydoc
module can create HTML from the doc strings in your Python
source code. An alternative for creating API documentation purely from
docstrings is epydoc. Sphinx can also include docstring content.
How do I get a single keypress at a time?¶
For Unix variants there are several solutions. It’s straightforward to do this using curses, but curses is a fairly large module to learn.
Threads¶
How do I program using threads?¶
Be sure to use the threading
module and not the _thread
module.
The threading
module builds convenient abstractions on top of the
low-level primitives provided by the _thread
module.
None of my threads seem to run: why?¶
As soon as the main thread exits, all threads are killed. Your main thread is running too quickly, giving the threads no time to do any work.
A simple fix is to add a sleep to the end of the program that’s long enough for all the threads to finish:
import threading, time
def thread_task(name, n):
for i in range(n):
print(name, i)
for i in range(10):
T = threading.Thread(target=thread_task, args=(str(i), i))
T.start()
time.sleep(10) # <---------------------------!
But now (on many platforms) the threads don’t run in parallel, but appear to run sequentially, one at a time! The reason is that the OS thread scheduler doesn’t start a new thread until the previous thread is blocked.
A simple fix is to add a tiny sleep to the start of the run function:
def thread_task(name, n):
time.sleep(0.001) # <--------------------!
for i in range(n):
print(name, i)
for i in range(10):
T = threading.Thread(target=thread_task, args=(str(i), i))
T.start()
time.sleep(10)
Instead of trying to guess a good delay value for time.sleep()
,
it’s better to use some kind of semaphore mechanism. One idea is to use the
queue
module to create a queue object, let each thread append a token to
the queue when it finishes, and let the main thread read as many tokens from the
queue as there are threads.
How do I parcel out work among a bunch of worker threads?¶
The easiest way is to use the concurrent.futures
module,
especially the ThreadPoolExecutor
class.
Or, if you want fine control over the dispatching algorithm, you can write
your own logic manually. Use the queue
module to create a queue
containing a list of jobs. The Queue
class maintains a
list of objects and has a .put(obj)
method that adds items to the queue and
a .get()
method to return them. The class will take care of the locking
necessary to ensure that each job is handed out exactly once.
Here’s a trivial example:
import threading, queue, time
# The worker thread gets jobs off the queue. When the queue is empty, it
# assumes there will be no more work and exits.
# (Realistically workers will run until terminated.)
def worker():
print('Running worker')
time.sleep(0.1)
while True:
try:
arg = q.get(block=False)
except queue.Empty:
print('Worker', threading.current_thread(), end=' ')
print('queue empty')
break
else:
print('Worker', threading.current_thread(), end=' ')
print('running with argument', arg)
time.sleep(0.5)
# Create queue
q = queue.Queue()
# Start a pool of 5 workers
for i in range(5):
t = threading.Thread(target=worker, name='worker %i' % (i+1))
t.start()
# Begin adding work to the queue
for i in range(50):
q.put(i)
# Give threads time to run
print('Main thread sleeping')
time.sleep(5)
When run, this will produce the following output:
Running worker
Running worker
Running worker
Running worker
Running worker
Main thread sleeping
Worker <Thread(worker 1, started 130283832797456)> running with argument 0
Worker <Thread(worker 2, started 130283824404752)> running with argument 1
Worker <Thread(worker 3, started 130283816012048)> running with argument 2
Worker <Thread(worker 4, started 130283807619344)> running with argument 3
Worker <Thread(worker 5, started 130283799226640)> running with argument 4
Worker <Thread(worker 1, started 130283832797456)> running with argument 5
...
Consult the module’s documentation for more details; the Queue
class provides a featureful interface.
What kinds of global value mutation are thread-safe?¶
A global interpreter lock (GIL) is used internally to ensure that only one
thread runs in the Python VM at a time. In general, Python offers to switch
among threads only between bytecode instructions; how frequently it switches can
be set via sys.setswitchinterval()
. Each bytecode instruction and
therefore all the C implementation code reached from each instruction is
therefore atomic from the point of view of a Python program.
In theory, this means an exact accounting requires an exact understanding of the PVM bytecode implementation. In practice, it means that operations on shared variables of built-in data types (ints, lists, dicts, etc) that “look atomic” really are.
For example, the following operations are all atomic (L, L1, L2 are lists, D, D1, D2 are dicts, x, y are objects, i, j are ints):
L.append(x)
L1.extend(L2)
x = L[i]
x = L.pop()
L1[i:j] = L2
L.sort()
x = y
x.field = y
D[x] = y
D1.update(D2)
D.keys()
These aren’t:
i = i+1
L.append(L[-1])
L[i] = L[j]
D[x] = D[x] + 1
Operations that replace other objects may invoke those other objects’
__del__()
method when their reference count reaches zero, and that can
affect things. This is especially true for the mass updates to dictionaries and
lists. When in doubt, use a mutex!
Can’t we get rid of the Global Interpreter Lock?¶
The global interpreter lock (GIL) is often seen as a hindrance to Python’s deployment on high-end multiprocessor server machines, because a multi-threaded Python program effectively only uses one CPU, due to the insistence that (almost) all Python code can only run while the GIL is held.
With the approval of PEP 703 work is now underway to remove the GIL from the CPython implementation of Python. Initially it will be implemented as an optional compiler flag when building the interpreter, and so separate builds will be available with and without the GIL. Long-term, the hope is to settle on a single build, once the performance implications of removing the GIL are fully understood. Python 3.13 is likely to be the first release containing this work, although it may not be completely functional in this release.
The current work to remove the GIL is based on a fork of Python 3.9 with the GIL removed by Sam Gross. Prior to that, in the days of Python 1.5, Greg Stein actually implemented a comprehensive patch set (the “free threading” patches) that removed the GIL and replaced it with fine-grained locking. Adam Olsen did a similar experiment in his python-safethread project. Unfortunately, both of these earlier experiments exhibited a sharp drop in single-thread performance (at least 30% slower), due to the amount of fine-grained locking necessary to compensate for the removal of the GIL. The Python 3.9 fork is the first attempt at removing the GIL with an acceptable performance impact.
The presence of the GIL in current Python releases
doesn’t mean that you can’t make good use of Python on multi-CPU machines!
You just have to be creative with dividing the work up between multiple
processes rather than multiple threads. The
ProcessPoolExecutor
class in the new
concurrent.futures
module provides an easy way of doing so; the
multiprocessing
module provides a lower-level API in case you want
more control over dispatching of tasks.
Judicious use of C extensions will also help; if you use a C extension to
perform a time-consuming task, the extension can release the GIL while the
thread of execution is in the C code and allow other threads to get some work
done. Some standard library modules such as zlib
and hashlib
already do this.
An alternative approach to reducing the impact of the GIL is to make the GIL a per-interpreter-state lock rather than truly global. This was first implemented in Python 3.12 and is available in the C API. A Python interface to it is expected in Python 3.13. The main limitation to it at the moment is likely to be 3rd party extension modules, since these must be written with multiple interpreters in mind in order to be usable, so many older extension modules will not be usable.
Input and Output¶
How do I delete a file? (And other file questions…)¶
Use os.remove(filename)
or os.unlink(filename)
; for documentation, see
the os
module. The two functions are identical; unlink()
is simply
the name of the Unix system call for this function.
To remove a directory, use os.rmdir()
; use os.mkdir()
to create one.
os.makedirs(path)
will create any intermediate directories in path
that
don’t exist. os.removedirs(path)
will remove intermediate directories as
long as they’re empty; if you want to delete an entire directory tree and its
contents, use shutil.rmtree()
.
To rename a file, use os.rename(old_path, new_path)
.
To truncate a file, open it using f = open(filename, "rb+")
, and use
f.truncate(offset)
; offset defaults to the current seek position. There’s
also os.ftruncate(fd, offset)
for files opened with os.open()
, where
fd is the file descriptor (a small integer).
The shutil
module also contains a number of functions to work on files
including copyfile()
, copytree()
, and
rmtree()
.
How do I copy a file?¶
The shutil
module contains a copyfile()
function.
Note that on Windows NTFS volumes, it does not copy
alternate data streams
nor resource forks
on macOS HFS+ volumes, though both are now rarely used.
It also doesn’t copy file permissions and metadata, though using
shutil.copy2()
instead will preserve most (though not all) of it.
How do I read (or write) binary data?¶
To read or write complex binary data formats, it’s best to use the struct
module. It allows you to take a string containing binary data (usually numbers)
and convert it to Python objects; and vice versa.
For example, the following code reads two 2-byte integers and one 4-byte integer in big-endian format from a file:
import struct
with open(filename, "rb") as f:
s = f.read(8)
x, y, z = struct.unpack(">hhl", s)
The ‘>’ in the format string forces big-endian data; the letter ‘h’ reads one “short integer” (2 bytes), and ‘l’ reads one “long integer” (4 bytes) from the string.
For data that is more regular (e.g. a homogeneous list of ints or floats),
you can also use the array
module.
I can’t seem to use os.read() on a pipe created with os.popen(); why?¶
os.read()
is a low-level function which takes a file descriptor, a small
integer representing the opened file. os.popen()
creates a high-level
file object, the same type returned by the built-in open()
function.
Thus, to read n bytes from a pipe p created with os.popen()
, you need to
use p.read(n)
.
How do I access the serial (RS232) port?¶
For Win32, OSX, Linux, BSD, Jython, IronPython:
For Unix, see a Usenet post by Mitch Chapman:
Why doesn’t closing sys.stdout (stdin, stderr) really close it?¶
Python file objects are a high-level layer of abstraction on low-level C file descriptors.
For most file objects you create in Python via the built-in open()
function, f.close()
marks the Python file object as being closed from
Python’s point of view, and also arranges to close the underlying C file
descriptor. This also happens automatically in f
’s destructor, when
f
becomes garbage.
But stdin, stdout and stderr are treated specially by Python, because of the
special status also given to them by C. Running sys.stdout.close()
marks
the Python-level file object as being closed, but does not close the
associated C file descriptor.
To close the underlying C file descriptor for one of these three, you should
first be sure that’s what you really want to do (e.g., you may confuse
extension modules trying to do I/O). If it is, use os.close()
:
os.close(stdin.fileno())
os.close(stdout.fileno())
os.close(stderr.fileno())
Or you can use the numeric constants 0, 1 and 2, respectively.
Network/Internet Programming¶
What WWW tools are there for Python?¶
See the chapters titled Internet Protocols and Support and Internet Data Handling in the Library Reference Manual. Python has many modules that will help you build server-side and client-side web systems.
A summary of available frameworks is maintained by Paul Boddie at https://wiki.python.org/moin/WebProgramming.
What module should I use to help with generating HTML?¶
You can find a collection of useful links on the Web Programming wiki page.
How do I send mail from a Python script?¶
Use the standard library module smtplib
.
Here’s a very simple interactive mail sender that uses it. This method will work on any host that supports an SMTP listener.
import sys, smtplib
fromaddr = input("From: ")
toaddrs = input("To: ").split(',')
print("Enter message, end with ^D:")
msg = ''
while True:
line = sys.stdin.readline()
if not line:
break
msg += line
# The actual mail send
server = smtplib.SMTP('localhost')
server.sendmail(fromaddr, toaddrs, msg)
server.quit()
A Unix-only alternative uses sendmail. The location of the sendmail program
varies between systems; sometimes it is /usr/lib/sendmail
, sometimes
/usr/sbin/sendmail
. The sendmail manual page will help you out. Here’s
some sample code:
import os
SENDMAIL = "/usr/sbin/sendmail" # sendmail location
p = os.popen("%s -t -i" % SENDMAIL, "w")
p.write("To: receiver@example.com\n")
p.write("Subject: test\n")
p.write("\n") # blank line separating headers from body
p.write("Some text\n")
p.write("some more text\n")
sts = p.close()
if sts != 0:
print("Sendmail exit status", sts)
How do I avoid blocking in the connect() method of a socket?¶
The select
module is commonly used to help with asynchronous I/O on
sockets.
To prevent the TCP connect from blocking, you can set the socket to non-blocking
mode. Then when you do the connect()
,
you will either connect immediately
(unlikely) or get an exception that contains the error number as .errno
.
errno.EINPROGRESS
indicates that the connection is in progress, but hasn’t
finished yet. Different OSes will return different values, so you’re going to
have to check what’s returned on your system.
You can use the connect_ex()
method
to avoid creating an exception.
It will just return the errno value.
To poll, you can call connect_ex()
again later
– 0
or errno.EISCONN
indicate that you’re connected – or you can pass this
socket to select.select()
to check if it’s writable.
Databases¶
Are there any interfaces to database packages in Python?¶
Yes.
Interfaces to disk-based hashes such as DBM
and GDBM
are also included with standard Python. There is also the
sqlite3
module, which provides a lightweight disk-based relational
database.
Support for most relational databases is available. See the DatabaseProgramming wiki page for details.
How do you implement persistent objects in Python?¶
The pickle
library module solves this in a very general way (though you
still can’t store things like open files, sockets or windows), and the
shelve
library module uses pickle and (g)dbm to create persistent
mappings containing arbitrary Python objects.
Mathematics and Numerics¶
How do I generate random numbers in Python?¶
The standard module random
implements a random number generator. Usage
is simple:
import random
random.random()
This returns a random floating-point number in the range [0, 1).
There are also many other specialized generators in this module, such as:
randrange(a, b)
chooses an integer in the range [a, b).uniform(a, b)
chooses a floating-point number in the range [a, b).normalvariate(mean, sdev)
samples the normal (Gaussian) distribution.
Some higher-level functions operate on sequences directly, such as:
choice(S)
chooses a random element from a given sequence.shuffle(L)
shuffles a list in-place, i.e. permutes it randomly.
There’s also a Random
class you can instantiate to create independent
multiple random number generators.


Design and History FAQ¶
Why does Python use indentation for grouping of statements?¶
Guido van Rossum believes that using indentation for grouping is extremely elegant and contributes a lot to the clarity of the average Python program. Most people learn to love this feature after a while.
Since there are no begin/end brackets there cannot be a disagreement between grouping perceived by the parser and the human reader. Occasionally C programmers will encounter a fragment of code like this:
if (x <= y)
x++;
y--;
z++;
Only the x++
statement is executed if the condition is true, but the
indentation leads many to believe otherwise. Even experienced C programmers will
sometimes stare at it a long time wondering as to why y
is being decremented even
for x > y
.
Because there are no begin/end brackets, Python is much less prone to coding-style conflicts. In C there are many different ways to place the braces. After becoming used to reading and writing code using a particular style, it is normal to feel somewhat uneasy when reading (or being required to write) in a different one.
Many coding styles place begin/end brackets on a line by themselves. This makes programs considerably longer and wastes valuable screen space, making it harder to get a good overview of a program. Ideally, a function should fit on one screen (say, 20–30 lines). 20 lines of Python can do a lot more work than 20 lines of C. This is not solely due to the lack of begin/end brackets – the lack of declarations and the high-level data types are also responsible – but the indentation-based syntax certainly helps.
Why am I getting strange results with simple arithmetic operations?¶
See the next question.
Why are floating-point calculations so inaccurate?¶
Users are often surprised by results like this:
>>> 1.2 - 1.0
0.19999999999999996
and think it is a bug in Python. It’s not. This has little to do with Python, and much more to do with how the underlying platform handles floating-point numbers.
The float
type in CPython uses a C double
for storage. A
float
object’s value is stored in binary floating-point with a fixed
precision (typically 53 bits) and Python uses C operations, which in turn rely
on the hardware implementation in the processor, to perform floating-point
operations. This means that as far as floating-point operations are concerned,
Python behaves like many popular languages including C and Java.
Many numbers that can be written easily in decimal notation cannot be expressed exactly in binary floating point. For example, after:
>>> x = 1.2
the value stored for x
is a (very good) approximation to the decimal value
1.2
, but is not exactly equal to it. On a typical machine, the actual
stored value is:
1.0011001100110011001100110011001100110011001100110011 (binary)
which is exactly:
1.1999999999999999555910790149937383830547332763671875 (decimal)
The typical precision of 53 bits provides Python floats with 15–16 decimal digits of accuracy.
For a fuller explanation, please see the floating-point arithmetic chapter in the Python tutorial.
Why are Python strings immutable?¶
There are several advantages.
One is performance: knowing that a string is immutable means we can allocate space for it at creation time, and the storage requirements are fixed and unchanging. This is also one of the reasons for the distinction between tuples and lists.
Another advantage is that strings in Python are considered as “elemental” as numbers. No amount of activity will change the value 8 to anything else, and in Python, no amount of activity will change the string “eight” to anything else.
Why must ‘self’ be used explicitly in method definitions and calls?¶
The idea was borrowed from Modula-3. It turns out to be very useful, for a variety of reasons.
First, it’s more obvious that you are using a method or instance attribute
instead of a local variable. Reading self.x
or self.meth()
makes it
absolutely clear that an instance variable or method is used even if you don’t
know the class definition by heart. In C++, you can sort of tell by the lack of
a local variable declaration (assuming globals are rare or easily recognizable)
– but in Python, there are no local variable declarations, so you’d have to
look up the class definition to be sure. Some C++ and Java coding standards
call for instance attributes to have an m_
prefix, so this explicitness is
still useful in those languages, too.
Second, it means that no special syntax is necessary if you want to explicitly
reference or call the method from a particular class. In C++, if you want to
use a method from a base class which is overridden in a derived class, you have
to use the ::
operator – in Python you can write
baseclass.methodname(self, <argument list>)
. This is particularly useful
for __init__()
methods, and in general in cases where a derived class
method wants to extend the base class method of the same name and thus has to
call the base class method somehow.
Finally, for instance variables it solves a syntactic problem with assignment:
since local variables in Python are (by definition!) those variables to which a
value is assigned in a function body (and that aren’t explicitly declared
global), there has to be some way to tell the interpreter that an assignment was
meant to assign to an instance variable instead of to a local variable, and it
should preferably be syntactic (for efficiency reasons). C++ does this through
declarations, but Python doesn’t have declarations and it would be a pity having
to introduce them just for this purpose. Using the explicit self.var
solves
this nicely. Similarly, for using instance variables, having to write
self.var
means that references to unqualified names inside a method don’t
have to search the instance’s directories. To put it another way, local
variables and instance variables live in two different namespaces, and you need
to tell Python which namespace to use.
Why can’t I use an assignment in an expression?¶
Starting in Python 3.8, you can!
Assignment expressions using the walrus operator :=
assign a variable in an
expression:
while chunk := fp.read(200):
print(chunk)
See PEP 572 for more information.
Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?¶
As Guido said:
(a) For some operations, prefix notation just reads better than postfix – prefix (and infix!) operations have a long tradition in mathematics which likes notations where the visuals help the mathematician thinking about a problem. Compare the easy with which we rewrite a formula like x*(a+b) into x*a + x*b to the clumsiness of doing the same thing using a raw OO notation.
(b) When I read code that says len(x) I know that it is asking for the length of something. This tells me two things: the result is an integer, and the argument is some kind of container. To the contrary, when I read x.len(), I have to already know that x is some kind of container implementing an interface or inheriting from a class that has a standard len(). Witness the confusion we occasionally have when a class that is not implementing a mapping has a get() or keys() method, or something that isn’t a file has a write() method.
—https://mail.python.org/pipermail/python-3000/2006-November/004643.html
Why is join() a string method instead of a list or tuple method?¶
Strings became much more like other standard types starting in Python 1.6, when methods were added which give the same functionality that has always been available using the functions of the string module. Most of these new methods have been widely accepted, but the one which appears to make some programmers feel uncomfortable is:
", ".join(['1', '2', '4', '8', '16'])
which gives the result:
"1, 2, 4, 8, 16"
There are two common arguments against this usage.
The first runs along the lines of: “It looks really ugly using a method of a string literal (string constant)”, to which the answer is that it might, but a string literal is just a fixed value. If the methods are to be allowed on names bound to strings there is no logical reason to make them unavailable on literals.
The second objection is typically cast as: “I am really telling a sequence to
join its members together with a string constant”. Sadly, you aren’t. For some
reason there seems to be much less difficulty with having split()
as
a string method, since in that case it is easy to see that
"1, 2, 4, 8, 16".split(", ")
is an instruction to a string literal to return the substrings delimited by the given separator (or, by default, arbitrary runs of white space).
join()
is a string method because in using it you are telling the
separator string to iterate over a sequence of strings and insert itself between
adjacent elements. This method can be used with any argument which obeys the
rules for sequence objects, including any new classes you might define yourself.
Similar methods exist for bytes and bytearray objects.
How fast are exceptions?¶
A try
/except
block is extremely efficient if no exceptions
are raised. Actually
catching an exception is expensive. In versions of Python prior to 2.0 it was
common to use this idiom:
try:
value = mydict[key]
except KeyError:
mydict[key] = getvalue(key)
value = mydict[key]
This only made sense when you expected the dict to have the key almost all the time. If that wasn’t the case, you coded it like this:
if key in mydict:
value = mydict[key]
else:
value = mydict[key] = getvalue(key)
For this specific case, you could also use value = dict.setdefault(key,
getvalue(key))
, but only if the getvalue()
call is cheap enough because it
is evaluated in all cases.
Why isn’t there a switch or case statement in Python?¶
In general, structured switch statements execute one block of code
when an expression has a particular value or set of values.
Since Python 3.10 one can easily match literal values, or constants
within a namespace, with a match ... case
statement.
An older alternative is a sequence of if... elif... elif... else
.
For cases where you need to choose from a very large number of possibilities, you can create a dictionary mapping case values to functions to call. For example:
functions = {'a': function_1,
'b': function_2,
'c': self.method_1}
func = functions[value]
func()
For calling methods on objects, you can simplify yet further by using the
getattr()
built-in to retrieve methods with a particular name:
class MyVisitor:
def visit_a(self):
...
def dispatch(self, value):
method_name = 'visit_' + str(value)
method = getattr(self, method_name)
method()
It’s suggested that you use a prefix for the method names, such as visit_
in
this example. Without such a prefix, if values are coming from an untrusted
source, an attacker would be able to call any method on your object.
Imitating switch with fallthrough, as with C’s switch-case-default, is possible, much harder, and less needed.
Can’t you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?¶
Answer 1: Unfortunately, the interpreter pushes at least one C stack frame for each Python stack frame. Also, extensions can call back into Python at almost random moments. Therefore, a complete threads implementation requires thread support for C.
Answer 2: Fortunately, there is Stackless Python, which has a completely redesigned interpreter loop that avoids the C stack.
Why can’t lambda expressions contain statements?¶
Python lambda expressions cannot contain statements because Python’s syntactic framework can’t handle statements nested inside expressions. However, in Python, this is not a serious problem. Unlike lambda forms in other languages, where they add functionality, Python lambdas are only a shorthand notation if you’re too lazy to define a function.
Functions are already first class objects in Python, and can be declared in a local scope. Therefore the only advantage of using a lambda instead of a locally defined function is that you don’t need to invent a name for the function – but that’s just a local variable to which the function object (which is exactly the same type of object that a lambda expression yields) is assigned!
Can Python be compiled to machine code, C or some other language?¶
Cython compiles a modified version of Python with optional annotations into C extensions. Nuitka is an up-and-coming compiler of Python into C++ code, aiming to support the full Python language.
How does Python manage memory?¶
The details of Python memory management depend on the implementation. The
standard implementation of Python, CPython, uses reference counting to
detect inaccessible objects, and another mechanism to collect reference cycles,
periodically executing a cycle detection algorithm which looks for inaccessible
cycles and deletes the objects involved. The gc
module provides functions
to perform a garbage collection, obtain debugging statistics, and tune the
collector’s parameters.
Other implementations (such as Jython or PyPy), however, can rely on a different mechanism such as a full-blown garbage collector. This difference can cause some subtle porting problems if your Python code depends on the behavior of the reference counting implementation.
In some Python implementations, the following code (which is fine in CPython) will probably run out of file descriptors:
for file in very_long_list_of_files:
f = open(file)
c = f.read(1)
Indeed, using CPython’s reference counting and destructor scheme, each new
assignment to f
closes the previous file. With a traditional GC, however,
those file objects will only get collected (and closed) at varying and possibly
long intervals.
If you want to write code that will work with any Python implementation,
you should explicitly close the file or use the with
statement;
this will work regardless of memory management scheme:
for file in very_long_list_of_files:
with open(file) as f:
c = f.read(1)
Why doesn’t CPython use a more traditional garbage collection scheme?¶
For one thing, this is not a C standard feature and hence it’s not portable. (Yes, we know about the Boehm GC library. It has bits of assembler code for most common platforms, not for all of them, and although it is mostly transparent, it isn’t completely transparent; patches are required to get Python to work with it.)
Traditional GC also becomes a problem when Python is embedded into other
applications. While in a standalone Python it’s fine to replace the standard
malloc()
and free()
with versions provided by the GC library, an application
embedding Python may want to have its own substitute for malloc()
and free()
,
and may not want Python’s. Right now, CPython works with anything that
implements malloc()
and free()
properly.
Why isn’t all memory freed when CPython exits?¶
Objects referenced from the global namespaces of Python modules are not always deallocated when Python exits. This may happen if there are circular references. There are also certain bits of memory that are allocated by the C library that are impossible to free (e.g. a tool like Purify will complain about these). Python is, however, aggressive about cleaning up memory on exit and does try to destroy every single object.
If you want to force Python to delete certain things on deallocation use the
atexit
module to run a function that will force those deletions.
Why are there separate tuple and list data types?¶
Lists and tuples, while similar in many respects, are generally used in
fundamentally different ways. Tuples can be thought of as being similar to
Pascal records
or C structs
; they’re small collections of related data which may
be of different types which are operated on as a group. For example, a
Cartesian coordinate is appropriately represented as a tuple of two or three
numbers.
Lists, on the other hand, are more like arrays in other languages. They tend to
hold a varying number of objects all of which have the same type and which are
operated on one-by-one. For example, os.listdir('.')
returns a list of
strings representing the files in the current directory. Functions which
operate on this output would generally not break if you added another file or
two to the directory.
Tuples are immutable, meaning that once a tuple has been created, you can’t replace any of its elements with a new value. Lists are mutable, meaning that you can always change a list’s elements. Only immutable elements can be used as dictionary keys, and hence only tuples and not lists can be used as keys.
How are lists implemented in CPython?¶
CPython’s lists are really variable-length arrays, not Lisp-style linked lists. The implementation uses a contiguous array of references to other objects, and keeps a pointer to this array and the array’s length in a list head structure.
This makes indexing a list a[i]
an operation whose cost is independent of
the size of the list or the value of the index.
When items are appended or inserted, the array of references is resized. Some cleverness is applied to improve the performance of appending items repeatedly; when the array must be grown, some extra space is allocated so the next few times don’t require an actual resize.
How are dictionaries implemented in CPython?¶
CPython’s dictionaries are implemented as resizable hash tables. Compared to B-trees, this gives better performance for lookup (the most common operation by far) under most circumstances, and the implementation is simpler.
Dictionaries work by computing a hash code for each key stored in the dictionary
using the hash()
built-in function. The hash code varies widely depending
on the key and a per-process seed; for example, 'Python'
could hash to
-539294296
while 'python'
, a string that differs by a single bit, could hash
to 1142331976
. The hash code is then used to calculate a location in an
internal array where the value will be stored. Assuming that you’re storing
keys that all have different hash values, this means that dictionaries take
constant time – O(1), in Big-O notation – to retrieve a key.
Why must dictionary keys be immutable?¶
The hash table implementation of dictionaries uses a hash value calculated from the key value to find the key. If the key were a mutable object, its value could change, and thus its hash could also change. But since whoever changes the key object can’t tell that it was being used as a dictionary key, it can’t move the entry around in the dictionary. Then, when you try to look up the same object in the dictionary it won’t be found because its hash value is different. If you tried to look up the old value it wouldn’t be found either, because the value of the object found in that hash bin would be different.
If you want a dictionary indexed with a list, simply convert the list to a tuple
first; the function tuple(L)
creates a tuple with the same entries as the
list L
. Tuples are immutable and can therefore be used as dictionary keys.
Some unacceptable solutions that have been proposed:
Hash lists by their address (object ID). This doesn’t work because if you construct a new list with the same value it won’t be found; e.g.:
mydict = {[1, 2]: '12'} print(mydict[[1, 2]])
would raise a
KeyError
exception because the id of the[1, 2]
used in the second line differs from that in the first line. In other words, dictionary keys should be compared using==
, not usingis
.Make a copy when using a list as a key. This doesn’t work because the list, being a mutable object, could contain a reference to itself, and then the copying code would run into an infinite loop.
Allow lists as keys but tell the user not to modify them. This would allow a class of hard-to-track bugs in programs when you forgot or modified a list by accident. It also invalidates an important invariant of dictionaries: every value in
d.keys()
is usable as a key of the dictionary.Mark lists as read-only once they are used as a dictionary key. The problem is that it’s not just the top-level object that could change its value; you could use a tuple containing a list as a key. Entering anything as a key into a dictionary would require marking all objects reachable from there as read-only – and again, self-referential objects could cause an infinite loop.
There is a trick to get around this if you need to, but use it at your own risk:
You can wrap a mutable structure inside a class instance which has both a
__eq__()
and a __hash__()
method.
You must then make sure that the
hash value for all such wrapper objects that reside in a dictionary (or other
hash based structure), remain fixed while the object is in the dictionary (or
other structure).
class ListWrapper:
def __init__(self, the_list):
self.the_list = the_list
def __eq__(self, other):
return self.the_list == other.the_list
def __hash__(self):
l = self.the_list
result = 98767 - len(l)*555
for i, el in enumerate(l):
try:
result = result + (hash(el) % 9999999) * 1001 + i
except Exception:
result = (result % 7777777) + i * 333
return result
Note that the hash computation is complicated by the possibility that some members of the list may be unhashable and also by the possibility of arithmetic overflow.
Furthermore it must always be the case that if o1 == o2
(ie o1.__eq__(o2)
is True
) then hash(o1) == hash(o2)
(ie, o1.__hash__() == o2.__hash__()
),
regardless of whether the object is in a dictionary or not. If you fail to meet
these restrictions dictionaries and other hash based structures will misbehave.
In the case of ListWrapper
, whenever the wrapper object is in a dictionary the
wrapped list must not change to avoid anomalies. Don’t do this unless you are
prepared to think hard about the requirements and the consequences of not
meeting them correctly. Consider yourself warned.
Why doesn’t list.sort() return the sorted list?¶
In situations where performance matters, making a copy of the list just to sort
it would be wasteful. Therefore, list.sort()
sorts the list in place. In
order to remind you of that fact, it does not return the sorted list. This way,
you won’t be fooled into accidentally overwriting a list when you need a sorted
copy but also need to keep the unsorted version around.
If you want to return a new list, use the built-in sorted()
function
instead. This function creates a new list from a provided iterable, sorts
it and returns it. For example, here’s how to iterate over the keys of a
dictionary in sorted order:
for key in sorted(mydict):
... # do whatever with mydict[key]...
How do you specify and enforce an interface spec in Python?¶
An interface specification for a module as provided by languages such as C++ and Java describes the prototypes for the methods and functions of the module. Many feel that compile-time enforcement of interface specifications helps in the construction of large programs.
Python 2.6 adds an abc
module that lets you define Abstract Base Classes
(ABCs). You can then use isinstance()
and issubclass()
to check
whether an instance or a class implements a particular ABC. The
collections.abc
module defines a set of useful ABCs such as
Iterable
, Container
, and
MutableMapping
.
For Python, many of the advantages of interface specifications can be obtained by an appropriate test discipline for components.
A good test suite for a module can both provide a regression test and serve as a
module interface specification and a set of examples. Many Python modules can
be run as a script to provide a simple “self test.” Even modules which use
complex external interfaces can often be tested in isolation using trivial
“stub” emulations of the external interface. The doctest
and
unittest
modules or third-party test frameworks can be used to construct
exhaustive test suites that exercise every line of code in a module.
An appropriate testing discipline can help build large complex applications in
Python as well as having interface specifications would. In fact, it can be
better because an interface specification cannot test certain properties of a
program. For example, the list.append()
method is expected to add new elements
to the end of some internal list; an interface specification cannot test that
your list.append()
implementation will actually do this correctly, but it’s
trivial to check this property in a test suite.
Writing test suites is very helpful, and you might want to design your code to make it easily tested. One increasingly popular technique, test-driven development, calls for writing parts of the test suite first, before you write any of the actual code. Of course Python allows you to be sloppy and not write test cases at all.
Why is there no goto?¶
In the 1970s people realized that unrestricted goto could lead
to messy “spaghetti” code that was hard to understand and revise.
In a high-level language, it is also unneeded as long as there
are ways to branch (in Python, with if
statements and or
,
and
, and if
/else
expressions) and loop (with while
and for
statements, possibly containing continue
and break
).
One can also use exceptions to provide a “structured goto”
that works even across
function calls. Many feel that exceptions can conveniently emulate all
reasonable uses of the go
or goto
constructs of C, Fortran, and other
languages. For example:
class label(Exception): pass # declare a label
try:
...
if condition: raise label() # goto label
...
except label: # where to goto
pass
...
This doesn’t allow you to jump into the middle of a loop, but that’s usually
considered an abuse of goto
anyway. Use sparingly.
Why can’t raw strings (r-strings) end with a backslash?¶
More precisely, they can’t end with an odd number of backslashes: the unpaired backslash at the end escapes the closing quote character, leaving an unterminated string.
Raw strings were designed to ease creating input for processors (chiefly regular expression engines) that want to do their own backslash escape processing. Such processors consider an unmatched trailing backslash to be an error anyway, so raw strings disallow that. In return, they allow you to pass on the string quote character by escaping it with a backslash. These rules work well when r-strings are used for their intended purpose.
If you’re trying to build Windows pathnames, note that all Windows system calls accept forward slashes too:
f = open("/mydir/file.txt") # works fine!
If you’re trying to build a pathname for a DOS command, try e.g. one of
dir = r"\this\is\my\dos\dir" "\\"
dir = r"\this\is\my\dos\dir\ "[:-1]
dir = "\\this\\is\\my\\dos\\dir\\"
Why doesn’t Python have a “with” statement for attribute assignments?¶
Python has a with
statement that wraps the execution of a block, calling code
on the entrance and exit from the block. Some languages have a construct that
looks like this:
with obj:
a = 1 # equivalent to obj.a = 1
total = total + 1 # obj.total = obj.total + 1
In Python, such a construct would be ambiguous.
Other languages, such as Object Pascal, Delphi, and C++, use static types, so it’s possible to know, in an unambiguous way, what member is being assigned to. This is the main point of static typing – the compiler always knows the scope of every variable at compile time.
Python uses dynamic types. It is impossible to know in advance which attribute will be referenced at runtime. Member attributes may be added or removed from objects on the fly. This makes it impossible to know, from a simple reading, what attribute is being referenced: a local one, a global one, or a member attribute?
For instance, take the following incomplete snippet:
def foo(a):
with a:
print(x)
The snippet assumes that a
must have a member attribute called x
. However,
there is nothing in Python that tells the interpreter this. What should happen
if a
is, let us say, an integer? If there is a global variable named x
,
will it be used inside the with
block? As you see, the dynamic nature of Python
makes such choices much harder.
The primary benefit of with
and similar language features (reduction of code
volume) can, however, easily be achieved in Python by assignment. Instead of:
function(args).mydict[index][index].a = 21
function(args).mydict[index][index].b = 42
function(args).mydict[index][index].c = 63
write this:
ref = function(args).mydict[index][index]
ref.a = 21
ref.b = 42
ref.c = 63
This also has the side-effect of increasing execution speed because name bindings are resolved at run-time in Python, and the second version only needs to perform the resolution once.
Similar proposals that would introduce syntax to further reduce code volume, such as using a ‘leading dot’, have been rejected in favour of explicitness (see https://mail.python.org/pipermail/python-ideas/2016-May/040070.html).
Why don’t generators support the with statement?¶
For technical reasons, a generator used directly as a context manager
would not work correctly. When, as is most common, a generator is used as
an iterator run to completion, no closing is needed. When it is, wrap
it as contextlib.closing(generator)
in the with
statement.
Why are colons required for the if/while/def/class statements?¶
The colon is required primarily to enhance readability (one of the results of the experimental ABC language). Consider this:
if a == b
print(a)
versus
if a == b:
print(a)
Notice how the second one is slightly easier to read. Notice further how a colon sets off the example in this FAQ answer; it’s a standard usage in English.
Another minor reason is that the colon makes it easier for editors with syntax highlighting; they can look for colons to decide when indentation needs to be increased instead of having to do a more elaborate parsing of the program text.
Why does Python allow commas at the end of lists and tuples?¶
Python lets you add a trailing comma at the end of lists, tuples, and dictionaries:
[1, 2, 3,]
('a', 'b', 'c',)
d = {
"A": [1, 5],
"B": [6, 7], # last trailing comma is optional but good style
}
There are several reasons to allow this.
When you have a literal value for a list, tuple, or dictionary spread across multiple lines, it’s easier to add more elements because you don’t have to remember to add a comma to the previous line. The lines can also be reordered without creating a syntax error.
Accidentally omitting the comma can lead to errors that are hard to diagnose. For example:
x = [
"fee",
"fie"
"foo",
"fum"
]
This list looks like it has four elements, but it actually contains three: “fee”, “fiefoo” and “fum”. Always adding the comma avoids this source of error.
Allowing the trailing comma may also make programmatic code generation easier.