Modules

6. Modules¶
If you quit from the Python interpreter and enter it again, the definitions you have made (functions and variables) are lost. Therefore, if you want to write a somewhat longer program, you are better off using a text editor to prepare the input for the interpreter and running it with that file as input instead. This is known as creating a script. As your program gets longer, you may want to split it into several files for easier maintenance. You may also want to use a handy function that you’ve written in several programs without copying its definition into each program.
To support this, Python has a way to put definitions in a file and use them in a script or in an interactive instance of the interpreter. Such a file is called a module; definitions from a module can be imported into other modules or into the main module (the collection of variables that you have access to in a script executed at the top level and in calculator mode).
A module is a file containing Python definitions and statements. The file name
is the module name with the suffix .py
appended. Within a module, the
module’s name (as a string) is available as the value of the global variable
__name__
. For instance, use your favorite text editor to create a file
called fibo.py
in the current directory with the following contents:
# Fibonacci numbers module
def fib(n):
"""Write Fibonacci series up to n."""
a, b = 0, 1
while a < n:
print(a, end=' ')
a, b = b, a+b
print()
def fib2(n):
"""Return Fibonacci series up to n."""
result = []
a, b = 0, 1
while a < n:
result.append(a)
a, b = b, a+b
return result
Now enter the Python interpreter and import this module with the following command:
>>> import fibo
This does not add the names of the functions defined in fibo
directly to
the current namespace (see Python Scopes and Namespaces for more details);
it only adds the module name fibo
there. Using
the module name you can access the functions:
>>> fibo.fib(1000)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
>>> fibo.fib2(100)
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
>>> fibo.__name__
'fibo'
If you intend to use a function often you can assign it to a local name:
>>> fib = fibo.fib
>>> fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
6.1. More on Modules¶
A module can contain executable statements as well as function definitions. These statements are intended to initialize the module. They are executed only the first time the module name is encountered in an import statement. [1] (They are also run if the file is executed as a script.)
Each module has its own private namespace, which is used as the global namespace
by all functions defined in the module. Thus, the author of a module can
use global variables in the module without worrying about accidental clashes
with a user’s global variables. On the other hand, if you know what you are
doing you can touch a module’s global variables with the same notation used to
refer to its functions, modname.itemname
.
Modules can import other modules. It is customary but not required to place all
import
statements at the beginning of a module (or script, for that
matter). The imported module names, if placed at the top level of a module
(outside any functions or classes), are added to the module’s global namespace.
There is a variant of the import
statement that imports names from a
module directly into the importing module’s namespace. For example:
>>> from fibo import fib, fib2
>>> fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
This does not introduce the module name from which the imports are taken in the
local namespace (so in the example, fibo
is not defined).
There is even a variant to import all names that a module defines:
>>> from fibo import *
>>> fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
This imports all names except those beginning with an underscore (_
).
In most cases Python programmers do not use this facility since it introduces
an unknown set of names into the interpreter, possibly hiding some things
you have already defined.
Note that in general the practice of importing *
from a module or package is
frowned upon, since it often causes poorly readable code. However, it is okay to
use it to save typing in interactive sessions.
If the module name is followed by as
, then the name
following as
is bound directly to the imported module.
>>> import fibo as fib
>>> fib.fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
This is effectively importing the module in the same way that import fibo
will do, with the only difference of it being available as fib
.
It can also be used when utilising from
with similar effects:
>>> from fibo import fib as fibonacci
>>> fibonacci(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377
Note
For efficiency reasons, each module is only imported once per interpreter
session. Therefore, if you change your modules, you must restart the
interpreter – or, if it’s just one module you want to test interactively,
use importlib.reload()
, e.g. import importlib;
importlib.reload(modulename)
.
6.1.1. Executing modules as scripts¶
When you run a Python module with
python fibo.py <arguments>
the code in the module will be executed, just as if you imported it, but with
the __name__
set to "__main__"
. That means that by adding this code at
the end of your module:
if __name__ == "__main__":
import sys
fib(int(sys.argv[1]))
you can make the file usable as a script as well as an importable module, because the code that parses the command line only runs if the module is executed as the “main” file:
$ python fibo.py 50
0 1 1 2 3 5 8 13 21 34
If the module is imported, the code is not run:
>>> import fibo
>>>
This is often used either to provide a convenient user interface to a module, or for testing purposes (running the module as a script executes a test suite).
6.1.2. The Module Search Path¶
When a module named spam
is imported, the interpreter first searches for
a built-in module with that name. These module names are listed in
sys.builtin_module_names
. If not found, it then searches for a file
named spam.py
in a list of directories given by the variable
sys.path
. sys.path
is initialized from these locations:
The directory containing the input script (or the current directory when no file is specified).
PYTHONPATH
(a list of directory names, with the same syntax as the shell variablePATH
).The installation-dependent default (by convention including a
site-packages
directory, handled by thesite
module).
More details are at The initialization of the sys.path module search path.
Note
On file systems which support symlinks, the directory containing the input script is calculated after the symlink is followed. In other words the directory containing the symlink is not added to the module search path.
After initialization, Python programs can modify sys.path
. The
directory containing the script being run is placed at the beginning of the
search path, ahead of the standard library path. This means that scripts in that
directory will be loaded instead of modules of the same name in the library
directory. This is an error unless the replacement is intended. See section
Standard Modules for more information.
6.1.3. “Compiled” Python files¶
To speed up loading modules, Python caches the compiled version of each module
in the __pycache__
directory under the name module.version.pyc
,
where the version encodes the format of the compiled file; it generally contains
the Python version number. For example, in CPython release 3.3 the compiled
version of spam.py would be cached as __pycache__/spam.cpython-33.pyc
. This
naming convention allows compiled modules from different releases and different
versions of Python to coexist.
Python checks the modification date of the source against the compiled version to see if it’s out of date and needs to be recompiled. This is a completely automatic process. Also, the compiled modules are platform-independent, so the same library can be shared among systems with different architectures.
Python does not check the cache in two circumstances. First, it always recompiles and does not store the result for the module that’s loaded directly from the command line. Second, it does not check the cache if there is no source module. To support a non-source (compiled only) distribution, the compiled module must be in the source directory, and there must not be a source module.
Some tips for experts:
You can use the
-O
or-OO
switches on the Python command to reduce the size of a compiled module. The-O
switch removes assert statements, the-OO
switch removes both assert statements and __doc__ strings. Since some programs may rely on having these available, you should only use this option if you know what you’re doing. “Optimized” modules have anopt-
tag and are usually smaller. Future releases may change the effects of optimization.A program doesn’t run any faster when it is read from a
.pyc
file than when it is read from a.py
file; the only thing that’s faster about.pyc
files is the speed with which they are loaded.The module
compileall
can create .pyc files for all modules in a directory.There is more detail on this process, including a flow chart of the decisions, in PEP 3147.
6.2. Standard Modules¶
Python comes with a library of standard modules, described in a separate
document, the Python Library Reference (“Library Reference” hereafter). Some
modules are built into the interpreter; these provide access to operations that
are not part of the core of the language but are nevertheless built in, either
for efficiency or to provide access to operating system primitives such as
system calls. The set of such modules is a configuration option which also
depends on the underlying platform. For example, the winreg
module is only
provided on Windows systems. One particular module deserves some attention:
sys
, which is built into every Python interpreter. The variables
sys.ps1
and sys.ps2
define the strings used as primary and secondary
prompts:
>>> import sys
>>> sys.ps1
'>>> '
>>> sys.ps2
'... '
>>> sys.ps1 = 'C> '
C> print('Yuck!')
Yuck!
C>
These two variables are only defined if the interpreter is in interactive mode.
The variable sys.path
is a list of strings that determines the interpreter’s
search path for modules. It is initialized to a default path taken from the
environment variable PYTHONPATH
, or from a built-in default if
PYTHONPATH
is not set. You can modify it using standard list
operations:
>>> import sys
>>> sys.path.append('/ufs/guido/lib/python')
6.3. The dir()
Function¶
The built-in function dir()
is used to find out which names a module
defines. It returns a sorted list of strings:
>>> import fibo, sys
>>> dir(fibo)
['__name__', 'fib', 'fib2']
>>> dir(sys)
['__breakpointhook__', '__displayhook__', '__doc__', '__excepthook__',
'__interactivehook__', '__loader__', '__name__', '__package__', '__spec__',
'__stderr__', '__stdin__', '__stdout__', '__unraisablehook__',
'_clear_type_cache', '_current_frames', '_debugmallocstats', '_framework',
'_getframe', '_git', '_home', '_xoptions', 'abiflags', 'addaudithook',
'api_version', 'argv', 'audit', 'base_exec_prefix', 'base_prefix',
'breakpointhook', 'builtin_module_names', 'byteorder', 'call_tracing',
'callstats', 'copyright', 'displayhook', 'dont_write_bytecode', 'exc_info',
'excepthook', 'exec_prefix', 'executable', 'exit', 'flags', 'float_info',
'float_repr_style', 'get_asyncgen_hooks', 'get_coroutine_origin_tracking_depth',
'getallocatedblocks', 'getdefaultencoding', 'getdlopenflags',
'getfilesystemencodeerrors', 'getfilesystemencoding', 'getprofile',
'getrecursionlimit', 'getrefcount', 'getsizeof', 'getswitchinterval',
'gettrace', 'hash_info', 'hexversion', 'implementation', 'int_info',
'intern', 'is_finalizing', 'last_traceback', 'last_type', 'last_value',
'maxsize', 'maxunicode', 'meta_path', 'modules', 'path', 'path_hooks',
'path_importer_cache', 'platform', 'prefix', 'ps1', 'ps2', 'pycache_prefix',
'set_asyncgen_hooks', 'set_coroutine_origin_tracking_depth', 'setdlopenflags',
'setprofile', 'setrecursionlimit', 'setswitchinterval', 'settrace', 'stderr',
'stdin', 'stdout', 'thread_info', 'unraisablehook', 'version', 'version_info',
'warnoptions']
Without arguments, dir()
lists the names you have defined currently:
>>> a = [1, 2, 3, 4, 5]
>>> import fibo
>>> fib = fibo.fib
>>> dir()
['__builtins__', '__name__', 'a', 'fib', 'fibo', 'sys']
Note that it lists all types of names: variables, modules, functions, etc.
dir()
does not list the names of built-in functions and variables. If you
want a list of those, they are defined in the standard module
builtins
:
>>> import builtins
>>> dir(builtins)
['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException',
'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning',
'ChildProcessError', 'ConnectionAbortedError', 'ConnectionError',
'ConnectionRefusedError', 'ConnectionResetError', 'DeprecationWarning',
'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False',
'FileExistsError', 'FileNotFoundError', 'FloatingPointError',
'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError',
'ImportWarning', 'IndentationError', 'IndexError', 'InterruptedError',
'IsADirectoryError', 'KeyError', 'KeyboardInterrupt', 'LookupError',
'MemoryError', 'NameError', 'None', 'NotADirectoryError', 'NotImplemented',
'NotImplementedError', 'OSError', 'OverflowError',
'PendingDeprecationWarning', 'PermissionError', 'ProcessLookupError',
'ReferenceError', 'ResourceWarning', 'RuntimeError', 'RuntimeWarning',
'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError',
'SystemExit', 'TabError', 'TimeoutError', 'True', 'TypeError',
'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError',
'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning',
'ValueError', 'Warning', 'ZeroDivisionError', '_', '__build_class__',
'__debug__', '__doc__', '__import__', '__name__', '__package__', 'abs',
'all', 'any', 'ascii', 'bin', 'bool', 'bytearray', 'bytes', 'callable',
'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits',
'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit',
'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr',
'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass',
'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview',
'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property',
'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice',
'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars',
'zip']
6.4. Packages¶
Packages are a way of structuring Python’s module namespace by using “dotted
module names”. For example, the module name A.B
designates a submodule
named B
in a package named A
. Just like the use of modules saves the
authors of different modules from having to worry about each other’s global
variable names, the use of dotted module names saves the authors of multi-module
packages like NumPy or Pillow from having to worry about
each other’s module names.
Suppose you want to design a collection of modules (a “package”) for the uniform
handling of sound files and sound data. There are many different sound file
formats (usually recognized by their extension, for example: .wav
,
.aiff
, .au
), so you may need to create and maintain a growing
collection of modules for the conversion between the various file formats.
There are also many different operations you might want to perform on sound data
(such as mixing, adding echo, applying an equalizer function, creating an
artificial stereo effect), so in addition you will be writing a never-ending
stream of modules to perform these operations. Here’s a possible structure for
your package (expressed in terms of a hierarchical filesystem):
sound/ Top-level package
__init__.py Initialize the sound package
formats/ Subpackage for file format conversions
__init__.py
wavread.py
wavwrite.py
aiffread.py
aiffwrite.py
auread.py
auwrite.py
...
effects/ Subpackage for sound effects
__init__.py
echo.py
surround.py
reverse.py
...
filters/ Subpackage for filters
__init__.py
equalizer.py
vocoder.py
karaoke.py
...
When importing the package, Python searches through the directories on
sys.path
looking for the package subdirectory.
The __init__.py
files are required to make Python treat directories
containing the file as packages (unless using a namespace package, a
relatively advanced feature). This prevents directories with a common name,
such as string
, from unintentionally hiding valid modules that occur later
on the module search path. In the simplest case, __init__.py
can just be
an empty file, but it can also execute initialization code for the package or
set the __all__
variable, described later.
Users of the package can import individual modules from the package, for example:
import sound.effects.echo
This loads the submodule sound.effects.echo
. It must be referenced with
its full name.
sound.effects.echo.echofilter(input, output, delay=0.7, atten=4)
An alternative way of importing the submodule is:
from sound.effects import echo
This also loads the submodule echo
, and makes it available without its
package prefix, so it can be used as follows:
echo.echofilter(input, output, delay=0.7, atten=4)
Yet another variation is to import the desired function or variable directly:
from sound.effects.echo import echofilter
Again, this loads the submodule echo
, but this makes its function
echofilter()
directly available:
echofilter(input, output, delay=0.7, atten=4)
Note that when using from package import item
, the item can be either a
submodule (or subpackage) of the package, or some other name defined in the
package, like a function, class or variable. The import
statement first
tests whether the item is defined in the package; if not, it assumes it is a
module and attempts to load it. If it fails to find it, an ImportError
exception is raised.
Contrarily, when using syntax like import item.subitem.subsubitem
, each item
except for the last must be a package; the last item can be a module or a
package but can’t be a class or function or variable defined in the previous
item.
6.4.1. Importing * From a Package¶
Now what happens when the user writes from sound.effects import *
? Ideally,
one would hope that this somehow goes out to the filesystem, finds which
submodules are present in the package, and imports them all. This could take a
long time and importing sub-modules might have unwanted side-effects that should
only happen when the sub-module is explicitly imported.
The only solution is for the package author to provide an explicit index of the
package. The import
statement uses the following convention: if a package’s
__init__.py
code defines a list named __all__
, it is taken to be the
list of module names that should be imported when from package import *
is
encountered. It is up to the package author to keep this list up-to-date when a
new version of the package is released. Package authors may also decide not to
support it, if they don’t see a use for importing * from their package. For
example, the file sound/effects/__init__.py
could contain the following
code:
__all__ = ["echo", "surround", "reverse"]
This would mean that from sound.effects import *
would import the three
named submodules of the sound.effects
package.
Be aware that submodules might become shadowed by locally defined names. For
example, if you added a reverse
function to the
sound/effects/__init__.py
file, the from sound.effects import *
would only import the two submodules echo
and surround
, but not the
reverse
submodule, because it is shadowed by the locally defined
reverse
function:
__all__ = [
"echo", # refers to the 'echo.py' file
"surround", # refers to the 'surround.py' file
"reverse", # !!! refers to the 'reverse' function now !!!
]
def reverse(msg: str): # <-- this name shadows the 'reverse.py' submodule
return msg[::-1] # in the case of a 'from sound.effects import *'
If __all__
is not defined, the statement from sound.effects import *
does not import all submodules from the package sound.effects
into the
current namespace; it only ensures that the package sound.effects
has
been imported (possibly running any initialization code in __init__.py
)
and then imports whatever names are defined in the package. This includes any
names defined (and submodules explicitly loaded) by __init__.py
. It
also includes any submodules of the package that were explicitly loaded by
previous import
statements. Consider this code:
import sound.effects.echo
import sound.effects.surround
from sound.effects import *
In this example, the echo
and surround
modules are imported in the
current namespace because they are defined in the sound.effects
package
when the from...import
statement is executed. (This also works when
__all__
is defined.)
Although certain modules are designed to export only names that follow certain
patterns when you use import *
, it is still considered bad practice in
production code.
Remember, there is nothing wrong with using from package import
specific_submodule
! In fact, this is the recommended notation unless the
importing module needs to use submodules with the same name from different
packages.
6.4.2. Intra-package References¶
When packages are structured into subpackages (as with the sound
package
in the example), you can use absolute imports to refer to submodules of siblings
packages. For example, if the module sound.filters.vocoder
needs to use
the echo
module in the sound.effects
package, it can use from
sound.effects import echo
.
You can also write relative imports, with the from module import name
form
of import statement. These imports use leading dots to indicate the current and
parent packages involved in the relative import. From the surround
module for example, you might use:
from . import echo
from .. import formats
from ..filters import equalizer
Note that relative imports are based on the name of the current module’s package. Since the main module does not have a package, modules intended for use as the main module of a Python application must always use absolute imports.
6.4.3. Packages in Multiple Directories¶
Packages support one more special attribute, __path__
. This is
initialized to be a sequence of strings containing the name of the
directory holding the
package’s __init__.py
before the code in that file is executed. This
variable can be modified; doing so affects future searches for modules and
subpackages contained in the package.
While this feature is not often needed, it can be used to extend the set of modules found in a package.
Footnotes


Data Structures

5. Data Structures¶
This chapter describes some things you’ve learned about already in more detail, and adds some new things as well.
5.1. More on Lists¶
The list data type has some more methods. Here are all of the methods of list objects:
- list.append(x)
Add an item to the end of the list. Similar to
a[len(a):] = [x]
.
- list.extend(iterable)
Extend the list by appending all the items from the iterable. Similar to
a[len(a):] = iterable
.
- list.insert(i, x)
Insert an item at a given position. The first argument is the index of the element before which to insert, so
a.insert(0, x)
inserts at the front of the list, anda.insert(len(a), x)
is equivalent toa.append(x)
.
- list.remove(x)
Remove the first item from the list whose value is equal to x. It raises a
ValueError
if there is no such item.
- list.pop([i])
Remove the item at the given position in the list, and return it. If no index is specified,
a.pop()
removes and returns the last item in the list. It raises anIndexError
if the list is empty or the index is outside the list range.
- list.clear()
Remove all items from the list. Similar to
del a[:]
.
- list.index(x[, start[, end]])
Return zero-based index of the first occurrence of x in the list. Raises a
ValueError
if there is no such item.The optional arguments start and end are interpreted as in the slice notation and are used to limit the search to a particular subsequence of the list. The returned index is computed relative to the beginning of the full sequence rather than the start argument.
- list.count(x)
Return the number of times x appears in the list.
- list.sort(*, key=None, reverse=False)
Sort the items of the list in place (the arguments can be used for sort customization, see
sorted()
for their explanation).
- list.reverse()
Reverse the elements of the list in place.
- list.copy()
Return a shallow copy of the list. Similar to
a[:]
.
An example that uses most of the list methods:
>>> fruits = ['orange', 'apple', 'pear', 'banana', 'kiwi', 'apple', 'banana']
>>> fruits.count('apple')
2
>>> fruits.count('tangerine')
0
>>> fruits.index('banana')
3
>>> fruits.index('banana', 4) # Find next banana starting at position 4
6
>>> fruits.reverse()
>>> fruits
['banana', 'apple', 'kiwi', 'banana', 'pear', 'apple', 'orange']
>>> fruits.append('grape')
>>> fruits
['banana', 'apple', 'kiwi', 'banana', 'pear', 'apple', 'orange', 'grape']
>>> fruits.sort()
>>> fruits
['apple', 'apple', 'banana', 'banana', 'grape', 'kiwi', 'orange', 'pear']
>>> fruits.pop()
'pear'
You might have noticed that methods like insert
, remove
or sort
that
only modify the list have no return value printed – they return the default
None
. [1] This is a design principle for all mutable data structures in
Python.
Another thing you might notice is that not all data can be sorted or
compared. For instance, [None, 'hello', 10]
doesn’t sort because
integers can’t be compared to strings and None
can’t be compared to
other types. Also, there are some types that don’t have a defined
ordering relation. For example, 3+4j < 5+7j
isn’t a valid
comparison.
5.1.1. Using Lists as Stacks¶
The list methods make it very easy to use a list as a stack, where the last
element added is the first element retrieved (“last-in, first-out”). To add an
item to the top of the stack, use append()
. To retrieve an item from the
top of the stack, use pop()
without an explicit index. For example:
>>> stack = [3, 4, 5]
>>> stack.append(6)
>>> stack.append(7)
>>> stack
[3, 4, 5, 6, 7]
>>> stack.pop()
7
>>> stack
[3, 4, 5, 6]
>>> stack.pop()
6
>>> stack.pop()
5
>>> stack
[3, 4]
5.1.2. Using Lists as Queues¶
It is also possible to use a list as a queue, where the first element added is the first element retrieved (“first-in, first-out”); however, lists are not efficient for this purpose. While appends and pops from the end of list are fast, doing inserts or pops from the beginning of a list is slow (because all of the other elements have to be shifted by one).
To implement a queue, use collections.deque
which was designed to
have fast appends and pops from both ends. For example:
>>> from collections import deque
>>> queue = deque(["Eric", "John", "Michael"])
>>> queue.append("Terry") # Terry arrives
>>> queue.append("Graham") # Graham arrives
>>> queue.popleft() # The first to arrive now leaves
'Eric'
>>> queue.popleft() # The second to arrive now leaves
'John'
>>> queue # Remaining queue in order of arrival
deque(['Michael', 'Terry', 'Graham'])
5.1.3. List Comprehensions¶
List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.
For example, assume we want to create a list of squares, like:
>>> squares = []
>>> for x in range(10):
... squares.append(x**2)
...
>>> squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Note that this creates (or overwrites) a variable named x
that still exists
after the loop completes. We can calculate the list of squares without any
side effects using:
squares = list(map(lambda x: x**2, range(10)))
or, equivalently:
squares = [x**2 for x in range(10)]
which is more concise and readable.
A list comprehension consists of brackets containing an expression followed
by a for
clause, then zero or more for
or if
clauses. The result will be a new list resulting from evaluating the expression
in the context of the for
and if
clauses which follow it.
For example, this listcomp combines the elements of two lists if they are not
equal:
>>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]
and it’s equivalent to:
>>> combs = []
>>> for x in [1,2,3]:
... for y in [3,1,4]:
... if x != y:
... combs.append((x, y))
...
>>> combs
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]
Note how the order of the for
and if
statements is the
same in both these snippets.
If the expression is a tuple (e.g. the (x, y)
in the previous example),
it must be parenthesized.
>>> vec = [-4, -2, 0, 2, 4]
>>> # create a new list with the values doubled
>>> [x*2 for x in vec]
[-8, -4, 0, 4, 8]
>>> # filter the list to exclude negative numbers
>>> [x for x in vec if x >= 0]
[0, 2, 4]
>>> # apply a function to all the elements
>>> [abs(x) for x in vec]
[4, 2, 0, 2, 4]
>>> # call a method on each element
>>> freshfruit = [' banana', ' loganberry ', 'passion fruit ']
>>> [weapon.strip() for weapon in freshfruit]
['banana', 'loganberry', 'passion fruit']
>>> # create a list of 2-tuples like (number, square)
>>> [(x, x**2) for x in range(6)]
[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)]
>>> # the tuple must be parenthesized, otherwise an error is raised
>>> [x, x**2 for x in range(6)]
File "<stdin>", line 1
[x, x**2 for x in range(6)]
^^^^^^^
SyntaxError: did you forget parentheses around the comprehension target?
>>> # flatten a list using a listcomp with two 'for'
>>> vec = [[1,2,3], [4,5,6], [7,8,9]]
>>> [num for elem in vec for num in elem]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
List comprehensions can contain complex expressions and nested functions:
>>> from math import pi
>>> [str(round(pi, i)) for i in range(1, 6)]
['3.1', '3.14', '3.142', '3.1416', '3.14159']
5.1.4. Nested List Comprehensions¶
The initial expression in a list comprehension can be any arbitrary expression, including another list comprehension.
Consider the following example of a 3x4 matrix implemented as a list of 3 lists of length 4:
>>> matrix = [
... [1, 2, 3, 4],
... [5, 6, 7, 8],
... [9, 10, 11, 12],
... ]
The following list comprehension will transpose rows and columns:
>>> [[row[i] for row in matrix] for i in range(4)]
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
As we saw in the previous section, the inner list comprehension is evaluated in
the context of the for
that follows it, so this example is
equivalent to:
>>> transposed = []
>>> for i in range(4):
... transposed.append([row[i] for row in matrix])
...
>>> transposed
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
which, in turn, is the same as:
>>> transposed = []
>>> for i in range(4):
... # the following 3 lines implement the nested listcomp
... transposed_row = []
... for row in matrix:
... transposed_row.append(row[i])
... transposed.append(transposed_row)
...
>>> transposed
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
In the real world, you should prefer built-in functions to complex flow statements.
The zip()
function would do a great job for this use case:
>>> list(zip(*matrix))
[(1, 5, 9), (2, 6, 10), (3, 7, 11), (4, 8, 12)]
See Unpacking Argument Lists for details on the asterisk in this line.
5.2. The del
statement¶
There is a way to remove an item from a list given its index instead of its
value: the del
statement. This differs from the pop()
method
which returns a value. The del
statement can also be used to remove
slices from a list or clear the entire list (which we did earlier by assignment
of an empty list to the slice). For example:
>>> a = [-1, 1, 66.25, 333, 333, 1234.5]
>>> del a[0]
>>> a
[1, 66.25, 333, 333, 1234.5]
>>> del a[2:4]
>>> a
[1, 66.25, 1234.5]
>>> del a[:]
>>> a
[]
del
can also be used to delete entire variables:
>>> del a
Referencing the name a
hereafter is an error (at least until another value
is assigned to it). We’ll find other uses for del
later.
5.3. Tuples and Sequences¶
We saw that lists and strings have many common properties, such as indexing and slicing operations. They are two examples of sequence data types (see Sequence Types — list, tuple, range). Since Python is an evolving language, other sequence data types may be added. There is also another standard sequence data type: the tuple.
A tuple consists of a number of values separated by commas, for instance:
>>> t = 12345, 54321, 'hello!'
>>> t[0]
12345
>>> t
(12345, 54321, 'hello!')
>>> # Tuples may be nested:
>>> u = t, (1, 2, 3, 4, 5)
>>> u
((12345, 54321, 'hello!'), (1, 2, 3, 4, 5))
>>> # Tuples are immutable:
>>> t[0] = 88888
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> # but they can contain mutable objects:
>>> v = ([1, 2, 3], [3, 2, 1])
>>> v
([1, 2, 3], [3, 2, 1])
As you see, on output tuples are always enclosed in parentheses, so that nested tuples are interpreted correctly; they may be input with or without surrounding parentheses, although often parentheses are necessary anyway (if the tuple is part of a larger expression). It is not possible to assign to the individual items of a tuple, however it is possible to create tuples which contain mutable objects, such as lists.
Though tuples may seem similar to lists, they are often used in different
situations and for different purposes.
Tuples are immutable, and usually contain a heterogeneous sequence of
elements that are accessed via unpacking (see later in this section) or indexing
(or even by attribute in the case of namedtuples
).
Lists are mutable, and their elements are usually homogeneous and are
accessed by iterating over the list.
A special problem is the construction of tuples containing 0 or 1 items: the syntax has some extra quirks to accommodate these. Empty tuples are constructed by an empty pair of parentheses; a tuple with one item is constructed by following a value with a comma (it is not sufficient to enclose a single value in parentheses). Ugly, but effective. For example:
>>> empty = ()
>>> singleton = 'hello', # <-- note trailing comma
>>> len(empty)
0
>>> len(singleton)
1
>>> singleton
('hello',)
The statement t = 12345, 54321, 'hello!'
is an example of tuple packing:
the values 12345
, 54321
and 'hello!'
are packed together in a tuple.
The reverse operation is also possible:
>>> x, y, z = t
This is called, appropriately enough, sequence unpacking and works for any sequence on the right-hand side. Sequence unpacking requires that there are as many variables on the left side of the equals sign as there are elements in the sequence. Note that multiple assignment is really just a combination of tuple packing and sequence unpacking.
5.4. Sets¶
Python also includes a data type for sets. A set is an unordered collection with no duplicate elements. Basic uses include membership testing and eliminating duplicate entries. Set objects also support mathematical operations like union, intersection, difference, and symmetric difference.
Curly braces or the set()
function can be used to create sets. Note: to
create an empty set you have to use set()
, not {}
; the latter creates an
empty dictionary, a data structure that we discuss in the next section.
Here is a brief demonstration:
>>> basket = {'apple', 'orange', 'apple', 'pear', 'orange', 'banana'}
>>> print(basket) # show that duplicates have been removed
{'orange', 'banana', 'pear', 'apple'}
>>> 'orange' in basket # fast membership testing
True
>>> 'crabgrass' in basket
False
>>> # Demonstrate set operations on unique letters from two words
>>>
>>> a = set('abracadabra')
>>> b = set('alacazam')
>>> a # unique letters in a
{'a', 'r', 'b', 'c', 'd'}
>>> a - b # letters in a but not in b
{'r', 'd', 'b'}
>>> a | b # letters in a or b or both
{'a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'}
>>> a & b # letters in both a and b
{'a', 'c'}
>>> a ^ b # letters in a or b but not both
{'r', 'd', 'b', 'm', 'z', 'l'}
Similarly to list comprehensions, set comprehensions are also supported:
>>> a = {x for x in 'abracadabra' if x not in 'abc'}
>>> a
{'r', 'd'}
5.5. Dictionaries¶
Another useful data type built into Python is the dictionary (see
Mapping Types — dict). Dictionaries are sometimes found in other languages as
“associative memories” or “associative arrays”. Unlike sequences, which are
indexed by a range of numbers, dictionaries are indexed by keys, which can be
any immutable type; strings and numbers can always be keys. Tuples can be used
as keys if they contain only strings, numbers, or tuples; if a tuple contains
any mutable object either directly or indirectly, it cannot be used as a key.
You can’t use lists as keys, since lists can be modified in place using index
assignments, slice assignments, or methods like append()
and
extend()
.
It is best to think of a dictionary as a set of key: value pairs,
with the requirement that the keys are unique (within one dictionary). A pair of
braces creates an empty dictionary: {}
. Placing a comma-separated list of
key:value pairs within the braces adds initial key:value pairs to the
dictionary; this is also the way dictionaries are written on output.
The main operations on a dictionary are storing a value with some key and
extracting the value given the key. It is also possible to delete a key:value
pair with del
. If you store using a key that is already in use, the old
value associated with that key is forgotten.
Extracting a value for a non-existent key by subscripting (d[key]
) raises a
KeyError
. To avoid getting this error when trying to access a possibly
non-existent key, use the get()
method instead, which returns
None
(or a specified default value) if the key is not in the dictionary.
Performing list(d)
on a dictionary returns a list of all the keys
used in the dictionary, in insertion order (if you want it sorted, just use
sorted(d)
instead). To check whether a single key is in the
dictionary, use the in
keyword.
Here is a small example using a dictionary:
>>> tel = {'jack': 4098, 'sape': 4139}
>>> tel['guido'] = 4127
>>> tel
{'jack': 4098, 'sape': 4139, 'guido': 4127}
>>> tel['jack']
4098
>>> tel['irv']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'irv'
>>> print(tel.get('irv'))
None
>>> del tel['sape']
>>> tel['irv'] = 4127
>>> tel
{'jack': 4098, 'guido': 4127, 'irv': 4127}
>>> list(tel)
['jack', 'guido', 'irv']
>>> sorted(tel)
['guido', 'irv', 'jack']
>>> 'guido' in tel
True
>>> 'jack' not in tel
False
The dict()
constructor builds dictionaries directly from sequences of
key-value pairs:
>>> dict([('sape', 4139), ('guido', 4127), ('jack', 4098)])
{'sape': 4139, 'guido': 4127, 'jack': 4098}
In addition, dict comprehensions can be used to create dictionaries from arbitrary key and value expressions:
>>> {x: x**2 for x in (2, 4, 6)}
{2: 4, 4: 16, 6: 36}
When the keys are simple strings, it is sometimes easier to specify pairs using keyword arguments:
>>> dict(sape=4139, guido=4127, jack=4098)
{'sape': 4139, 'guido': 4127, 'jack': 4098}
5.6. Looping Techniques¶
When looping through dictionaries, the key and corresponding value can be
retrieved at the same time using the items()
method.
>>> knights = {'gallahad': 'the pure', 'robin': 'the brave'}
>>> for k, v in knights.items():
... print(k, v)
...
gallahad the pure
robin the brave
When looping through a sequence, the position index and corresponding value can
be retrieved at the same time using the enumerate()
function.
>>> for i, v in enumerate(['tic', 'tac', 'toe']):
... print(i, v)
...
0 tic
1 tac
2 toe
To loop over two or more sequences at the same time, the entries can be paired
with the zip()
function.
>>> questions = ['name', 'quest', 'favorite color']
>>> answers = ['lancelot', 'the holy grail', 'blue']
>>> for q, a in zip(questions, answers):
... print('What is your {0}? It is {1}.'.format(q, a))
...
What is your name? It is lancelot.
What is your quest? It is the holy grail.
What is your favorite color? It is blue.
To loop over a sequence in reverse, first specify the sequence in a forward
direction and then call the reversed()
function.
>>> for i in reversed(range(1, 10, 2)):
... print(i)
...
9
7
5
3
1
To loop over a sequence in sorted order, use the sorted()
function which
returns a new sorted list while leaving the source unaltered.
>>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
>>> for i in sorted(basket):
... print(i)
...
apple
apple
banana
orange
orange
pear
Using set()
on a sequence eliminates duplicate elements. The use of
sorted()
in combination with set()
over a sequence is an idiomatic
way to loop over unique elements of the sequence in sorted order.
>>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
>>> for f in sorted(set(basket)):
... print(f)
...
apple
banana
orange
pear
It is sometimes tempting to change a list while you are looping over it; however, it is often simpler and safer to create a new list instead.
>>> import math
>>> raw_data = [56.2, float('NaN'), 51.7, 55.3, 52.5, float('NaN'), 47.8]
>>> filtered_data = []
>>> for value in raw_data:
... if not math.isnan(value):
... filtered_data.append(value)
...
>>> filtered_data
[56.2, 51.7, 55.3, 52.5, 47.8]
5.7. More on Conditions¶
The conditions used in while
and if
statements can contain any
operators, not just comparisons.
The comparison operators in
and not in
are membership tests that
determine whether a value is in (or not in) a container. The operators is
and is not
compare whether two objects are really the same object. All
comparison operators have the same priority, which is lower than that of all
numerical operators.
Comparisons can be chained. For example, a < b == c
tests whether a
is
less than b
and moreover b
equals c
.
Comparisons may be combined using the Boolean operators and
and or
, and
the outcome of a comparison (or of any other Boolean expression) may be negated
with not
. These have lower priorities than comparison operators; between
them, not
has the highest priority and or
the lowest, so that A and
not B or C
is equivalent to (A and (not B)) or C
. As always, parentheses
can be used to express the desired composition.
The Boolean operators and
and or
are so-called short-circuit
operators: their arguments are evaluated from left to right, and evaluation
stops as soon as the outcome is determined. For example, if A
and C
are
true but B
is false, A and B and C
does not evaluate the expression
C
. When used as a general value and not as a Boolean, the return value of a
short-circuit operator is the last evaluated argument.
It is possible to assign the result of a comparison or other Boolean expression to a variable. For example,
>>> string1, string2, string3 = '', 'Trondheim', 'Hammer Dance'
>>> non_null = string1 or string2 or string3
>>> non_null
'Trondheim'
Note that in Python, unlike C, assignment inside expressions must be done
explicitly with the
walrus operator :=
.
This avoids a common class of problems encountered in C programs: typing =
in an expression when ==
was intended.
5.8. Comparing Sequences and Other Types¶
Sequence objects typically may be compared to other objects with the same sequence type. The comparison uses lexicographical ordering: first the first two items are compared, and if they differ this determines the outcome of the comparison; if they are equal, the next two items are compared, and so on, until either sequence is exhausted. If two items to be compared are themselves sequences of the same type, the lexicographical comparison is carried out recursively. If all items of two sequences compare equal, the sequences are considered equal. If one sequence is an initial sub-sequence of the other, the shorter sequence is the smaller (lesser) one. Lexicographical ordering for strings uses the Unicode code point number to order individual characters. Some examples of comparisons between sequences of the same type:
(1, 2, 3) < (1, 2, 4)
[1, 2, 3] < [1, 2, 4]
'ABC' < 'C' < 'Pascal' < 'Python'
(1, 2, 3, 4) < (1, 2, 4)
(1, 2) < (1, 2, -1)
(1, 2, 3) == (1.0, 2.0, 3.0)
(1, 2, ('aa', 'ab')) < (1, 2, ('abc', 'a'), 4)
Note that comparing objects of different types with <
or >
is legal
provided that the objects have appropriate comparison methods. For example,
mixed numeric types are compared according to their numeric value, so 0 equals
0.0, etc. Otherwise, rather than providing an arbitrary ordering, the
interpreter will raise a TypeError
exception.
Footnotes

File
, line
:
File
, line
, in
:
File
, line
, in
: