— Byte-compile Python libraries

compileall
— Byte-compile Python libraries¶
Source code: Lib/compileall.py
This module provides some utility functions to support installing Python libraries. These functions compile Python source files in a directory tree. This module can be used to create the cached byte-code files at library installation time, which makes them available for use even by users who don’t have write permission to the library directories.
Availability: not WASI.
This module does not work or is not available on WebAssembly. See WebAssembly platforms for more information.
Command-line use¶
This module can work as a script (using python -m compileall) to compile Python sources.
- directory ...¶
- file ...¶
Positional arguments are files to compile or directories that contain source files, traversed recursively. If no argument is given, behave as if the command line was
-l <directories from sys.path>
.
- -l¶
Do not recurse into subdirectories, only compile source code files directly contained in the named or implied directories.
- -f¶
Force rebuild even if timestamps are up-to-date.
- -q¶
Do not print the list of files compiled. If passed once, error messages will still be printed. If passed twice (
-qq
), all output is suppressed.
- -d destdir¶
Directory prepended to the path to each file being compiled. This will appear in compilation time tracebacks, and is also compiled in to the byte-code file, where it will be used in tracebacks and other messages in cases where the source file does not exist at the time the byte-code file is executed.
- -s strip_prefix¶
Remove the given prefix from paths recorded in the
.pyc
files. Paths are made relative to the prefix.This option can be used with
-p
but not with-d
.
- -p prepend_prefix¶
Prepend the given prefix to paths recorded in the
.pyc
files. Use-p /
to make the paths absolute.This option can be used with
-s
but not with-d
.
- -x regex¶
regex is used to search the full path to each file considered for compilation, and if the regex produces a match, the file is skipped.
- -i list¶
Read the file
list
and add each line that it contains to the list of files and directories to compile. Iflist
is-
, read lines fromstdin
.
- -b¶
Write the byte-code files to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their PEP 3147 locations and names, which allows byte-code files from multiple versions of Python to coexist.
- -r¶
Control the maximum recursion level for subdirectories. If this is given, then
-l
option will not be taken into account. python -m compileall <directory> -r 0 is equivalent to python -m compileall <directory> -l.
- -j N¶
Use N workers to compile the files within the given directory. If
0
is used, then the result ofos.process_cpu_count()
will be used.
- --invalidation-mode [timestamp|checked-hash|unchecked-hash]¶
Control how the generated byte-code files are invalidated at runtime. The
timestamp
value, means that.pyc
files with the source timestamp and size embedded will be generated. Thechecked-hash
andunchecked-hash
values cause hash-based pycs to be generated. Hash-based pycs embed a hash of the source file contents rather than a timestamp. See Cached bytecode invalidation for more information on how Python validates bytecode cache files at runtime. The default istimestamp
if theSOURCE_DATE_EPOCH
environment variable is not set, andchecked-hash
if theSOURCE_DATE_EPOCH
environment variable is set.
- -o level¶
Compile with the given optimization level. May be used multiple times to compile for multiple levels at a time (for example,
compileall -o 1 -o 2
).
- -e dir¶
Ignore symlinks pointing outside the given directory.
- --hardlink-dupes¶
If two
.pyc
files with different optimization level have the same content, use hard links to consolidate duplicate files.
Changed in version 3.2: Added the -i
, -b
and -h
options.
Changed in version 3.5: Added the -j
, -r
, and -qq
options. -q
option
was changed to a multilevel value. -b
will always produce a
byte-code file ending in .pyc
, never .pyo
.
Changed in version 3.7: Added the --invalidation-mode
option.
Changed in version 3.9: Added the -s
, -p
, -e
and --hardlink-dupes
options.
Raised the default recursion limit from 10 to
sys.getrecursionlimit()
.
Added the possibility to specify the -o
option multiple times.
There is no command-line option to control the optimization level used by the
compile()
function, because the Python interpreter itself already
provides the option: python -O -m compileall.
Similarly, the compile()
function respects the sys.pycache_prefix
setting. The generated bytecode cache will only be useful if compile()
is
run with the same sys.pycache_prefix
(if any) that will be used at
runtime.
Public functions¶
- compileall.compile_dir(dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)¶
Recursively descend the directory tree named by dir, compiling all
.py
files along the way. Return a true value if all the files compiled successfully, and a false value otherwise.The maxlevels parameter is used to limit the depth of the recursion; it defaults to
sys.getrecursionlimit()
.If ddir is given, it is prepended to the path to each file being compiled for use in compilation time tracebacks, and is also compiled in to the byte-code file, where it will be used in tracebacks and other messages in cases where the source file does not exist at the time the byte-code file is executed.
If force is true, modules are re-compiled even if the timestamps are up to date.
If rx is given, its
search
method is called on the complete path to each file considered for compilation, and if it returns a true value, the file is skipped. This can be used to exclude files matching a regular expression, given as a re.Pattern object.If quiet is
False
or0
(the default), the filenames and other information are printed to standard out. Set to1
, only errors are printed. Set to2
, all output is suppressed.If legacy is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their PEP 3147 locations and names, which allows byte-code files from multiple versions of Python to coexist.
optimize specifies the optimization level for the compiler. It is passed to the built-in
compile()
function. Accepts also a sequence of optimization levels which lead to multiple compilations of one.py
file in one call.The argument workers specifies how many workers are used to compile files in parallel. The default is to not use multiple workers. If the platform can’t use multiple workers and workers argument is given, then sequential compilation will be used as a fallback. If workers is 0, the number of cores in the system is used. If workers is lower than
0
, aValueError
will be raised.invalidation_mode should be a member of the
py_compile.PycInvalidationMode
enum and controls how the generated pycs are invalidated at runtime.The stripdir, prependdir and limit_sl_dest arguments correspond to the
-s
,-p
and-e
options described above. They may be specified asstr
oros.PathLike
.If hardlink_dupes is true and two
.pyc
files with different optimization level have the same content, use hard links to consolidate duplicate files.Changed in version 3.2: Added the legacy and optimize parameter.
Changed in version 3.5: Added the workers parameter.
Changed in version 3.5: quiet parameter was changed to a multilevel value.
Changed in version 3.5: The legacy parameter only writes out
.pyc
files, not.pyo
files no matter what the value of optimize is.Changed in version 3.6: Accepts a path-like object.
Changed in version 3.7: The invalidation_mode parameter was added.
Changed in version 3.7.2: The invalidation_mode parameter’s default value is updated to
None
.Changed in version 3.8: Setting workers to 0 now chooses the optimal number of cores.
Changed in version 3.9: Added stripdir, prependdir, limit_sl_dest and hardlink_dupes arguments. Default value of maxlevels was changed from
10
tosys.getrecursionlimit()
- compileall.compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)¶
Compile the file with path fullname. Return a true value if the file compiled successfully, and a false value otherwise.
If ddir is given, it is prepended to the path to the file being compiled for use in compilation time tracebacks, and is also compiled in to the byte-code file, where it will be used in tracebacks and other messages in cases where the source file does not exist at the time the byte-code file is executed.
If rx is given, its
search
method is passed the full path name to the file being compiled, and if it returns a true value, the file is not compiled andTrue
is returned. This can be used to exclude files matching a regular expression, given as a re.Pattern object.If quiet is
False
or0
(the default), the filenames and other information are printed to standard out. Set to1
, only errors are printed. Set to2
, all output is suppressed.If legacy is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their PEP 3147 locations and names, which allows byte-code files from multiple versions of Python to coexist.
optimize specifies the optimization level for the compiler. It is passed to the built-in
compile()
function. Accepts also a sequence of optimization levels which lead to multiple compilations of one.py
file in one call.invalidation_mode should be a member of the
py_compile.PycInvalidationMode
enum and controls how the generated pycs are invalidated at runtime.The stripdir, prependdir and limit_sl_dest arguments correspond to the
-s
,-p
and-e
options described above. They may be specified asstr
oros.PathLike
.If hardlink_dupes is true and two
.pyc
files with different optimization level have the same content, use hard links to consolidate duplicate files.Added in version 3.2.
Changed in version 3.5: quiet parameter was changed to a multilevel value.
Changed in version 3.5: The legacy parameter only writes out
.pyc
files, not.pyo
files no matter what the value of optimize is.Changed in version 3.7: The invalidation_mode parameter was added.
Changed in version 3.7.2: The invalidation_mode parameter’s default value is updated to
None
.Changed in version 3.9: Added stripdir, prependdir, limit_sl_dest and hardlink_dupes arguments.
- compileall.compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=None)¶
Byte-compile all the
.py
files found alongsys.path
. Return a true value if all the files compiled successfully, and a false value otherwise.If skip_curdir is true (the default), the current directory is not included in the search. All other parameters are passed to the
compile_dir()
function. Note that unlike the other compile functions,maxlevels
defaults to0
.Changed in version 3.2: Added the legacy and optimize parameter.
Changed in version 3.5: quiet parameter was changed to a multilevel value.
Changed in version 3.5: The legacy parameter only writes out
.pyc
files, not.pyo
files no matter what the value of optimize is.Changed in version 3.7: The invalidation_mode parameter was added.
Changed in version 3.7.2: The invalidation_mode parameter’s default value is updated to
None
.
To force a recompile of all the .py
files in the Lib/
subdirectory and all its subdirectories:
import compileall
compileall.compile_dir('Lib/', force=True)
# Perform same compilation, excluding files in .svn directories.
import re
compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True)
# pathlib.Path objects can also be used.
import pathlib
compileall.compile_dir(pathlib.Path('Lib/'), force=True)
See also
- Module
py_compile
Byte-compile a single source file.


— Compile Python source files

py_compile
— Compile Python source files¶
Source code: Lib/py_compile.py
The py_compile
module provides a function to generate a byte-code file
from a source file, and another function used when the module source file is
invoked as a script.
Though not often needed, this function can be useful when installing modules for shared use, especially if some of the users may not have permission to write the byte-code cache files in the directory containing the source code.
- exception py_compile.PyCompileError¶
Exception raised when an error occurs while attempting to compile the file.
- py_compile.compile(file, cfile=None, dfile=None, doraise=False, optimize=-1, invalidation_mode=PycInvalidationMode.TIMESTAMP, quiet=0)¶
Compile a source file to byte-code and write out the byte-code cache file. The source code is loaded from the file named file. The byte-code is written to cfile, which defaults to the PEP 3147/PEP 488 path, ending in
.pyc
. For example, if file is/foo/bar/baz.py
cfile will default to/foo/bar/__pycache__/baz.cpython-32.pyc
for Python 3.2. If dfile is specified, it is used instead of file as the name of the source file from which source lines are obtained for display in exception tracebacks. If doraise is true, aPyCompileError
is raised when an error is encountered while compiling file. If doraise is false (the default), an error string is written tosys.stderr
, but no exception is raised. This function returns the path to byte-compiled file, i.e. whatever cfile value was used.The doraise and quiet arguments determine how errors are handled while compiling file. If quiet is 0 or 1, and doraise is false, the default behaviour is enabled: an error string is written to
sys.stderr
, and the function returnsNone
instead of a path. If doraise is true, aPyCompileError
is raised instead. However if quiet is 2, no message is written, and doraise has no effect.If the path that cfile becomes (either explicitly specified or computed) is a symlink or non-regular file,
FileExistsError
will be raised. This is to act as a warning that import will turn those paths into regular files if it is allowed to write byte-compiled files to those paths. This is a side-effect of import using file renaming to place the final byte-compiled file into place to prevent concurrent file writing issues.optimize controls the optimization level and is passed to the built-in
compile()
function. The default of-1
selects the optimization level of the current interpreter.invalidation_mode should be a member of the
PycInvalidationMode
enum and controls how the generated bytecode cache is invalidated at runtime. The default isPycInvalidationMode.CHECKED_HASH
if theSOURCE_DATE_EPOCH
environment variable is set, otherwise the default isPycInvalidationMode.TIMESTAMP
.Changed in version 3.2: Changed default value of cfile to be PEP 3147-compliant. Previous default was file +
'c'
('o'
if optimization was enabled). Also added the optimize parameter.Changed in version 3.4: Changed code to use
importlib
for the byte-code cache file writing. This means file creation/writing semantics now match whatimportlib
does, e.g. permissions, write-and-move semantics, etc. Also added the caveat thatFileExistsError
is raised if cfile is a symlink or non-regular file.Changed in version 3.7: The invalidation_mode parameter was added as specified in PEP 552. If the
SOURCE_DATE_EPOCH
environment variable is set, invalidation_mode will be forced toPycInvalidationMode.CHECKED_HASH
.Changed in version 3.7.2: The
SOURCE_DATE_EPOCH
environment variable no longer overrides the value of the invalidation_mode argument, and determines its default value instead.Changed in version 3.8: The quiet parameter was added.
- class py_compile.PycInvalidationMode¶
An enumeration of possible methods the interpreter can use to determine whether a bytecode file is up to date with a source file. The
.pyc
file indicates the desired invalidation mode in its header. See Cached bytecode invalidation for more information on how Python invalidates.pyc
files at runtime.Added in version 3.7.
- TIMESTAMP¶
The
.pyc
file includes the timestamp and size of the source file, which Python will compare against the metadata of the source file at runtime to determine if the.pyc
file needs to be regenerated.
- CHECKED_HASH¶
The
.pyc
file includes a hash of the source file content, which Python will compare against the source at runtime to determine if the.pyc
file needs to be regenerated.
- UNCHECKED_HASH¶
Like
CHECKED_HASH
, the.pyc
file includes a hash of the source file content. However, Python will at runtime assume the.pyc
file is up to date and not validate the.pyc
against the source file at all.This option is useful when the
.pycs
are kept up to date by some system external to Python like a build system.
Command-Line Interface¶
This module can be invoked as a script to compile several source files. The files named in filenames are compiled and the resulting bytecode is cached in the normal manner. This program does not search a directory structure to locate source files; it only compiles files named explicitly. The exit status is nonzero if one of the files could not be compiled.
- <file> ... <fileN>¶
- -¶
Positional arguments are files to compile. If
-
is the only parameter, the list of files is taken from standard input.
- -q, --quiet¶
Suppress errors output.
Changed in version 3.2: Added support for -
.
Changed in version 3.10: Added support for -q
.
See also
- Module
compileall
Utilities to compile all Python source files in a directory tree.


— Python module browser support

pyclbr
— Python module browser support¶
Source code: Lib/pyclbr.py
The pyclbr
module provides limited information about the
functions, classes, and methods defined in a Python-coded module. The
information is sufficient to implement a module browser. The
information is extracted from the Python source code rather than by
importing the module, so this module is safe to use with untrusted code.
This restriction makes it impossible to use this module with modules not
implemented in Python, including all standard and optional extension
modules.
- pyclbr.readmodule(module, path=None)¶
Return a dictionary mapping module-level class names to class descriptors. If possible, descriptors for imported base classes are included. Parameter module is a string with the name of the module to read; it may be the name of a module within a package. If given, path is a sequence of directory paths prepended to
sys.path
, which is used to locate the module source code.This function is the original interface and is only kept for back compatibility. It returns a filtered version of the following.
- pyclbr.readmodule_ex(module, path=None)¶
Return a dictionary-based tree containing a function or class descriptors for each function and class defined in the module with a
def
orclass
statement. The returned dictionary maps module-level function and class names to their descriptors. Nested objects are entered into the children dictionary of their parent. As with readmodule, module names the module to be read and path is prepended to sys.path. If the module being read is a package, the returned dictionary has a key'__path__'
whose value is a list containing the package search path.
Added in version 3.7: Descriptors for nested definitions. They are accessed through the new children attribute. Each has a new parent attribute.
The descriptors returned by these functions are instances of Function and Class classes. Users are not expected to create instances of these classes.
Function Objects¶
- class pyclbr.Function¶
Class
Function
instances describe functions defined by def statements. They have the following attributes:- file¶
Name of the file in which the function is defined.
- module¶
The name of the module defining the function described.
- name¶
The name of the function.
- lineno¶
The line number in the file where the definition starts.
- parent¶
For top-level functions,
None
. For nested functions, the parent.Added in version 3.7.
- children¶
A
dictionary
mapping names to descriptors for nested functions and classes.Added in version 3.7.
Class Objects¶
- class pyclbr.Class¶
Class
Class
instances describe classes defined by class statements. They have the same attributes asFunctions
and two more.- file¶
Name of the file in which the class is defined.
- module¶
The name of the module defining the class described.
- name¶
The name of the class.
- lineno¶
The line number in the file where the definition starts.
- parent¶
For top-level classes,
None
. For nested classes, the parent.Added in version 3.7.
- children¶
A dictionary mapping names to descriptors for nested functions and classes.
Added in version 3.7.
- super¶
A list of
Class
objects which describe the immediate base classes of the class being described. Classes which are named as superclasses but which are not discoverable byreadmodule_ex()
are listed as a string with the class name instead of asClass
objects.
- methods¶
A
dictionary
mapping method names to line numbers. This can be derived from the newerchildren
dictionary, but remains for back-compatibility.


— Detection of ambiguous indentation

tabnanny
— Detection of ambiguous indentation¶
Source code: Lib/tabnanny.py
For the time being this module is intended to be called as a script. However it
is possible to import it into an IDE and use the function check()
described below.
Note
The API provided by this module is likely to change in future releases; such changes may not be backward compatible.
- tabnanny.check(file_or_dir)¶
If file_or_dir is a directory and not a symbolic link, then recursively descend the directory tree named by file_or_dir, checking all
.py
files along the way. If file_or_dir is an ordinary Python source file, it is checked for whitespace related problems. The diagnostic messages are written to standard output using theprint()
function.
- tabnanny.verbose¶
Flag indicating whether to print verbose messages. This is incremented by the
-v
option if called as a script.
- tabnanny.filename_only¶
Flag indicating whether to print only the filenames of files containing whitespace related problems. This is set to true by the
-q
option if called as a script.
- exception tabnanny.NannyNag¶
Raised by
process_tokens()
if detecting an ambiguous indent. Captured and handled incheck()
.
- tabnanny.process_tokens(tokens)¶
This function is used by
check()
to process tokens generated by thetokenize
module.
See also
- Module
tokenize
Lexical scanner for Python source code.


— Tokenizer for Python source

tokenize
— Tokenizer for Python source¶
Source code: Lib/tokenize.py
The tokenize
module provides a lexical scanner for Python source code,
implemented in Python. The scanner in this module returns comments as tokens
as well, making it useful for implementing “pretty-printers”, including
colorizers for on-screen displays.
To simplify token stream handling, all operator and
delimiter tokens and Ellipsis
are returned using
the generic OP
token type. The exact
type can be determined by checking the exact_type
property on the
named tuple returned from tokenize.tokenize()
.
Warning
Note that the functions in this module are only designed to parse
syntactically valid Python code (code that does not raise when parsed
using ast.parse()
). The behavior of the functions in this module is
undefined when providing invalid Python code and it can change at any
point.
Tokenizing Input¶
The primary entry point is a generator:
- tokenize.tokenize(readline)¶
The
tokenize()
generator requires one argument, readline, which must be a callable object which provides the same interface as theio.IOBase.readline()
method of file objects. Each call to the function should return one line of input as bytes.The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple
(srow, scol)
of ints specifying the row and column where the token begins in the source; a 2-tuple(erow, ecol)
of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed (the last tuple item) is the physical line. The 5 tuple is returned as a named tuple with the field names:type string start end line
.The returned named tuple has an additional property named
exact_type
that contains the exact operator type forOP
tokens. For all other token typesexact_type
equals the named tupletype
field.Changed in version 3.1: Added support for named tuples.
Changed in version 3.3: Added support for
exact_type
.tokenize()
determines the source encoding of the file by looking for a UTF-8 BOM or encoding cookie, according to PEP 263.
- tokenize.generate_tokens(readline)¶
Tokenize a source reading unicode strings instead of bytes.
Like
tokenize()
, the readline argument is a callable returning a single line of input. However,generate_tokens()
expects readline to return a str object rather than bytes.The result is an iterator yielding named tuples, exactly like
tokenize()
. It does not yield anENCODING
token.
All constants from the token
module are also exported from
tokenize
.
Another function is provided to reverse the tokenization process. This is useful for creating tools that tokenize a script, modify the token stream, and write back the modified script.
- tokenize.untokenize(iterable)¶
Converts tokens back into Python source code. The iterable must return sequences with at least two elements, the token type and the token string. Any additional sequence elements are ignored.
The result is guaranteed to tokenize back to match the input so that the conversion is lossless and round-trips are assured. The guarantee applies only to the token type and token string as the spacing between tokens (column positions) may change.
It returns bytes, encoded using the
ENCODING
token, which is the first token sequence output bytokenize()
. If there is no encoding token in the input, it returns a str instead.
tokenize()
needs to detect the encoding of source files it tokenizes. The
function it uses to do this is available:
- tokenize.detect_encoding(readline)¶
The
detect_encoding()
function is used to detect the encoding that should be used to decode a Python source file. It requires one argument, readline, in the same way as thetokenize()
generator.It will call readline a maximum of twice, and return the encoding used (as a string) and a list of any lines (not decoded from bytes) it has read in.
It detects the encoding from the presence of a UTF-8 BOM or an encoding cookie as specified in PEP 263. If both a BOM and a cookie are present, but disagree, a
SyntaxError
will be raised. Note that if the BOM is found,'utf-8-sig'
will be returned as an encoding.If no encoding is specified, then the default of
'utf-8'
will be returned.Use
open()
to open Python source files: it usesdetect_encoding()
to detect the file encoding.
- tokenize.open(filename)¶
Open a file in read only mode using the encoding detected by
detect_encoding()
.Added in version 3.2.
- exception tokenize.TokenError¶
Raised when either a docstring or expression that may be split over several lines is not completed anywhere in the file, for example:
"""Beginning of docstring
or:
[1, 2, 3
Command-Line Usage¶
Added in version 3.3.
The tokenize
module can be executed as a script from the command line.
It is as simple as:
python -m tokenize [-e] [filename.py]
The following options are accepted:
- -h, --help¶
show this help message and exit
- -e, --exact¶
display token names using the exact type
If filename.py
is specified its contents are tokenized to stdout.
Otherwise, tokenization is performed on stdin.
Examples¶
Example of a script rewriter that transforms float literals into Decimal objects:
from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP
from io import BytesIO
def decistmt(s):
"""Substitute Decimals for floats in a string of statements.
>>> from decimal import Decimal
>>> s = 'print(+21.3e-5*-.1234/81.7)'
>>> decistmt(s)
"print (+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7'))"
The format of the exponent is inherited from the platform C library.
Known cases are "e-007" (Windows) and "e-07" (not Windows). Since
we're only showing 12 digits, and the 13th isn't close to 5, the
rest of the output should be platform-independent.
>>> exec(s) #doctest: +ELLIPSIS
-3.21716034272e-0...7
Output from calculations with Decimal should be identical across all
platforms.
>>> exec(decistmt(s))
-3.217160342717258261933904529E-7
"""
result = []
g = tokenize(BytesIO(s.encode('utf-8')).readline) # tokenize the string
for toknum, tokval, _, _, _ in g:
if toknum == NUMBER and '.' in tokval: # replace NUMBER tokens
result.extend([
(NAME, 'Decimal'),
(OP, '('),
(STRING, repr(tokval)),
(OP, ')')
])
else:
result.append((toknum, tokval))
return untokenize(result).decode('utf-8')
Example of tokenizing from the command line. The script:
def say_hello():
print("Hello, World!")
say_hello()
will be tokenized to the following output where the first column is the range of the line/column coordinates where the token is found, the second column is the name of the token, and the final column is the value of the token (if any)
$ python -m tokenize hello.py
0,0-0,0: ENCODING 'utf-8'
1,0-1,3: NAME 'def'
1,4-1,13: NAME 'say_hello'
1,13-1,14: OP '('
1,14-1,15: OP ')'
1,15-1,16: OP ':'
1,16-1,17: NEWLINE '\n'
2,0-2,4: INDENT ' '
2,4-2,9: NAME 'print'
2,9-2,10: OP '('
2,10-2,25: STRING '"Hello, World!"'
2,25-2,26: OP ')'
2,26-2,27: NEWLINE '\n'
3,0-3,1: NL '\n'
4,0-4,0: DEDENT ''
4,0-4,9: NAME 'say_hello'
4,9-4,10: OP '('
4,10-4,11: OP ')'
4,11-4,12: NEWLINE '\n'
5,0-5,0: ENDMARKER ''
The exact token type names can be displayed using the -e
option:
$ python -m tokenize -e hello.py
0,0-0,0: ENCODING 'utf-8'
1,0-1,3: NAME 'def'
1,4-1,13: NAME 'say_hello'
1,13-1,14: LPAR '('
1,14-1,15: RPAR ')'
1,15-1,16: COLON ':'
1,16-1,17: NEWLINE '\n'
2,0-2,4: INDENT ' '
2,4-2,9: NAME 'print'
2,9-2,10: LPAR '('
2,10-2,25: STRING '"Hello, World!"'
2,25-2,26: RPAR ')'
2,26-2,27: NEWLINE '\n'
3,0-3,1: NL '\n'
4,0-4,0: DEDENT ''
4,0-4,9: NAME 'say_hello'
4,9-4,10: LPAR '('
4,10-4,11: RPAR ')'
4,11-4,12: NEWLINE '\n'
5,0-5,0: ENDMARKER ''
Example of tokenizing a file programmatically, reading unicode
strings instead of bytes with generate_tokens()
:
import tokenize
with tokenize.open('hello.py') as f:
tokens = tokenize.generate_tokens(f.readline)
for token in tokens:
print(token)
Or reading bytes directly with tokenize()
:
import tokenize
with open('hello.py', 'rb') as f:
tokens = tokenize.tokenize(f.readline)
for token in tokens:
print(token)


— Testing for Python keywords

keyword
— Testing for Python keywords¶
Source code: Lib/keyword.py
This module allows a Python program to determine if a string is a keyword or soft keyword.
- keyword.kwlist¶
Sequence containing all the keywords defined for the interpreter. If any keywords are defined to only be active when particular
__future__
statements are in effect, these will be included as well.
- keyword.issoftkeyword(s)¶
Return
True
if s is a Python soft keyword.Added in version 3.9.
- keyword.softkwlist¶
Sequence containing all the soft keywords defined for the interpreter. If any soft keywords are defined to only be active when particular
__future__
statements are in effect, these will be included as well.Added in version 3.9.


token
— Constants used with Python parse trees¶
Source code: Lib/token.py
This module provides constants which represent the numeric values of leaf nodes
of the parse tree (terminal tokens). Refer to the file Grammar/Tokens
in the Python distribution for the definitions of the names in the context of
the language grammar. The specific numeric values which the names map to may
change between Python versions.
The module also provides a mapping from numeric codes to names and some functions. The functions mirror definitions in the Python C header files.
Note that a token’s value may depend on tokenizer options. For example, a
"+"
token may be reported as either PLUS
or OP
, or
a "match"
token may be either NAME
or SOFT_KEYWORD
.
- token.tok_name¶
Dictionary mapping the numeric values of the constants defined in this module back to name strings, allowing more human-readable representation of parse trees to be generated.
- token.ISTERMINAL(x)¶
Return
True
for terminal token values.
- token.ISNONTERMINAL(x)¶
Return
True
for non-terminal token values.
- token.ISEOF(x)¶
Return
True
if x is the marker indicating the end of input.
The token constants are:
- token.NAME¶
Token value that indicates an identifier or keyword.
- token.NUMBER¶
Token value that indicates a numeric literal
- token.STRING¶
Token value that indicates a string or byte literal, excluding formatted string literals. The token string is not interpreted: it includes the surrounding quotation marks and the prefix (if given); backslashes are included literally, without processing escape sequences.
- token.OP¶
A generic token value that indicates an operator or delimiter.
CPython implementation detail: This value is only reported by the
tokenize
module. Internally, the tokenizer uses exact token types instead.
- token.COMMENT¶
Token value used to indicate a comment. The parser ignores
COMMENT
tokens.
- token.NEWLINE¶
Token value that indicates the end of a logical line.
- token.NL¶
Token value used to indicate a non-terminating newline.
NL
tokens are generated when a logical line of code is continued over multiple physical lines. The parser ignoresNL
tokens.
- token.INDENT¶
Token value used at the beginning of a logical line to indicate the start of an indented block.
- token.DEDENT¶
Token value used at the beginning of a logical line to indicate the end of an indented block.
- token.FSTRING_START¶
Token value used to indicate the beginning of an f-string literal.
CPython implementation detail: The token string includes the prefix and the opening quote(s), but none of the contents of the literal.
- token.FSTRING_MIDDLE¶
Token value used for literal text inside an f-string literal, including format specifications.
CPython implementation detail: Replacement fields (that is, the non-literal parts of f-strings) use the same tokens as other expressions, and are delimited by
LBRACE
,RBRACE
,EXCLAMATION
andCOLON
tokens.
- token.FSTRING_END¶
Token value used to indicate the end of a f-string.
CPython implementation detail: The token string contains the closing quote(s).
- token.TSTRING_START¶
Token value used to indicate the beginning of a template string literal.
CPython implementation detail: The token string includes the prefix and the opening quote(s), but none of the contents of the literal.
Added in version 3.14.
- token.TSTRING_MIDDLE¶
Token value used for literal text inside a template string literal including format specifications.
CPython implementation detail: Replacement fields (that is, the non-literal parts of t-strings) use the same tokens as other expressions, and are delimited by
LBRACE
,RBRACE
,EXCLAMATION
andCOLON
tokens.Added in version 3.14.
- token.TSTRING_END¶
Token value used to indicate the end of a template string literal.
CPython implementation detail: The token string contains the closing quote(s).
Added in version 3.14.
- token.ENDMARKER¶
Token value that indicates the end of input. Used in top-level grammar rules.
- token.ENCODING¶
Token value that indicates the encoding used to decode the source bytes into text. The first token returned by
tokenize.tokenize()
will always be anENCODING
token.CPython implementation detail: This token type isn’t used by the C tokenizer but is needed for the
tokenize
module.
The following token types are not produced by the tokenize
module,
and are defined for special uses in the tokenizer or parser:
- token.TYPE_IGNORE¶
Token value indicating that a
type: ignore
comment was recognized. Such tokens are produced instead of regularCOMMENT
tokens only with thePyCF_TYPE_COMMENTS
flag.
- token.TYPE_COMMENT¶
Token value indicating that a type comment was recognized. Such tokens are produced instead of regular
COMMENT
tokens only with thePyCF_TYPE_COMMENTS
flag.
- token.SOFT_KEYWORD¶
Token value indicating a soft keyword.
The tokenizer never produces this value. To check for a soft keyword, pass a
NAME
token’s string tokeyword.issoftkeyword()
.
- token.ERRORTOKEN¶
Token value used to indicate wrong input.
The
tokenize
module generally indicates errors by raising exceptions instead of emitting this token. It can also emit tokens such asOP
orNAME
with strings that are later rejected by the parser.
The remaining tokens represent specific operators and
delimiters.
(The tokenize
module reports these as OP
; see exact_type
in the tokenize
documentation for details.)
Token |
Value |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The following non-token constants are provided:
- token.N_TOKENS¶
The number of token types defined in this module.
- token.EXACT_TOKEN_TYPES¶
A dictionary mapping the string representation of a token to its numeric code.
Added in version 3.8.
Changed in version 3.5: Added AWAIT
and ASYNC
tokens.
Changed in version 3.7: Removed AWAIT
and ASYNC
tokens. “async” and “await” are
now tokenized as NAME
tokens.
Changed in version 3.8: Added TYPE_COMMENT
, TYPE_IGNORE
, COLONEQUAL
.
Added AWAIT
and ASYNC
tokens back (they’re needed
to support parsing older Python versions for ast.parse()
with
feature_version
set to 6 or lower).
Changed in version 3.12: Added EXCLAMATION
.
Changed in version 3.13: Removed AWAIT
and ASYNC
tokens again.


symtable
— Access to the compiler’s symbol tables¶
Source code: Lib/symtable.py
Symbol tables are generated by the compiler from AST just before bytecode is
generated. The symbol table is responsible for calculating the scope of every
identifier in the code. symtable
provides an interface to examine these
tables.
Generating Symbol Tables¶
- symtable.symtable(code, filename, compile_type)¶
Return the toplevel
SymbolTable
for the Python source code. filename is the name of the file containing the code. compile_type is like the mode argument tocompile()
.
Examining Symbol Tables¶
- class symtable.SymbolTableType¶
An enumeration indicating the type of a
SymbolTable
object.- MODULE = "module"¶
Used for the symbol table of a module.
- FUNCTION = "function"¶
Used for the symbol table of a function.
- CLASS = "class"¶
Used for the symbol table of a class.
The following members refer to different flavors of annotation scopes.
- ANNOTATION = "annotation"¶
Used for annotations if
from __future__ import annotations
is active.
- TYPE_PARAMETERS = "type parameters"¶
Used for the symbol table of generic functions or generic classes.
- TYPE_VARIABLE = "type variable"¶
Used for the symbol table of the bound, the constraint tuple or the default value of a single type variable in the formal sense, i.e., a TypeVar, a TypeVarTuple or a ParamSpec object (the latter two do not support a bound or a constraint tuple).
Added in version 3.13.
- class symtable.SymbolTable¶
A namespace table for a block. The constructor is not public.
- get_type()¶
Return the type of the symbol table. Possible values are members of the
SymbolTableType
enumeration.Changed in version 3.12: Added
'annotation'
,'TypeVar bound'
,'type alias'
, and'type parameter'
as possible return values.Changed in version 3.13: Return values are members of the
SymbolTableType
enumeration.The exact values of the returned string may change in the future, and thus, it is recommended to use
SymbolTableType
members instead of hard-coded strings.
- get_id()¶
Return the table’s identifier.
- get_name()¶
Return the table’s name. This is the name of the class if the table is for a class, the name of the function if the table is for a function, or
'top'
if the table is global (get_type()
returns'module'
). For type parameter scopes (which are used for generic classes, functions, and type aliases), it is the name of the underlying class, function, or type alias. For type alias scopes, it is the name of the type alias. ForTypeVar
bound scopes, it is the name of theTypeVar
.
- get_lineno()¶
Return the number of the first line in the block this table represents.
- is_optimized()¶
Return
True
if the locals in this table can be optimized.
- is_nested()¶
Return
True
if the block is a nested class or function.
- has_children()¶
Return
True
if the block has nested namespaces within it. These can be obtained withget_children()
.
- get_identifiers()¶
Return a view object containing the names of symbols in the table. See the documentation of view objects.
- get_children()¶
Return a list of the nested symbol tables.
- class symtable.Function¶
A namespace for a function or method. This class inherits from
SymbolTable
.- get_parameters()¶
Return a tuple containing names of parameters to this function.
- get_locals()¶
Return a tuple containing names of locals in this function.
- get_globals()¶
Return a tuple containing names of globals in this function.
- get_nonlocals()¶
Return a tuple containing names of explicitly declared nonlocals in this function.
- get_frees()¶
Return a tuple containing names of free (closure) variables in this function.
- class symtable.Class¶
A namespace of a class. This class inherits from
SymbolTable
.- get_methods()¶
Return a tuple containing the names of method-like functions declared in the class.
Here, the term ‘method’ designates any function defined in the class body via
def
orasync def
.Functions defined in a deeper scope (e.g., in an inner class) are not picked up by
get_methods()
.For example:
>>> import symtable >>> st = symtable.symtable(''' ... def outer(): pass ... ... class A: ... def f(): ... def w(): pass ... ... def g(self): pass ... ... @classmethod ... async def h(cls): pass ... ... global outer ... def outer(self): pass ... ''', 'test', 'exec') >>> class_A = st.get_children()[2] >>> class_A.get_methods() ('f', 'g', 'h')
Although
A().f()
raisesTypeError
at runtime,A.f
is still considered as a method-like function.Deprecated since version 3.14, will be removed in version 3.16.
- class symtable.Symbol¶
An entry in a
SymbolTable
corresponding to an identifier in the source. The constructor is not public.- get_name()¶
Return the symbol’s name.
- is_referenced()¶
Return
True
if the symbol is used in its block.
- is_imported()¶
Return
True
if the symbol is created from an import statement.
- is_parameter()¶
Return
True
if the symbol is a parameter.
- is_type_parameter()¶
Return
True
if the symbol is a type parameter.Added in version 3.14.
- is_global()¶
Return
True
if the symbol is global.
- is_nonlocal()¶
Return
True
if the symbol is nonlocal.
- is_declared_global()¶
Return
True
if the symbol is declared global with a global statement.
- is_local()¶
Return
True
if the symbol is local to its block.
- is_annotated()¶
Return
True
if the symbol is annotated.Added in version 3.6.
- is_free()¶
Return
True
if the symbol is referenced in its block, but not assigned to.
- is_free_class()¶
Return True if a class-scoped symbol is free from the perspective of a method.
Consider the following example:
def f(): x = 1 # function-scoped class C: x = 2 # class-scoped def method(self): return x
In this example, the class-scoped symbol
x
is considered to be free from the perspective ofC.method
, thereby allowing the latter to return 1 at runtime and not 2.Added in version 3.14.
- is_assigned()¶
Return
True
if the symbol is assigned to in its block.
- is_comp_iter()¶
Return
True
if the symbol is a comprehension iteration variable.Added in version 3.14.
- is_comp_cell()¶
Return
True
if the symbol is a cell in an inlined comprehension.Added in version 3.14.
- is_namespace()¶
Return
True
if name binding introduces new namespace.If the name is used as the target of a function or class statement, this will be true.
For example:
>>> table = symtable.symtable("def some_func(): pass", "string", "exec") >>> table.lookup("some_func").is_namespace() True
Note that a single name can be bound to multiple objects. If the result is
True
, the name may also be bound to other objects, like an int or list, that does not introduce a new namespace.
- get_namespaces()¶
Return a list of namespaces bound to this name.
- get_namespace()¶
Return the namespace bound to this name. If more than one or no namespace is bound to this name, a
ValueError
is raised.
Command-Line Usage¶
Added in version 3.13.
The symtable
module can be executed as a script from the command line.
python -m symtable [infile...]
Symbol tables are generated for the specified Python source files and dumped to stdout. If no input file is specified, the content is read from stdin.