What’s New In Python 3.0

What’s New In Python 3.0¶
- Author:
Guido van Rossum
This article explains the new features in Python 3.0, compared to 2.6. Python 3.0, also known as “Python 3000” or “Py3K”, is the first ever intentionally backwards incompatible Python release. Python 3.0 was released on December 3, 2008. There are more changes than in a typical release, and more that are important for all Python users. Nevertheless, after digesting the changes, you’ll find that Python really hasn’t changed all that much – by and large, we’re mostly fixing well-known annoyances and warts, and removing a lot of old cruft.
This article doesn’t attempt to provide a complete specification of all new features, but instead tries to give a convenient overview. For full details, you should refer to the documentation for Python 3.0, and/or the many PEPs referenced in the text. If you want to understand the complete implementation and design rationale for a particular feature, PEPs usually have more details than the regular documentation; but note that PEPs usually are not kept up-to-date once a feature has been fully implemented.
Due to time constraints this document is not as complete as it should
have been. As always for a new release, the Misc/NEWS
file in the
source distribution contains a wealth of detailed information about
every small thing that was changed.
Common Stumbling Blocks¶
This section lists those few changes that are most likely to trip you up if you’re used to Python 2.5.
Print Is A Function¶
The print
statement has been replaced with a print()
function, with keyword arguments to replace most of the special syntax
of the old print
statement (PEP 3105). Examples:
Old: print "The answer is", 2*2
New: print("The answer is", 2*2)
Old: print x, # Trailing comma suppresses newline
New: print(x, end=" ") # Appends a space instead of a newline
Old: print # Prints a newline
New: print() # You must call the function!
Old: print >>sys.stderr, "fatal error"
New: print("fatal error", file=sys.stderr)
Old: print (x, y) # prints repr((x, y))
New: print((x, y)) # Not the same as print(x, y)!
You can also customize the separator between items, e.g.:
print("There are <", 2**32, "> possibilities!", sep="")
which produces:
There are <4294967296> possibilities!
Note:
The
print()
function doesn’t support the “softspace” feature of the oldprint
statement. For example, in Python 2.x,print "A\n", "B"
would write"A\nB\n"
; but in Python 3.0,print("A\n", "B")
writes"A\n B\n"
.Initially, you’ll be finding yourself typing the old
print x
a lot in interactive mode. Time to retrain your fingers to typeprint(x)
instead!When using the
2to3
source-to-source conversion tool, allprint
statements are automatically converted toprint()
function calls, so this is mostly a non-issue for larger projects.
Views And Iterators Instead Of Lists¶
Some well-known APIs no longer return lists:
dict
methodsdict.keys()
,dict.items()
anddict.values()
return “views” instead of lists. For example, this no longer works:k = d.keys(); k.sort()
. Usek = sorted(d)
instead (this works in Python 2.5 too and is just as efficient).Also, the
dict.iterkeys()
,dict.iteritems()
anddict.itervalues()
methods are no longer supported.map()
andfilter()
return iterators. If you really need a list and the input sequences are all of equal length, a quick fix is to wrapmap()
inlist()
, e.g.list(map(...))
, but a better fix is often to use a list comprehension (especially when the original code useslambda
), or rewriting the code so it doesn’t need a list at all. Particularly tricky ismap()
invoked for the side effects of the function; the correct transformation is to use a regularfor
loop (since creating a list would just be wasteful).If the input sequences are not of equal length,
map()
will stop at the termination of the shortest of the sequences. For full compatibility withmap()
from Python 2.x, also wrap the sequences initertools.zip_longest()
, e.g.map(func, *sequences)
becomeslist(map(func, itertools.zip_longest(*sequences)))
.range()
now behaves likexrange()
used to behave, except it works with values of arbitrary size. The latter no longer exists.zip()
now returns an iterator.
Ordering Comparisons¶
Python 3.0 has simplified the rules for ordering comparisons:
The ordering comparison operators (
<
,<=
,>=
,>
) raise a TypeError exception when the operands don’t have a meaningful natural ordering. Thus, expressions like1 < ''
,0 > None
orlen <= len
are no longer valid, and e.g.None < None
raisesTypeError
instead of returningFalse
. A corollary is that sorting a heterogeneous list no longer makes sense – all the elements must be comparable to each other. Note that this does not apply to the==
and!=
operators: objects of different incomparable types always compare unequal to each other.sorted()
andlist.sort()
no longer accept the cmp argument providing a comparison function. Use the key argument instead. N.B. the key and reverse arguments are now “keyword-only”.The
cmp()
function should be treated as gone, and the__cmp__()
special method is no longer supported. Use__lt__()
for sorting,__eq__()
with__hash__()
, and other rich comparisons as needed. (If you really need thecmp()
functionality, you could use the expression(a > b) - (a < b)
as the equivalent forcmp(a, b)
.)
Integers¶
PEP 237: Essentially,
long
renamed toint
. That is, there is only one built-in integral type, namedint
; but it behaves mostly like the oldlong
type.PEP 238: An expression like
1/2
returns a float. Use1//2
to get the truncating behavior. (The latter syntax has existed for years, at least since Python 2.2.)The
sys.maxint
constant was removed, since there is no longer a limit to the value of integers. However,sys.maxsize
can be used as an integer larger than any practical list or string index. It conforms to the implementation’s “natural” integer size and is typically the same assys.maxint
in previous releases on the same platform (assuming the same build options).The
repr()
of a long integer doesn’t include the trailingL
anymore, so code that unconditionally strips that character will chop off the last digit instead. (Usestr()
instead.)Octal literals are no longer of the form
0720
; use0o720
instead.
Text Vs. Data Instead Of Unicode Vs. 8-bit¶
Everything you thought you knew about binary data and Unicode has changed.
Python 3.0 uses the concepts of text and (binary) data instead of Unicode strings and 8-bit strings. All text is Unicode; however encoded Unicode is represented as binary data. The type used to hold text is
str
, the type used to hold data isbytes
. The biggest difference with the 2.x situation is that any attempt to mix text and data in Python 3.0 raisesTypeError
, whereas if you were to mix Unicode and 8-bit strings in Python 2.x, it would work if the 8-bit string happened to contain only 7-bit (ASCII) bytes, but you would getUnicodeDecodeError
if it contained non-ASCII values. This value-specific behavior has caused numerous sad faces over the years.As a consequence of this change in philosophy, pretty much all code that uses Unicode, encodings or binary data most likely has to change. The change is for the better, as in the 2.x world there were numerous bugs having to do with mixing encoded and unencoded text. To be prepared in Python 2.x, start using
unicode
for all unencoded text, andstr
for binary or encoded data only. Then the2to3
tool will do most of the work for you.You can no longer use
u"..."
literals for Unicode text. However, you must useb"..."
literals for binary data.As the
str
andbytes
types cannot be mixed, you must always explicitly convert between them. Usestr.encode()
to go fromstr
tobytes
, andbytes.decode()
to go frombytes
tostr
. You can also usebytes(s, encoding=...)
andstr(b, encoding=...)
, respectively.Like
str
, thebytes
type is immutable. There is a separate mutable type to hold buffered binary data,bytearray
. Nearly all APIs that acceptbytes
also acceptbytearray
. The mutable API is based oncollections.MutableSequence
.All backslashes in raw string literals are interpreted literally. This means that
'\U'
and'\u'
escapes in raw strings are not treated specially. For example,r'\u20ac'
is a string of 6 characters in Python 3.0, whereas in 2.6,ur'\u20ac'
was the single “euro” character. (Of course, this change only affects raw string literals; the euro character is'\u20ac'
in Python 3.0.)The built-in
basestring
abstract type was removed. Usestr
instead. Thestr
andbytes
types don’t have functionality enough in common to warrant a shared base class. The2to3
tool (see below) replaces every occurrence ofbasestring
withstr
.Files opened as text files (still the default mode for
open()
) always use an encoding to map between strings (in memory) and bytes (on disk). Binary files (opened with ab
in the mode argument) always use bytes in memory. This means that if a file is opened using an incorrect mode or encoding, I/O will likely fail loudly, instead of silently producing incorrect data. It also means that even Unix users will have to specify the correct mode (text or binary) when opening a file. There is a platform-dependent default encoding, which on Unixy platforms can be set with theLANG
environment variable (and sometimes also with some other platform-specific locale-related environment variables). In many cases, but not all, the system default is UTF-8; you should never count on this default. Any application reading or writing more than pure ASCII text should probably have a way to override the encoding. There is no longer any need for using the encoding-aware streams in thecodecs
module.The initial values of
sys.stdin
,sys.stdout
andsys.stderr
are now unicode-only text files (i.e., they are instances ofio.TextIOBase
). To read and write bytes data with these streams, you need to use theirio.TextIOBase.buffer
attribute.Filenames are passed to and returned from APIs as (Unicode) strings. This can present platform-specific problems because on some platforms filenames are arbitrary byte strings. (On the other hand, on Windows filenames are natively stored as Unicode.) As a work-around, most APIs (e.g.
open()
and many functions in theos
module) that take filenames acceptbytes
objects as well as strings, and a few APIs have a way to ask for abytes
return value. Thus,os.listdir()
returns a list ofbytes
instances if the argument is abytes
instance, andos.getcwdb()
returns the current working directory as abytes
instance. Note that whenos.listdir()
returns a list of strings, filenames that cannot be decoded properly are omitted rather than raisingUnicodeError
.Some system APIs like
os.environ
andsys.argv
can also present problems when the bytes made available by the system is not interpretable using the default encoding. Setting theLANG
variable and rerunning the program is probably the best approach.PEP 3138: The
repr()
of a string no longer escapes non-ASCII characters. It still escapes control characters and code points with non-printable status in the Unicode standard, however.PEP 3120: The default source encoding is now UTF-8.
PEP 3131: Non-ASCII letters are now allowed in identifiers. (However, the standard library remains ASCII-only with the exception of contributor names in comments.)
The
StringIO
andcStringIO
modules are gone. Instead, import theio
module and useio.StringIO
orio.BytesIO
for text and data respectively.See also the Unicode HOWTO, which was updated for Python 3.0.
Overview Of Syntax Changes¶
This section gives a brief overview of every syntactic change in Python 3.0.
New Syntax¶
PEP 3107: Function argument and return value annotations. This provides a standardized way of annotating a function’s parameters and return value. There are no semantics attached to such annotations except that they can be introspected at runtime using the
__annotations__
attribute. The intent is to encourage experimentation through metaclasses, decorators or frameworks.PEP 3102: Keyword-only arguments. Named parameters occurring after
*args
in the parameter list must be specified using keyword syntax in the call. You can also use a bare*
in the parameter list to indicate that you don’t accept a variable-length argument list, but you do have keyword-only arguments.Keyword arguments are allowed after the list of base classes in a class definition. This is used by the new convention for specifying a metaclass (see next section), but can be used for other purposes as well, as long as the metaclass supports it.
PEP 3104:
nonlocal
statement. Usingnonlocal x
you can now assign directly to a variable in an outer (but non-global) scope.nonlocal
is a new reserved word.PEP 3132: Extended Iterable Unpacking. You can now write things like
a, b, *rest = some_sequence
. And even*rest, a = stuff
. Therest
object is always a (possibly empty) list; the right-hand side may be any iterable. Example:(a, *rest, b) = range(5)
This sets a to
0
, b to4
, and rest to[1, 2, 3]
.Dictionary comprehensions:
{k: v for k, v in stuff}
means the same thing asdict(stuff)
but is more flexible. (This is PEP 274 vindicated. :-)Set literals, e.g.
{1, 2}
. Note that{}
is an empty dictionary; useset()
for an empty set. Set comprehensions are also supported; e.g.,{x for x in stuff}
means the same thing asset(stuff)
but is more flexible.New octal literals, e.g.
0o720
(already in 2.6). The old octal literals (0720
) are gone.New binary literals, e.g.
0b1010
(already in 2.6), and there is a new corresponding built-in function,bin()
.Bytes literals are introduced with a leading
b
orB
, and there is a new corresponding built-in function,bytes()
.
Changed Syntax¶
PEP 3109 and PEP 3134: new
raise
statement syntax:raise [expr [from expr]]
. See below.as
andwith
are now reserved words. (Since 2.6, actually.)True
,False
, andNone
are reserved words. (2.6 partially enforced the restrictions onNone
already.)Change from
except
exc, var toexcept
excas
var. See PEP 3110.PEP 3115: New Metaclass Syntax. Instead of:
class C: __metaclass__ = M ...
you must now use:
class C(metaclass=M): ...
The module-global
__metaclass__
variable is no longer supported. (It was a crutch to make it easier to default to new-style classes without deriving every class fromobject
.)List comprehensions no longer support the syntactic form
[... for var in item1, item2, ...]
. Use[... for var in (item1, item2, ...)]
instead. Also note that list comprehensions have different semantics: they are closer to syntactic sugar for a generator expression inside alist()
constructor, and in particular the loop control variables are no longer leaked into the surrounding scope.The ellipsis (
...
) can be used as an atomic expression anywhere. (Previously it was only allowed in slices.) Also, it must now be spelled as...
. (Previously it could also be spelled as. . .
, by a mere accident of the grammar.)
Removed Syntax¶
PEP 3113: Tuple parameter unpacking removed. You can no longer write
def foo(a, (b, c)): ...
. Usedef foo(a, b_c): b, c = b_c
instead.Removed backticks (use
repr()
instead).Removed
<>
(use!=
instead).Removed keyword:
exec()
is no longer a keyword; it remains as a function. (Fortunately the function syntax was also accepted in 2.x.) Also note thatexec()
no longer takes a stream argument; instead ofexec(f)
you can useexec(f.read())
.Integer literals no longer support a trailing
l
orL
.String literals no longer support a leading
u
orU
.The
from
moduleimport
*
syntax is only allowed at the module level, no longer inside functions.The only acceptable syntax for relative imports is
from .[module] import name
. Allimport
forms not starting with.
are interpreted as absolute imports. (PEP 328)Classic classes are gone.
Changes Already Present In Python 2.6¶
Since many users presumably make the jump straight from Python 2.5 to Python 3.0, this section reminds the reader of new features that were originally designed for Python 3.0 but that were back-ported to Python 2.6. The corresponding sections in What’s New in Python 2.6 should be consulted for longer descriptions.
PEP 343: The ‘with’ statement. The
with
statement is now a standard feature and no longer needs to be imported from the__future__
. Also check out Writing Context Managers and The contextlib module.PEP 366: Explicit Relative Imports From a Main Module. This enhances the usefulness of the
-m
option when the referenced module lives in a package.PEP 3101: Advanced String Formatting. Note: the 2.6 description mentions the
format()
method for both 8-bit and Unicode strings. In 3.0, only thestr
type (text strings with Unicode support) supports this method; thebytes
type does not. The plan is to eventually make this the only API for string formatting, and to start deprecating the%
operator in Python 3.1.PEP 3105: print As a Function. This is now a standard feature and no longer needs to be imported from
__future__
. More details were given above.PEP 3110: Exception-Handling Changes. The
except
excas
var syntax is now standard andexcept
exc, var is no longer supported. (Of course, theas
var part is still optional.)PEP 3112: Byte Literals. The
b"..."
string literal notation (and its variants likeb'...'
,b"""..."""
, andbr"..."
) now produces a literal of typebytes
.PEP 3116: New I/O Library. The
io
module is now the standard way of doing file I/O. The built-inopen()
function is now an alias forio.open()
and has additional keyword arguments encoding, errors, newline and closefd. Also note that an invalid mode argument now raisesValueError
, notIOError
. The binary file object underlying a text file object can be accessed asf.buffer
(but beware that the text object maintains a buffer of itself in order to speed up the encoding and decoding operations).PEP 3118: Revised Buffer Protocol. The old builtin
buffer()
is now really gone; the new builtinmemoryview()
provides (mostly) similar functionality.PEP 3119: Abstract Base Classes. The
abc
module and the ABCs defined in thecollections
module plays a somewhat more prominent role in the language now, and built-in collection types likedict
andlist
conform to thecollections.MutableMapping
andcollections.MutableSequence
ABCs, respectively.PEP 3127: Integer Literal Support and Syntax. As mentioned above, the new octal literal notation is the only one supported, and binary literals have been added.
PEP 3141: A Type Hierarchy for Numbers. The
numbers
module is another new use of ABCs, defining Python’s “numeric tower”. Also note the newfractions
module which implementsnumbers.Rational
.
Library Changes¶
Due to time constraints, this document does not exhaustively cover the very extensive changes to the standard library. PEP 3108 is the reference for the major changes to the library. Here’s a capsule review:
Many old modules were removed. Some, like
gopherlib
(no longer used) andmd5
(replaced byhashlib
), were already deprecated by PEP 4. Others were removed as a result of the removal of support for various platforms such as Irix, BeOS and Mac OS 9 (see PEP 11). Some modules were also selected for removal in Python 3.0 due to lack of use or because a better replacement exists. See PEP 3108 for an exhaustive list.The
bsddb3
package was removed because its presence in the core standard library has proved over time to be a particular burden for the core developers due to testing instability and Berkeley DB’s release schedule. However, the package is alive and well, externally maintained at https://www.jcea.es/programacion/pybsddb.htm.Some modules were renamed because their old name disobeyed PEP 8, or for various other reasons. Here’s the list:
Old Name
New Name
_winreg
winreg
ConfigParser
configparser
copy_reg
copyreg
Queue
queue
SocketServer
socketserver
markupbase
_markupbase
repr
reprlib
test.test_support
test.support
A common pattern in Python 2.x is to have one version of a module implemented in pure Python, with an optional accelerated version implemented as a C extension; for example,
pickle
andcPickle
. This places the burden of importing the accelerated version and falling back on the pure Python version on each user of these modules. In Python 3.0, the accelerated versions are considered implementation details of the pure Python versions. Users should always import the standard version, which attempts to import the accelerated version and falls back to the pure Python version. Thepickle
/cPickle
pair received this treatment. Theprofile
module is on the list for 3.1. TheStringIO
module has been turned into a class in theio
module.Some related modules have been grouped into packages, and usually the submodule names have been simplified. The resulting new packages are:
dbm
(anydbm
,dbhash
,dbm
,dumbdbm
,gdbm
,whichdb
).html
(HTMLParser
,htmlentitydefs
).http
(httplib
,BaseHTTPServer
,CGIHTTPServer
,SimpleHTTPServer
,Cookie
,cookielib
).tkinter
(allTkinter
-related modules exceptturtle
). The target audience ofturtle
doesn’t really care abouttkinter
. Also note that as of Python 2.6, the functionality ofturtle
has been greatly enhanced.urllib
(urllib
,urllib2
,urlparse
,robotparse
).xmlrpc
(xmlrpclib
,DocXMLRPCServer
,SimpleXMLRPCServer
).
Some other changes to standard library modules, not covered by PEP 3108:
Killed
sets
. Use the built-inset()
class.Cleanup of the
sys
module: removedsys.exitfunc()
,sys.exc_clear()
,sys.exc_type
,sys.exc_value
,sys.exc_traceback
. (Note thatsys.last_type
etc. remain.)Cleanup of the
array.array
type: theread()
andwrite()
methods are gone; usefromfile()
andtofile()
instead. Also, the'c'
typecode for array is gone – use either'b'
for bytes or'u'
for Unicode characters.Cleanup of the
operator
module: removedsequenceIncludes()
andisCallable()
.Cleanup of the
thread
module:acquire_lock()
andrelease_lock()
are gone; useacquire()
andrelease()
instead.Cleanup of the
random
module: removed thejumpahead()
API.The
new
module is gone.The functions
os.tmpnam()
,os.tempnam()
andos.tmpfile()
have been removed in favor of thetempfile
module.The
tokenize
module has been changed to work with bytes. The main entry point is nowtokenize.tokenize()
, instead of generate_tokens.string.letters
and its friends (string.lowercase
andstring.uppercase
) are gone. Usestring.ascii_letters
etc. instead. (The reason for the removal is thatstring.letters
and friends had locale-specific behavior, which is a bad idea for such attractively named global “constants”.)Renamed module
__builtin__
tobuiltins
(removing the underscores, adding an ‘s’). The__builtins__
variable found in most global namespaces is unchanged. To modify a builtin, you should usebuiltins
, not__builtins__
!
PEP 3101: A New Approach To String Formatting¶
A new system for built-in string formatting operations replaces the
%
string formatting operator. (However, the%
operator is still supported; it will be deprecated in Python 3.1 and removed from the language at some later time.) Read PEP 3101 for the full scoop.
Changes To Exceptions¶
The APIs for raising and catching exception have been cleaned up and new powerful features added:
PEP 352: All exceptions must be derived (directly or indirectly) from
BaseException
. This is the root of the exception hierarchy. This is not new as a recommendation, but the requirement to inherit fromBaseException
is new. (Python 2.6 still allowed classic classes to be raised, and placed no restriction on what you can catch.) As a consequence, string exceptions are finally truly and utterly dead.Almost all exceptions should actually derive from
Exception
;BaseException
should only be used as a base class for exceptions that should only be handled at the top level, such asSystemExit
orKeyboardInterrupt
. The recommended idiom for handling all exceptions except for this latter category is to useexcept
Exception
.StandardError
was removed.Exceptions no longer behave as sequences. Use the
args
attribute instead.PEP 3109: Raising exceptions. You must now use
raise Exception(args)
instead ofraise Exception, args
. Additionally, you can no longer explicitly specify a traceback; instead, if you have to do this, you can assign directly to the__traceback__
attribute (see below).PEP 3110: Catching exceptions. You must now use
except SomeException as variable
instead ofexcept SomeException, variable
. Moreover, the variable is explicitly deleted when theexcept
block is left.PEP 3134: Exception chaining. There are two cases: implicit chaining and explicit chaining. Implicit chaining happens when an exception is raised in an
except
orfinally
handler block. This usually happens due to a bug in the handler block; we call this a secondary exception. In this case, the original exception (that was being handled) is saved as the__context__
attribute of the secondary exception. Explicit chaining is invoked with this syntax:raise SecondaryException() from primary_exception
(where primary_exception is any expression that produces an exception object, probably an exception that was previously caught). In this case, the primary exception is stored on the
__cause__
attribute of the secondary exception. The traceback printed when an unhandled exception occurs walks the chain of__cause__
and__context__
attributes and prints a separate traceback for each component of the chain, with the primary exception at the top. (Java users may recognize this behavior.)PEP 3134: Exception objects now store their traceback as the
__traceback__
attribute. This means that an exception object now contains all the information pertaining to an exception, and there are fewer reasons to usesys.exc_info()
(though the latter is not removed).A few exception messages are improved when Windows fails to load an extension module. For example,
error code 193
is now%1 is not a valid Win32 application
. Strings now deal with non-English locales.
Miscellaneous Other Changes¶
Operators And Special Methods¶
!=
now returns the opposite of==
, unless==
returnsNotImplemented
.The concept of “unbound methods” has been removed from the language. When referencing a method as a class attribute, you now get a plain function object.
__getslice__()
,__setslice__()
and__delslice__()
were killed. The syntaxa[i:j]
now translates toa.__getitem__(slice(i, j))
(or__setitem__()
or__delitem__()
, when used as an assignment or deletion target, respectively).PEP 3114: the standard
next()
method has been renamed to__next__()
.The
__oct__()
and__hex__()
special methods are removed –oct()
andhex()
use__index__()
now to convert the argument to an integer.Removed support for
__members__
and__methods__
.The function attributes named
func_X
have been renamed to use the__X__
form, freeing up these names in the function attribute namespace for user-defined attributes. To wit,func_closure
,func_code
,func_defaults
,func_dict
,func_doc
,func_globals
,func_name
were renamed to__closure__
,__code__
,__defaults__
,__dict__
,__doc__
,__globals__
,__name__
, respectively.__nonzero__()
is now__bool__()
.
Builtins¶
PEP 3135: New
super()
. You can now invokesuper()
without arguments and (assuming this is in a regular instance method defined inside aclass
statement) the right class and instance will automatically be chosen. With arguments, the behavior ofsuper()
is unchanged.PEP 3111:
raw_input()
was renamed toinput()
. That is, the newinput()
function reads a line fromsys.stdin
and returns it with the trailing newline stripped. It raisesEOFError
if the input is terminated prematurely. To get the old behavior ofinput()
, useeval(input())
.A new built-in function
next()
was added to call the__next__()
method on an object.The
round()
function rounding strategy and return type have changed. Exact halfway cases are now rounded to the nearest even result instead of away from zero. (For example,round(2.5)
now returns2
rather than3
.)round(x[, n])
now delegates tox.__round__([n])
instead of always returning a float. It generally returns an integer when called with a single argument and a value of the same type asx
when called with two arguments.Moved
intern()
tosys.intern()
.Removed:
apply()
. Instead ofapply(f, args)
usef(*args)
.Removed
callable()
. Instead ofcallable(f)
you can useisinstance(f, collections.Callable)
. Theoperator.isCallable()
function is also gone.Removed
coerce()
. This function no longer serves a purpose now that classic classes are gone.Removed
execfile()
. Instead ofexecfile(fn)
useexec(open(fn).read())
.Removed the
file
type. Useopen()
. There are now several different kinds of streams that open can return in theio
module.Removed
reduce()
. Usefunctools.reduce()
if you really need it; however, 99 percent of the time an explicitfor
loop is more readable.Removed
reload()
. Useimp.reload()
.Removed.
dict.has_key()
– use thein
operator instead.
Build and C API Changes¶
Due to time constraints, here is a very incomplete list of changes to the C API.
Support for several platforms was dropped, including but not limited to Mac OS 9, BeOS, RISCOS, Irix, and Tru64.
PEP 3118: New Buffer API.
PEP 3121: Extension Module Initialization & Finalization.
PEP 3123: Making
PyObject_HEAD
conform to standard C.No more C API support for restricted execution.
PyNumber_Coerce()
,PyNumber_CoerceEx()
,PyMember_Get()
, andPyMember_Set()
C APIs are removed.New C API
PyImport_ImportModuleNoBlock()
, works likePyImport_ImportModule()
but won’t block on the import lock (returning an error instead).Renamed the boolean conversion C-level slot and method:
nb_nonzero
is nownb_bool
.Removed
METH_OLDARGS
andWITH_CYCLE_GC
from the C API.
Performance¶
The net result of the 3.0 generalizations is that Python 3.0 runs the pystone benchmark around 10% slower than Python 2.5. Most likely the biggest cause is the removal of special-casing for small integers. There’s room for improvement, but it will happen after 3.0 is released!
Porting To Python 3.0¶
For porting existing Python 2.5 or 2.6 source code to Python 3.0, the best strategy is the following:
(Prerequisite:) Start with excellent test coverage.
Port to Python 2.6. This should be no more work than the average port from Python 2.x to Python 2.(x+1). Make sure all your tests pass.
(Still using 2.6:) Turn on the
-3
command line switch. This enables warnings about features that will be removed (or change) in 3.0. Run your test suite again, and fix code that you get warnings about until there are no warnings left, and all your tests still pass.Run the
2to3
source-to-source translator over your source code tree. Run the result of the translation under Python 3.0. Manually fix up any remaining issues, fixing problems until all tests pass again.
It is not recommended to try to write source code that runs unchanged
under both Python 2.6 and 3.0; you’d have to use a very contorted
coding style, e.g. avoiding print
statements, metaclasses,
and much more. If you are maintaining a library that needs to support
both Python 2.6 and Python 3.0, the best approach is to modify step 3
above by editing the 2.6 version of the source code and running the
2to3
translator again, rather than editing the 3.0 version of the
source code.
For porting C extensions to Python 3.0, please see Porting Extension Modules to Python 3.


What’s New In Python 3.1

What’s New In Python 3.1¶
- Author:
Raymond Hettinger
This article explains the new features in Python 3.1, compared to 3.0. Python 3.1 was released on June 27, 2009.
PEP 372: Ordered Dictionaries¶
Regular Python dictionaries iterate over key/value pairs in arbitrary order.
Over the years, a number of authors have written alternative implementations
that remember the order that the keys were originally inserted. Based on
the experiences from those implementations, a new
collections.OrderedDict
class has been introduced.
The OrderedDict API is substantially the same as regular dictionaries but will iterate over keys and values in a guaranteed order depending on when a key was first inserted. If a new entry overwrites an existing entry, the original insertion position is left unchanged. Deleting an entry and reinserting it will move it to the end.
The standard library now supports use of ordered dictionaries in several
modules. The configparser
module uses them by default. This lets
configuration files be read, modified, and then written back in their original
order. The _asdict() method for collections.namedtuple()
now
returns an ordered dictionary with the values appearing in the same order as
the underlying tuple indices. The json
module is being built-out with
an object_pairs_hook to allow OrderedDicts to be built by the decoder.
Support was also added for third-party tools like PyYAML.
See also
- PEP 372 - Ordered Dictionaries
PEP written by Armin Ronacher and Raymond Hettinger. Implementation written by Raymond Hettinger.
Since an ordered dictionary remembers its insertion order, it can be used in conjunction with sorting to make a sorted dictionary:
>>> # regular unsorted dictionary
>>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2}
>>> # dictionary sorted by key
>>> OrderedDict(sorted(d.items(), key=lambda t: t[0]))
OrderedDict([('apple', 4), ('banana', 3), ('orange', 2), ('pear', 1)])
>>> # dictionary sorted by value
>>> OrderedDict(sorted(d.items(), key=lambda t: t[1]))
OrderedDict([('pear', 1), ('orange', 2), ('banana', 3), ('apple', 4)])
>>> # dictionary sorted by length of the key string
>>> OrderedDict(sorted(d.items(), key=lambda t: len(t[0])))
OrderedDict([('pear', 1), ('apple', 4), ('orange', 2), ('banana', 3)])
The new sorted dictionaries maintain their sort order when entries are deleted. But when new keys are added, the keys are appended to the end and the sort is not maintained.
PEP 378: Format Specifier for Thousands Separator¶
The built-in format()
function and the str.format()
method use
a mini-language that now includes a simple, non-locale aware way to format
a number with a thousands separator. That provides a way to humanize a
program’s output, improving its professional appearance and readability:
>>> format(1234567, ',d')
'1,234,567'
>>> format(1234567.89, ',.2f')
'1,234,567.89'
>>> format(12345.6 + 8901234.12j, ',f')
'12,345.600000+8,901,234.120000j'
>>> format(Decimal('1234567.89'), ',f')
'1,234,567.89'
The supported types are int
, float
, complex
and decimal.Decimal
.
Discussions are underway about how to specify alternative separators like dots, spaces, apostrophes, or underscores. Locale-aware applications should use the existing n format specifier which already has some support for thousands separators.
See also
- PEP 378 - Format Specifier for Thousands Separator
PEP written by Raymond Hettinger and implemented by Eric Smith and Mark Dickinson.
Other Language Changes¶
Some smaller changes made to the core Python language are:
Directories and zip archives containing a
__main__.py
file can now be executed directly by passing their name to the interpreter. The directory/zipfile is automatically inserted as the first entry in sys.path. (Suggestion and initial patch by Andy Chu; revised patch by Phillip J. Eby and Nick Coghlan; bpo-1739468.)The
int()
type gained abit_length
method that returns the number of bits necessary to represent its argument in binary:>>> n = 37 >>> bin(37) '0b100101' >>> n.bit_length() 6 >>> n = 2**123-1 >>> n.bit_length() 123 >>> (n+1).bit_length() 124
(Contributed by Fredrik Johansson, Victor Stinner, Raymond Hettinger, and Mark Dickinson; bpo-3439.)
The fields in
format()
strings can now be automatically numbered:>>> 'Sir {} of {}'.format('Gallahad', 'Camelot') 'Sir Gallahad of Camelot'
Formerly, the string would have required numbered fields such as:
'Sir {0} of {1}'
.(Contributed by Eric Smith; bpo-5237.)
The
string.maketrans()
function is deprecated and is replaced by new static methods,bytes.maketrans()
andbytearray.maketrans()
. This change solves the confusion around which types were supported by thestring
module. Now,str
,bytes
, andbytearray
each have their own maketrans and translate methods with intermediate translation tables of the appropriate type.(Contributed by Georg Brandl; bpo-5675.)
The syntax of the
with
statement now allows multiple context managers in a single statement:>>> with open('mylog.txt') as infile, open('a.out', 'w') as outfile: ... for line in infile: ... if '<critical>' in line: ... outfile.write(line)
With the new syntax, the
contextlib.nested()
function is no longer needed and is now deprecated.(Contributed by Georg Brandl and Mattias Brändström; appspot issue 53094.)
round(x, n)
now returns an integer if x is an integer. Previously it returned a float:>>> round(1123, -2) 1100
(Contributed by Mark Dickinson; bpo-4707.)
Python now uses David Gay’s algorithm for finding the shortest floating-point representation that doesn’t change its value. This should help mitigate some of the confusion surrounding binary floating-point numbers.
The significance is easily seen with a number like
1.1
which does not have an exact equivalent in binary floating point. Since there is no exact equivalent, an expression likefloat('1.1')
evaluates to the nearest representable value which is0x1.199999999999ap+0
in hex or1.100000000000000088817841970012523233890533447265625
in decimal. That nearest value was and still is used in subsequent floating-point calculations.What is new is how the number gets displayed. Formerly, Python used a simple approach. The value of
repr(1.1)
was computed asformat(1.1, '.17g')
which evaluated to'1.1000000000000001'
. The advantage of using 17 digits was that it relied on IEEE-754 guarantees to assure thateval(repr(1.1))
would round-trip exactly to its original value. The disadvantage is that many people found the output to be confusing (mistaking intrinsic limitations of binary floating-point representation as being a problem with Python itself).The new algorithm for
repr(1.1)
is smarter and returns'1.1'
. Effectively, it searches all equivalent string representations (ones that get stored with the same underlying float value) and returns the shortest representation.The new algorithm tends to emit cleaner representations when possible, but it does not change the underlying values. So, it is still the case that
1.1 + 2.2 != 3.3
even though the representations may suggest otherwise.The new algorithm depends on certain features in the underlying floating-point implementation. If the required features are not found, the old algorithm will continue to be used. Also, the text pickle protocols assure cross-platform portability by using the old algorithm.
(Contributed by Eric Smith and Mark Dickinson; bpo-1580)
New, Improved, and Deprecated Modules¶
Added a
collections.Counter
class to support convenient counting of unique items in a sequence or iterable:>>> Counter(['red', 'blue', 'red', 'green', 'blue', 'blue']) Counter({'blue': 3, 'red': 2, 'green': 1})
(Contributed by Raymond Hettinger; bpo-1696199.)
Added a new module,
tkinter.ttk
for access to the Tk themed widget set. The basic idea of ttk is to separate, to the extent possible, the code implementing a widget’s behavior from the code implementing its appearance.(Contributed by Guilherme Polo; bpo-2983.)
The
gzip.GzipFile
andbz2.BZ2File
classes now support the context management protocol:>>> # Automatically close file after writing >>> with gzip.GzipFile(filename, "wb") as f: ... f.write(b"xxx")
(Contributed by Antoine Pitrou.)
The
decimal
module now supports methods for creating a decimal object from a binaryfloat
. The conversion is exact but can sometimes be surprising:>>> Decimal.from_float(1.1) Decimal('1.100000000000000088817841970012523233890533447265625')
The long decimal result shows the actual binary fraction being stored for 1.1. The fraction has many digits because 1.1 cannot be exactly represented in binary.
(Contributed by Raymond Hettinger and Mark Dickinson.)
The
itertools
module grew two new functions. Theitertools.combinations_with_replacement()
function is one of four for generating combinatorics including permutations and Cartesian products. Theitertools.compress()
function mimics its namesake from APL. Also, the existingitertools.count()
function now has an optional step argument and can accept any type of counting sequence includingfractions.Fraction
anddecimal.Decimal
:>>> [p+q for p,q in combinations_with_replacement('LOVE', 2)] ['LL', 'LO', 'LV', 'LE', 'OO', 'OV', 'OE', 'VV', 'VE', 'EE'] >>> list(compress(data=range(10), selectors=[0,0,1,1,0,1,0,1,0,0])) [2, 3, 5, 7] >>> c = count(start=Fraction(1,2), step=Fraction(1,6)) >>> [next(c), next(c), next(c), next(c)] [Fraction(1, 2), Fraction(2, 3), Fraction(5, 6), Fraction(1, 1)]
(Contributed by Raymond Hettinger.)
collections.namedtuple()
now supports a keyword argument rename which lets invalid fieldnames be automatically converted to positional names in the form _0, _1, etc. This is useful when the field names are being created by an external source such as a CSV header, SQL field list, or user input:>>> query = input() SELECT region, dept, count(*) FROM main GROUPBY region, dept >>> cursor.execute(query) >>> query_fields = [desc[0] for desc in cursor.description] >>> UserQuery = namedtuple('UserQuery', query_fields, rename=True) >>> pprint.pprint([UserQuery(*row) for row in cursor]) [UserQuery(region='South', dept='Shipping', _2=185), UserQuery(region='North', dept='Accounting', _2=37), UserQuery(region='West', dept='Sales', _2=419)]
(Contributed by Raymond Hettinger; bpo-1818.)
The
re.sub()
,re.subn()
andre.split()
functions now accept a flags parameter.(Contributed by Gregory Smith.)
The
logging
module now implements a simplelogging.NullHandler
class for applications that are not using logging but are calling library code that does. Setting-up a null handler will suppress spurious warnings such as “No handlers could be found for logger foo”:>>> h = logging.NullHandler() >>> logging.getLogger("foo").addHandler(h)
(Contributed by Vinay Sajip; bpo-4384).
The
runpy
module which supports the-m
command line switch now supports the execution of packages by looking for and executing a__main__
submodule when a package name is supplied.(Contributed by Andi Vajda; bpo-4195.)
The
pdb
module can now access and display source code loaded viazipimport
(or any other conformant PEP 302 loader).(Contributed by Alexander Belopolsky; bpo-4201.)
functools.partial
objects can now be pickled.
(Suggested by Antoine Pitrou and Jesse Noller. Implemented by Jack Diederich; bpo-5228.)
Add
pydoc
help topics for symbols so thathelp('@')
works as expected in the interactive environment.(Contributed by David Laban; bpo-4739.)
The
unittest
module now supports skipping individual tests or classes of tests. And it supports marking a test as an expected failure, a test that is known to be broken, but shouldn’t be counted as a failure on a TestResult:class TestGizmo(unittest.TestCase): @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") def test_gizmo_on_windows(self): ... @unittest.expectedFailure def test_gimzo_without_required_library(self): ...
Also, tests for exceptions have been builtout to work with context managers using the
with
statement:def test_division_by_zero(self): with self.assertRaises(ZeroDivisionError): x / 0
In addition, several new assertion methods were added including
assertSetEqual()
,assertDictEqual()
,assertDictContainsSubset()
,assertListEqual()
,assertTupleEqual()
,assertSequenceEqual()
,assertRaisesRegexp()
,assertIsNone()
, andassertIsNotNone()
.(Contributed by Benjamin Peterson and Antoine Pitrou.)
The
io
module has three new constants for theseek()
method:SEEK_SET
,SEEK_CUR
, andSEEK_END
.The
sys.version_info
tuple is now a named tuple:>>> sys.version_info sys.version_info(major=3, minor=1, micro=0, releaselevel='alpha', serial=2)
(Contributed by Ross Light; bpo-4285.)
The
nntplib
andimaplib
modules now support IPv6.The
pickle
module has been adapted for better interoperability with Python 2.x when used with protocol 2 or lower. The reorganization of the standard library changed the formal reference for many objects. For example,__builtin__.set
in Python 2 is calledbuiltins.set
in Python 3. This change confounded efforts to share data between different versions of Python. But now when protocol 2 or lower is selected, the pickler will automatically use the old Python 2 names for both loading and dumping. This remapping is turned-on by default but can be disabled with the fix_imports option:>>> s = {1, 2, 3} >>> pickle.dumps(s, protocol=0) b'c__builtin__\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.' >>> pickle.dumps(s, protocol=0, fix_imports=False) b'cbuiltins\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.'
An unfortunate but unavoidable side-effect of this change is that protocol 2 pickles produced by Python 3.1 won’t be readable with Python 3.0. The latest pickle protocol, protocol 3, should be used when migrating data between Python 3.x implementations, as it doesn’t attempt to remain compatible with Python 2.x.
(Contributed by Alexandre Vassalotti and Antoine Pitrou, bpo-6137.)
A new module,
importlib
was added. It provides a complete, portable, pure Python reference implementation of theimport
statement and its counterpart, the__import__()
function. It represents a substantial step forward in documenting and defining the actions that take place during imports.(Contributed by Brett Cannon.)
Optimizations¶
Major performance enhancements have been added:
The new I/O library (as defined in PEP 3116) was mostly written in Python and quickly proved to be a problematic bottleneck in Python 3.0. In Python 3.1, the I/O library has been entirely rewritten in C and is 2 to 20 times faster depending on the task at hand. The pure Python version is still available for experimentation purposes through the
_pyio
module.(Contributed by Amaury Forgeot d’Arc and Antoine Pitrou.)
Added a heuristic so that tuples and dicts containing only untrackable objects are not tracked by the garbage collector. This can reduce the size of collections and therefore the garbage collection overhead on long-running programs, depending on their particular use of datatypes.
(Contributed by Antoine Pitrou, bpo-4688.)
Enabling a configure option named
--with-computed-gotos
on compilers that support it (notably: gcc, SunPro, icc), the bytecode evaluation loop is compiled with a new dispatch mechanism which gives speedups of up to 20%, depending on the system, the compiler, and the benchmark.(Contributed by Antoine Pitrou along with a number of other participants, bpo-4753).
The decoding of UTF-8, UTF-16 and LATIN-1 is now two to four times faster.
(Contributed by Antoine Pitrou and Amaury Forgeot d’Arc, bpo-4868.)
The
json
module now has a C extension to substantially improve its performance. In addition, the API was modified so that json works only withstr
, not withbytes
. That change makes the module closely match the JSON specification which is defined in terms of Unicode.(Contributed by Bob Ippolito and converted to Py3.1 by Antoine Pitrou and Benjamin Peterson; bpo-4136.)
Unpickling now interns the attribute names of pickled objects. This saves memory and allows pickles to be smaller.
(Contributed by Jake McGuire and Antoine Pitrou; bpo-5084.)
IDLE¶
IDLE’s format menu now provides an option to strip trailing whitespace from a source file.
(Contributed by Roger D. Serwy; bpo-5150.)
Build and C API Changes¶
Changes to Python’s build process and to the C API include:
Integers are now stored internally either in base
2**15
or in base2**30
, the base being determined at build time. Previously, they were always stored in base2**15
. Using base2**30
gives significant performance improvements on 64-bit machines, but benchmark results on 32-bit machines have been mixed. Therefore, the default is to use base2**30
on 64-bit machines and base2**15
on 32-bit machines; on Unix, there’s a new configure option--enable-big-digits
that can be used to override this default.Apart from the performance improvements this change should be invisible to end users, with one exception: for testing and debugging purposes there’s a new
sys.int_info
that provides information about the internal format, giving the number of bits per digit and the size in bytes of the C type used to store each digit:>>> import sys >>> sys.int_info sys.int_info(bits_per_digit=30, sizeof_digit=4)
(Contributed by Mark Dickinson; bpo-4258.)
The
PyLong_AsUnsignedLongLong()
function now handles a negative pylong by raisingOverflowError
instead ofTypeError
.(Contributed by Mark Dickinson and Lisandro Dalcrin; bpo-5175.)
Deprecated
PyNumber_Int()
. UsePyNumber_Long()
instead.(Contributed by Mark Dickinson; bpo-4910.)
Added a new
PyOS_string_to_double()
function to replace the deprecated functionsPyOS_ascii_strtod()
andPyOS_ascii_atof()
.(Contributed by Mark Dickinson; bpo-5914.)
Added
PyCapsule
as a replacement for thePyCObject
API. The principal difference is that the new type has a well defined interface for passing typing safety information and a less complicated signature for calling a destructor. The old type had a problematic API and is now deprecated.(Contributed by Larry Hastings; bpo-5630.)
Porting to Python 3.1¶
This section lists previously described changes and other bugfixes that may require changes to your code:
The new floating-point string representations can break existing doctests. For example:
def e(): '''Compute the base of natural logarithms. >>> e() 2.7182818284590451 ''' return sum(1/math.factorial(x) for x in reversed(range(30))) doctest.testmod() ********************************************************************** Failed example: e() Expected: 2.7182818284590451 Got: 2.718281828459045 **********************************************************************
The automatic name remapping in the pickle module for protocol 2 or lower can make Python 3.1 pickles unreadable in Python 3.0. One solution is to use protocol 3. Another solution is to set the fix_imports option to
False
. See the discussion above for more details.