Enum HOWTO

Enum HOWTO¶
An Enum
is a set of symbolic names bound to unique values. They are
similar to global variables, but they offer a more useful repr()
,
grouping, type-safety, and a few other features.
They are most useful when you have a variable that can take one of a limited selection of values. For example, the days of the week:
>>> from enum import Enum
>>> class Weekday(Enum):
... MONDAY = 1
... TUESDAY = 2
... WEDNESDAY = 3
... THURSDAY = 4
... FRIDAY = 5
... SATURDAY = 6
... SUNDAY = 7
Or perhaps the RGB primary colors:
>>> from enum import Enum
>>> class Color(Enum):
... RED = 1
... GREEN = 2
... BLUE = 3
As you can see, creating an Enum
is as simple as writing a class that
inherits from Enum
itself.
Note
Case of Enum Members
Because Enums are used to represent constants, and to help avoid issues with name clashes between mixin-class methods/attributes and enum names, we strongly recommend using UPPER_CASE names for members, and will be using that style in our examples.
Depending on the nature of the enum a member’s value may or may not be important, but either way that value can be used to get the corresponding member:
>>> Weekday(3)
<Weekday.WEDNESDAY: 3>
As you can see, the repr()
of a member shows the enum name, the member name,
and the value. The str()
of a member shows only the enum name and member
name:
>>> print(Weekday.THURSDAY)
Weekday.THURSDAY
The type of an enumeration member is the enum it belongs to:
>>> type(Weekday.MONDAY)
<enum 'Weekday'>
>>> isinstance(Weekday.FRIDAY, Weekday)
True
Enum members have an attribute that contains just their name
:
>>> print(Weekday.TUESDAY.name)
TUESDAY
Likewise, they have an attribute for their value
:
>>> Weekday.WEDNESDAY.value
3
Unlike many languages that treat enumerations solely as name/value pairs,
Python Enums can have behavior added. For example, datetime.date
has two methods for returning the weekday:
weekday()
and isoweekday()
.
The difference is that one of them counts from 0-6 and the other from 1-7.
Rather than keep track of that ourselves we can add a method to the Weekday
enum to extract the day from the date
instance and return the matching
enum member:
@classmethod
def from_date(cls, date):
return cls(date.isoweekday())
The complete Weekday
enum now looks like this:
>>> class Weekday(Enum):
... MONDAY = 1
... TUESDAY = 2
... WEDNESDAY = 3
... THURSDAY = 4
... FRIDAY = 5
... SATURDAY = 6
... SUNDAY = 7
... #
... @classmethod
... def from_date(cls, date):
... return cls(date.isoweekday())
Now we can find out what today is! Observe:
>>> from datetime import date
>>> Weekday.from_date(date.today())
<Weekday.TUESDAY: 2>
Of course, if you’re reading this on some other day, you’ll see that day instead.
This Weekday
enum is great if our variable only needs one day, but
what if we need several? Maybe we’re writing a function to plot chores during
a week, and don’t want to use a list
– we could use a different type
of Enum
:
>>> from enum import Flag
>>> class Weekday(Flag):
... MONDAY = 1
... TUESDAY = 2
... WEDNESDAY = 4
... THURSDAY = 8
... FRIDAY = 16
... SATURDAY = 32
... SUNDAY = 64
We’ve changed two things: we’re inherited from Flag
, and the values are
all powers of 2.
Just like the original Weekday
enum above, we can have a single selection:
>>> first_week_day = Weekday.MONDAY
>>> first_week_day
<Weekday.MONDAY: 1>
But Flag
also allows us to combine several members into a single
variable:
>>> weekend = Weekday.SATURDAY | Weekday.SUNDAY
>>> weekend
<Weekday.SATURDAY|SUNDAY: 96>
You can even iterate over a Flag
variable:
>>> for day in weekend:
... print(day)
Weekday.SATURDAY
Weekday.SUNDAY
Okay, let’s get some chores set up:
>>> chores_for_ethan = {
... 'feed the cat': Weekday.MONDAY | Weekday.WEDNESDAY | Weekday.FRIDAY,
... 'do the dishes': Weekday.TUESDAY | Weekday.THURSDAY,
... 'answer SO questions': Weekday.SATURDAY,
... }
And a function to display the chores for a given day:
>>> def show_chores(chores, day):
... for chore, days in chores.items():
... if day in days:
... print(chore)
...
>>> show_chores(chores_for_ethan, Weekday.SATURDAY)
answer SO questions
In cases where the actual values of the members do not matter, you can save
yourself some work and use auto()
for the values:
>>> from enum import auto
>>> class Weekday(Flag):
... MONDAY = auto()
... TUESDAY = auto()
... WEDNESDAY = auto()
... THURSDAY = auto()
... FRIDAY = auto()
... SATURDAY = auto()
... SUNDAY = auto()
... WEEKEND = SATURDAY | SUNDAY
Programmatic access to enumeration members and their attributes¶
Sometimes it’s useful to access members in enumerations programmatically (i.e.
situations where Color.RED
won’t do because the exact color is not known
at program-writing time). Enum
allows such access:
>>> Color(1)
<Color.RED: 1>
>>> Color(3)
<Color.BLUE: 3>
If you want to access enum members by name, use item access:
>>> Color['RED']
<Color.RED: 1>
>>> Color['GREEN']
<Color.GREEN: 2>
If you have an enum member and need its name
or value
:
>>> member = Color.RED
>>> member.name
'RED'
>>> member.value
1
Duplicating enum members and values¶
Having two enum members with the same name is invalid:
>>> class Shape(Enum):
... SQUARE = 2
... SQUARE = 3
...
Traceback (most recent call last):
...
TypeError: 'SQUARE' already defined as 2
However, an enum member can have other names associated with it. Given two
entries A
and B
with the same value (and A
defined first), B
is an alias for the member A
. By-value lookup of the value of A
will
return the member A
. By-name lookup of A
will return the member A
.
By-name lookup of B
will also return the member A
:
>>> class Shape(Enum):
... SQUARE = 2
... DIAMOND = 1
... CIRCLE = 3
... ALIAS_FOR_SQUARE = 2
...
>>> Shape.SQUARE
<Shape.SQUARE: 2>
>>> Shape.ALIAS_FOR_SQUARE
<Shape.SQUARE: 2>
>>> Shape(2)
<Shape.SQUARE: 2>
Note
Attempting to create a member with the same name as an already defined attribute (another member, a method, etc.) or attempting to create an attribute with the same name as a member is not allowed.
Ensuring unique enumeration values¶
By default, enumerations allow multiple names as aliases for the same value.
When this behavior isn’t desired, you can use the unique()
decorator:
>>> from enum import Enum, unique
>>> @unique
... class Mistake(Enum):
... ONE = 1
... TWO = 2
... THREE = 3
... FOUR = 3
...
Traceback (most recent call last):
...
ValueError: duplicate values found in <enum 'Mistake'>: FOUR -> THREE
Using automatic values¶
If the exact value is unimportant you can use auto
:
>>> from enum import Enum, auto
>>> class Color(Enum):
... RED = auto()
... BLUE = auto()
... GREEN = auto()
...
>>> [member.value for member in Color]
[1, 2, 3]
The values are chosen by _generate_next_value_()
, which can be
overridden:
>>> class AutoName(Enum):
... @staticmethod
... def _generate_next_value_(name, start, count, last_values):
... return name
...
>>> class Ordinal(AutoName):
... NORTH = auto()
... SOUTH = auto()
... EAST = auto()
... WEST = auto()
...
>>> [member.value for member in Ordinal]
['NORTH', 'SOUTH', 'EAST', 'WEST']
Note
The _generate_next_value_()
method must be defined before any members.
Iteration¶
Iterating over the members of an enum does not provide the aliases:
>>> list(Shape)
[<Shape.SQUARE: 2>, <Shape.DIAMOND: 1>, <Shape.CIRCLE: 3>]
>>> list(Weekday)
[<Weekday.MONDAY: 1>, <Weekday.TUESDAY: 2>, <Weekday.WEDNESDAY: 4>, <Weekday.THURSDAY: 8>, <Weekday.FRIDAY: 16>, <Weekday.SATURDAY: 32>, <Weekday.SUNDAY: 64>]
Note that the aliases Shape.ALIAS_FOR_SQUARE
and Weekday.WEEKEND
aren’t shown.
The special attribute __members__
is a read-only ordered mapping of names
to members. It includes all names defined in the enumeration, including the
aliases:
>>> for name, member in Shape.__members__.items():
... name, member
...
('SQUARE', <Shape.SQUARE: 2>)
('DIAMOND', <Shape.DIAMOND: 1>)
('CIRCLE', <Shape.CIRCLE: 3>)
('ALIAS_FOR_SQUARE', <Shape.SQUARE: 2>)
The __members__
attribute can be used for detailed programmatic access to
the enumeration members. For example, finding all the aliases:
>>> [name for name, member in Shape.__members__.items() if member.name != name]
['ALIAS_FOR_SQUARE']
Note
Aliases for flags include values with multiple flags set, such as 3
,
and no flags set, i.e. 0
.
Comparisons¶
Enumeration members are compared by identity:
>>> Color.RED is Color.RED
True
>>> Color.RED is Color.BLUE
False
>>> Color.RED is not Color.BLUE
True
Ordered comparisons between enumeration values are not supported. Enum members are not integers (but see IntEnum below):
>>> Color.RED < Color.BLUE
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'Color' and 'Color'
Equality comparisons are defined though:
>>> Color.BLUE == Color.RED
False
>>> Color.BLUE != Color.RED
True
>>> Color.BLUE == Color.BLUE
True
Comparisons against non-enumeration values will always compare not equal
(again, IntEnum
was explicitly designed to behave differently, see
below):
>>> Color.BLUE == 2
False
Warning
It is possible to reload modules – if a reloaded module contains enums, they will be recreated, and the new members may not compare identical/equal to the original members.
Allowed members and attributes of enumerations¶
Most of the examples above use integers for enumeration values. Using integers is short and handy (and provided by default by the Functional API), but not strictly enforced. In the vast majority of use-cases, one doesn’t care what the actual value of an enumeration is. But if the value is important, enumerations can have arbitrary values.
Enumerations are Python classes, and can have methods and special methods as usual. If we have this enumeration:
>>> class Mood(Enum):
... FUNKY = 1
... HAPPY = 3
...
... def describe(self):
... # self is the member here
... return self.name, self.value
...
... def __str__(self):
... return 'my custom str! {0}'.format(self.value)
...
... @classmethod
... def favorite_mood(cls):
... # cls here is the enumeration
... return cls.HAPPY
...
Then:
>>> Mood.favorite_mood()
<Mood.HAPPY: 3>
>>> Mood.HAPPY.describe()
('HAPPY', 3)
>>> str(Mood.FUNKY)
'my custom str! 1'
The rules for what is allowed are as follows: names that start and end with
a single underscore are reserved by enum and cannot be used; all other
attributes defined within an enumeration will become members of this
enumeration, with the exception of special methods (__str__()
,
__add__()
, etc.), descriptors (methods are also descriptors), and
variable names listed in _ignore_
.
Note: if your enumeration defines __new__()
and/or __init__()
,
any value(s) given to the enum member will be passed into those methods.
See Planet for an example.
Note
The __new__()
method, if defined, is used during creation of the Enum
members; it is then replaced by Enum’s __new__()
which is used after
class creation for lookup of existing members. See When to use __new__() vs. __init__() for
more details.
Restricted Enum subclassing¶
A new Enum
class must have one base enum class, up to one concrete
data type, and as many object
-based mixin classes as needed. The
order of these base classes is:
class EnumName([mix-in, ...,] [data-type,] base-enum):
pass
Also, subclassing an enumeration is allowed only if the enumeration does not define any members. So this is forbidden:
>>> class MoreColor(Color):
... PINK = 17
...
Traceback (most recent call last):
...
TypeError: <enum 'MoreColor'> cannot extend <enum 'Color'>
But this is allowed:
>>> class Foo(Enum):
... def some_behavior(self):
... pass
...
>>> class Bar(Foo):
... HAPPY = 1
... SAD = 2
...
Allowing subclassing of enums that define members would lead to a violation of some important invariants of types and instances. On the other hand, it makes sense to allow sharing some common behavior between a group of enumerations. (See OrderedEnum for an example.)
Dataclass support¶
When inheriting from a dataclass
,
the __repr__()
omits the inherited class’ name. For example:
>>> from dataclasses import dataclass, field
>>> @dataclass
... class CreatureDataMixin:
... size: str
... legs: int
... tail: bool = field(repr=False, default=True)
...
>>> class Creature(CreatureDataMixin, Enum):
... BEETLE = 'small', 6
... DOG = 'medium', 4
...
>>> Creature.DOG
<Creature.DOG: size='medium', legs=4>
Use the dataclass()
argument repr=False
to use the standard repr()
.
Changed in version 3.12: Only the dataclass fields are shown in the value area, not the dataclass’ name.
Note
Adding dataclass()
decorator to Enum
and its subclasses is not supported. It will not raise any errors,
but it will produce very strange results at runtime, such as members
being equal to each other:
>>> @dataclass # don't do this: it does not make any sense
... class Color(Enum):
... RED = 1
... BLUE = 2
...
>>> Color.RED is Color.BLUE
False
>>> Color.RED == Color.BLUE # problem is here: they should not be equal
True
Pickling¶
Enumerations can be pickled and unpickled:
>>> from test.test_enum import Fruit
>>> from pickle import dumps, loads
>>> Fruit.TOMATO is loads(dumps(Fruit.TOMATO))
True
The usual restrictions for pickling apply: picklable enums must be defined in the top level of a module, since unpickling requires them to be importable from that module.
Note
With pickle protocol version 4 it is possible to easily pickle enums nested in other classes.
It is possible to modify how enum members are pickled/unpickled by defining
__reduce_ex__()
in the enumeration class. The default method is by-value,
but enums with complicated values may want to use by-name:
>>> import enum
>>> class MyEnum(enum.Enum):
... __reduce_ex__ = enum.pickle_by_enum_name
Note
Using by-name for flags is not recommended, as unnamed aliases will not unpickle.
Functional API¶
The Enum
class is callable, providing the following functional API:
>>> Animal = Enum('Animal', 'ANT BEE CAT DOG')
>>> Animal
<enum 'Animal'>
>>> Animal.ANT
<Animal.ANT: 1>
>>> list(Animal)
[<Animal.ANT: 1>, <Animal.BEE: 2>, <Animal.CAT: 3>, <Animal.DOG: 4>]
The semantics of this API resemble namedtuple
. The first
argument of the call to Enum
is the name of the enumeration.
The second argument is the source of enumeration member names. It can be a
whitespace-separated string of names, a sequence of names, a sequence of
2-tuples with key/value pairs, or a mapping (e.g. dictionary) of names to
values. The last two options enable assigning arbitrary values to
enumerations; the others auto-assign increasing integers starting with 1 (use
the start
parameter to specify a different starting value). A
new class derived from Enum
is returned. In other words, the above
assignment to Animal
is equivalent to:
>>> class Animal(Enum):
... ANT = 1
... BEE = 2
... CAT = 3
... DOG = 4
...
The reason for defaulting to 1
as the starting number and not 0
is
that 0
is False
in a boolean sense, but by default enum members all
evaluate to True
.
Pickling enums created with the functional API can be tricky as frame stack implementation details are used to try and figure out which module the enumeration is being created in (e.g. it will fail if you use a utility function in a separate module, and also may not work on IronPython or Jython). The solution is to specify the module name explicitly as follows:
>>> Animal = Enum('Animal', 'ANT BEE CAT DOG', module=__name__)
Warning
If module
is not supplied, and Enum cannot determine what it is,
the new Enum members will not be unpicklable; to keep errors closer to
the source, pickling will be disabled.
The new pickle protocol 4 also, in some circumstances, relies on
__qualname__
being set to the location where pickle will be able
to find the class. For example, if the class was made available in class
SomeData in the global scope:
>>> Animal = Enum('Animal', 'ANT BEE CAT DOG', qualname='SomeData.Animal')
The complete signature is:
Enum(
value='NewEnumName',
names=<...>,
*,
module='...',
qualname='...',
type=<mixed-in class>,
start=1,
)
value: What the new enum class will record as its name.
names: The enum members. This can be a whitespace- or comma-separated string (values will start at 1 unless otherwise specified):
'RED GREEN BLUE' | 'RED,GREEN,BLUE' | 'RED, GREEN, BLUE'
or an iterator of names:
['RED', 'GREEN', 'BLUE']
or an iterator of (name, value) pairs:
[('CYAN', 4), ('MAGENTA', 5), ('YELLOW', 6)]
or a mapping:
{'CHARTREUSE': 7, 'SEA_GREEN': 11, 'ROSEMARY': 42}
module: name of module where new enum class can be found.
qualname: where in module new enum class can be found.
type: type to mix in to new enum class.
start: number to start counting at if only names are passed in.
Changed in version 3.5: The start parameter was added.
Derived Enumerations¶
IntEnum¶
The first variation of Enum
that is provided is also a subclass of
int
. Members of an IntEnum
can be compared to integers;
by extension, integer enumerations of different types can also be compared
to each other:
>>> from enum import IntEnum
>>> class Shape(IntEnum):
... CIRCLE = 1
... SQUARE = 2
...
>>> class Request(IntEnum):
... POST = 1
... GET = 2
...
>>> Shape == 1
False
>>> Shape.CIRCLE == 1
True
>>> Shape.CIRCLE == Request.POST
True
However, they still can’t be compared to standard Enum
enumerations:
>>> class Shape(IntEnum):
... CIRCLE = 1
... SQUARE = 2
...
>>> class Color(Enum):
... RED = 1
... GREEN = 2
...
>>> Shape.CIRCLE == Color.RED
False
IntEnum
values behave like integers in other ways you’d expect:
>>> int(Shape.CIRCLE)
1
>>> ['a', 'b', 'c'][Shape.CIRCLE]
'b'
>>> [i for i in range(Shape.SQUARE)]
[0, 1]
StrEnum¶
The second variation of Enum
that is provided is also a subclass of
str
. Members of a StrEnum
can be compared to strings;
by extension, string enumerations of different types can also be compared
to each other.
Added in version 3.11.
IntFlag¶
The next variation of Enum
provided, IntFlag
, is also based
on int
. The difference being IntFlag
members can be combined
using the bitwise operators (&, |, ^, ~) and the result is still an
IntFlag
member, if possible. Like IntEnum
, IntFlag
members are also integers and can be used wherever an int
is used.
Note
Any operation on an IntFlag
member besides the bit-wise operations will
lose the IntFlag
membership.
Bit-wise operations that result in invalid IntFlag
values will lose the
IntFlag
membership. See FlagBoundary
for
details.
Added in version 3.6.
Changed in version 3.11.
Sample IntFlag
class:
>>> from enum import IntFlag
>>> class Perm(IntFlag):
... R = 4
... W = 2
... X = 1
...
>>> Perm.R | Perm.W
<Perm.R|W: 6>
>>> Perm.R + Perm.W
6
>>> RW = Perm.R | Perm.W
>>> Perm.R in RW
True
It is also possible to name the combinations:
>>> class Perm(IntFlag):
... R = 4
... W = 2
... X = 1
... RWX = 7
...
>>> Perm.RWX
<Perm.RWX: 7>
>>> ~Perm.RWX
<Perm: 0>
>>> Perm(7)
<Perm.RWX: 7>
Note
Named combinations are considered aliases. Aliases do not show up during iteration, but can be returned from by-value lookups.
Changed in version 3.11.
Another important difference between IntFlag
and Enum
is that
if no flags are set (the value is 0), its boolean evaluation is False
:
>>> Perm.R & Perm.X
<Perm: 0>
>>> bool(Perm.R & Perm.X)
False
Because IntFlag
members are also subclasses of int
they can
be combined with them (but may lose IntFlag
membership:
>>> Perm.X | 4
<Perm.R|X: 5>
>>> Perm.X + 8
9
Note
The negation operator, ~
, always returns an IntFlag
member with a
positive value:
>>> (~Perm.X).value == (Perm.R|Perm.W).value == 6
True
IntFlag
members can also be iterated over:
>>> list(RW)
[<Perm.R: 4>, <Perm.W: 2>]
Added in version 3.11.
Flag¶
The last variation is Flag
. Like IntFlag
, Flag
members can be combined using the bitwise operators (&, |, ^, ~). Unlike
IntFlag
, they cannot be combined with, nor compared against, any
other Flag
enumeration, nor int
. While it is possible to
specify the values directly it is recommended to use auto
as the
value and let Flag
select an appropriate value.
Added in version 3.6.
Like IntFlag
, if a combination of Flag
members results in no
flags being set, the boolean evaluation is False
:
>>> from enum import Flag, auto
>>> class Color(Flag):
... RED = auto()
... BLUE = auto()
... GREEN = auto()
...
>>> Color.RED & Color.GREEN
<Color: 0>
>>> bool(Color.RED & Color.GREEN)
False
Individual flags should have values that are powers of two (1, 2, 4, 8, …), while combinations of flags will not:
>>> class Color(Flag):
... RED = auto()
... BLUE = auto()
... GREEN = auto()
... WHITE = RED | BLUE | GREEN
...
>>> Color.WHITE
<Color.WHITE: 7>
Giving a name to the “no flags set” condition does not change its boolean value:
>>> class Color(Flag):
... BLACK = 0
... RED = auto()
... BLUE = auto()
... GREEN = auto()
...
>>> Color.BLACK
<Color.BLACK: 0>
>>> bool(Color.BLACK)
False
Flag
members can also be iterated over:
>>> purple = Color.RED | Color.BLUE
>>> list(purple)
[<Color.RED: 1>, <Color.BLUE: 2>]
Added in version 3.11.
Note
For the majority of new code, Enum
and Flag
are strongly
recommended, since IntEnum
and IntFlag
break some
semantic promises of an enumeration (by being comparable to integers, and
thus by transitivity to other unrelated enumerations). IntEnum
and IntFlag
should be used only in cases where Enum
and
Flag
will not do; for example, when integer constants are replaced
with enumerations, or for interoperability with other systems.
Others¶
While IntEnum
is part of the enum
module, it would be very
simple to implement independently:
class IntEnum(int, ReprEnum): # or Enum instead of ReprEnum
pass
This demonstrates how similar derived enumerations can be defined; for example
a FloatEnum
that mixes in float
instead of int
.
Some rules:
When subclassing
Enum
, mix-in types must appear before theEnum
class itself in the sequence of bases, as in theIntEnum
example above.Mix-in types must be subclassable. For example,
bool
andrange
are not subclassable and will throw an error during Enum creation if used as the mix-in type.While
Enum
can have members of any type, once you mix in an additional type, all the members must have values of that type, e.g.int
above. This restriction does not apply to mix-ins which only add methods and don’t specify another type.When another data type is mixed in, the
value
attribute is not the same as the enum member itself, although it is equivalent and will compare equal.A
data type
is a mixin that defines__new__()
, or adataclass
%-style formatting:
%s
and%r
call theEnum
class’s__str__()
and__repr__()
respectively; other codes (such as%i
or%h
for IntEnum) treat the enum member as its mixed-in type.Formatted string literals,
str.format()
, andformat()
will use the enum’s__str__()
method.
When to use __new__()
vs. __init__()
¶
__new__()
must be used whenever you want to customize the actual value of
the Enum
member. Any other modifications may go in either
__new__()
or __init__()
, with __init__()
being preferred.
For example, if you want to pass several items to the constructor, but only want one of them to be the value:
>>> class Coordinate(bytes, Enum):
... """
... Coordinate with binary codes that can be indexed by the int code.
... """
... def __new__(cls, value, label, unit):
... obj = bytes.__new__(cls, [value])
... obj._value_ = value
... obj.label = label
... obj.unit = unit
... return obj
... PX = (0, 'P.X', 'km')
... PY = (1, 'P.Y', 'km')
... VX = (2, 'V.X', 'km/s')
... VY = (3, 'V.Y', 'km/s')
...
>>> print(Coordinate['PY'])
Coordinate.PY
>>> print(Coordinate(3))
Coordinate.VY
Warning
Do not call super().__new__()
, as the lookup-only __new__
is the one
that is found; instead, use the data type directly.
Finer Points¶
Supported __dunder__
names¶
__members__
is a read-only ordered mapping of member_name
:member
items. It is only available on the class.
__new__()
, if specified, must create and return the enum members; it is
also a very good idea to set the member’s _value_
appropriately. Once
all the members are created it is no longer used.
Supported _sunder_
names¶
_name_
– name of the member_value_
– value of the member; can be set in__new__
_missing_()
– a lookup function used when a value is not found; may be overridden_ignore_
– a list of names, either as alist
or astr
, that will not be transformed into members, and will be removed from the final class_generate_next_value_()
– used to get an appropriate value for an enum member; may be overridden_add_alias_()
– adds a new name as an alias to an existing member._add_value_alias_()
– adds a new value as an alias to an existing member. See MultiValueEnum for an example.Note
For standard
Enum
classes the next value chosen is the highest value seen incremented by one.For
Flag
classes the next value chosen will be the next highest power-of-two.Changed in version 3.13: Prior versions would use the last seen value instead of the highest value.
Added in version 3.6: _missing_
, _order_
, _generate_next_value_
Added in version 3.7: _ignore_
Added in version 3.13: _add_alias_
, _add_value_alias_
To help keep Python 2 / Python 3 code in sync an _order_
attribute can
be provided. It will be checked against the actual order of the enumeration
and raise an error if the two do not match:
>>> class Color(Enum):
... _order_ = 'RED GREEN BLUE'
... RED = 1
... BLUE = 3
... GREEN = 2
...
Traceback (most recent call last):
...
TypeError: member order does not match _order_:
['RED', 'BLUE', 'GREEN']
['RED', 'GREEN', 'BLUE']
Note
In Python 2 code the _order_
attribute is necessary as definition
order is lost before it can be recorded.
_Private__names¶
Private names are not converted to enum members, but remain normal attributes.
Changed in version 3.11.
Enum
member type¶
Enum members are instances of their enum class, and are normally accessed as
EnumClass.member
. In certain situations, such as writing custom enum
behavior, being able to access one member directly from another is useful,
and is supported; however, in order to avoid name clashes between member names
and attributes/methods from mixed-in classes, upper-case names are strongly
recommended.
Changed in version 3.5.
Creating members that are mixed with other data types¶
When subclassing other data types, such as int
or str
, with
an Enum
, all values after the =
are passed to that data type’s
constructor. For example:
>>> class MyEnum(IntEnum): # help(int) -> int(x, base=10) -> integer
... example = '11', 16 # so x='11' and base=16
...
>>> MyEnum.example.value # and hex(11) is...
17
Boolean value of Enum
classes and members¶
Enum classes that are mixed with non-Enum
types (such as
int
, str
, etc.) are evaluated according to the mixed-in
type’s rules; otherwise, all members evaluate as True
. To make your
own enum’s boolean evaluation depend on the member’s value add the following to
your class:
def __bool__(self):
return bool(self.value)
Enum
classes with methods¶
If you give your enum subclass extra methods, like the Planet
class below, those methods will show up in a dir()
of the member,
but not of the class:
>>> dir(Planet)
['EARTH', 'JUPITER', 'MARS', 'MERCURY', 'NEPTUNE', 'SATURN', 'URANUS', 'VENUS', '__class__', '__doc__', '__members__', '__module__']
>>> dir(Planet.EARTH)
['__class__', '__doc__', '__module__', 'mass', 'name', 'radius', 'surface_gravity', 'value']
Combining members of Flag
¶
Iterating over a combination of Flag
members will only return the members that
are comprised of a single bit:
>>> class Color(Flag):
... RED = auto()
... GREEN = auto()
... BLUE = auto()
... MAGENTA = RED | BLUE
... YELLOW = RED | GREEN
... CYAN = GREEN | BLUE
...
>>> Color(3) # named combination
<Color.YELLOW: 3>
>>> Color(7) # not named combination
<Color.RED|GREEN|BLUE: 7>
Flag
and IntFlag
minutia¶
Using the following snippet for our examples:
>>> class Color(IntFlag):
... BLACK = 0
... RED = 1
... GREEN = 2
... BLUE = 4
... PURPLE = RED | BLUE
... WHITE = RED | GREEN | BLUE
...
the following are true:
single-bit flags are canonical
multi-bit and zero-bit flags are aliases
only canonical flags are returned during iteration:
>>> list(Color.WHITE) [<Color.RED: 1>, <Color.GREEN: 2>, <Color.BLUE: 4>]
negating a flag or flag set returns a new flag/flag set with the corresponding positive integer value:
>>> Color.BLUE <Color.BLUE: 4> >>> ~Color.BLUE <Color.RED|GREEN: 3>
names of pseudo-flags are constructed from their members’ names:
>>> (Color.RED | Color.GREEN).name 'RED|GREEN' >>> class Perm(IntFlag): ... R = 4 ... W = 2 ... X = 1 ... >>> (Perm.R & Perm.W).name is None # effectively Perm(0) True
multi-bit flags, aka aliases, can be returned from operations:
>>> Color.RED | Color.BLUE <Color.PURPLE: 5> >>> Color(7) # or Color(-1) <Color.WHITE: 7> >>> Color(0) <Color.BLACK: 0>
membership / containment checking: zero-valued flags are always considered to be contained:
>>> Color.BLACK in Color.WHITE True
otherwise, only if all bits of one flag are in the other flag will True be returned:
>>> Color.PURPLE in Color.WHITE True >>> Color.GREEN in Color.PURPLE False
There is a new boundary mechanism that controls how out-of-range / invalid
bits are handled: STRICT
, CONFORM
, EJECT
, and KEEP
:
STRICT –> raises an exception when presented with invalid values
CONFORM –> discards any invalid bits
EJECT –> lose Flag status and become a normal int with the given value
KEEP –> keep the extra bits
keeps Flag status and extra bits
extra bits do not show up in iteration
extra bits do show up in repr() and str()
The default for Flag is STRICT
, the default for IntFlag
is EJECT
,
and the default for _convert_
is KEEP
(see ssl.Options
for an
example of when KEEP
is needed).
How are Enums and Flags different?¶
Enums have a custom metaclass that affects many aspects of both derived Enum
classes and their instances (members).
Enum Classes¶
The EnumType
metaclass is responsible for providing the
__contains__()
, __dir__()
, __iter__()
and other methods that
allow one to do things with an Enum
class that fail on a typical
class, such as list(Color)
or some_enum_var in Color
. EnumType
is
responsible for ensuring that various other methods on the final Enum
class are correct (such as __new__()
, __getnewargs__()
,
__str__()
and __repr__()
).
Flag Classes¶
Flags have an expanded view of aliasing: to be canonical, the value of a flag
needs to be a power-of-two value, and not a duplicate name. So, in addition to the
Enum
definition of alias, a flag with no value (a.k.a. 0
) or with more than one
power-of-two value (e.g. 3
) is considered an alias.
Enum Members (aka instances)¶
The most interesting thing about enum members is that they are singletons.
EnumType
creates them all while it is creating the enum class itself,
and then puts a custom __new__()
in place to ensure that no new ones are
ever instantiated by returning only the existing member instances.
Flag Members¶
Flag members can be iterated over just like the Flag
class, and only the
canonical members will be returned. For example:
>>> list(Color)
[<Color.RED: 1>, <Color.GREEN: 2>, <Color.BLUE: 4>]
(Note that BLACK
, PURPLE
, and WHITE
do not show up.)
Inverting a flag member returns the corresponding positive value, rather than a negative value — for example:
>>> ~Color.RED
<Color.GREEN|BLUE: 6>
Flag members have a length corresponding to the number of power-of-two values they contain. For example:
>>> len(Color.PURPLE)
2
Enum Cookbook¶
While Enum
, IntEnum
, StrEnum
, Flag
, and
IntFlag
are expected to cover the majority of use-cases, they cannot
cover them all. Here are recipes for some different types of enumerations
that can be used directly, or as examples for creating one’s own.
Omitting values¶
In many use-cases, one doesn’t care what the actual value of an enumeration is. There are several ways to define this type of simple enumeration:
use instances of
auto
for the valueuse instances of
object
as the valueuse a descriptive string as the value
use a tuple as the value and a custom
__new__()
to replace the tuple with anint
value
Using any of these methods signifies to the user that these values are not important, and also enables one to add, remove, or reorder members without having to renumber the remaining members.
Using auto
¶
Using auto
would look like:
>>> class Color(Enum):
... RED = auto()
... BLUE = auto()
... GREEN = auto()
...
>>> Color.GREEN
<Color.GREEN: 3>
Using object
¶
Using object
would look like:
>>> class Color(Enum):
... RED = object()
... GREEN = object()
... BLUE = object()
...
>>> Color.GREEN
<Color.GREEN: <object object at 0x...>>
This is also a good example of why you might want to write your own
__repr__()
:
>>> class Color(Enum):
... RED = object()
... GREEN = object()
... BLUE = object()
... def __repr__(self):
... return "<%s.%s>" % (self.__class__.__name__, self._name_)
...
>>> Color.GREEN
<Color.GREEN>
Using a descriptive string¶
Using a string as the value would look like:
>>> class Color(Enum):
... RED = 'stop'
... GREEN = 'go'
... BLUE = 'too fast!'
...
>>> Color.GREEN
<Color.GREEN: 'go'>
Using a custom __new__()
¶
Using an auto-numbering __new__()
would look like:
>>> class AutoNumber(Enum):
... def __new__(cls):
... value = len(cls.__members__) + 1
... obj = object.__new__(cls)
... obj._value_ = value
... return obj
...
>>> class Color(AutoNumber):
... RED = ()
... GREEN = ()
... BLUE = ()
...
>>> Color.GREEN
<Color.GREEN: 2>
To make a more general purpose AutoNumber
, add *args
to the signature:
>>> class AutoNumber(Enum):
... def __new__(cls, *args): # this is the only change from above
... value = len(cls.__members__) + 1
... obj = object.__new__(cls)
... obj._value_ = value
... return obj
...
Then when you inherit from AutoNumber
you can write your own __init__
to handle any extra arguments:
>>> class Swatch(AutoNumber):
... def __init__(self, pantone='unknown'):
... self.pantone = pantone
... AUBURN = '3497'
... SEA_GREEN = '1246'
... BLEACHED_CORAL = () # New color, no Pantone code yet!
...
>>> Swatch.SEA_GREEN
<Swatch.SEA_GREEN: 2>
>>> Swatch.SEA_GREEN.pantone
'1246'
>>> Swatch.BLEACHED_CORAL.pantone
'unknown'
Note
The __new__()
method, if defined, is used during creation of the Enum
members; it is then replaced by Enum’s __new__()
which is used after
class creation for lookup of existing members.
Warning
Do not call super().__new__()
, as the lookup-only __new__
is the one
that is found; instead, use the data type directly – e.g.:
obj = int.__new__(cls, value)
OrderedEnum¶
An ordered enumeration that is not based on IntEnum
and so maintains
the normal Enum
invariants (such as not being comparable to other
enumerations):
>>> class OrderedEnum(Enum):
... def __ge__(self, other):
... if self.__class__ is other.__class__:
... return self.value >= other.value
... return NotImplemented
... def __gt__(self, other):
... if self.__class__ is other.__class__:
... return self.value > other.value
... return NotImplemented
... def __le__(self, other):
... if self.__class__ is other.__class__:
... return self.value <= other.value
... return NotImplemented
... def __lt__(self, other):
... if self.__class__ is other.__class__:
... return self.value < other.value
... return NotImplemented
...
>>> class Grade(OrderedEnum):
... A = 5
... B = 4
... C = 3
... D = 2
... F = 1
...
>>> Grade.C < Grade.A
True
DuplicateFreeEnum¶
Raises an error if a duplicate member value is found instead of creating an alias:
>>> class DuplicateFreeEnum(Enum):
... def __init__(self, *args):
... cls = self.__class__
... if any(self.value == e.value for e in cls):
... a = self.name
... e = cls(self.value).name
... raise ValueError(
... "aliases not allowed in DuplicateFreeEnum: %r --> %r"
... % (a, e))
...
>>> class Color(DuplicateFreeEnum):
... RED = 1
... GREEN = 2
... BLUE = 3
... GRENE = 2
...
Traceback (most recent call last):
...
ValueError: aliases not allowed in DuplicateFreeEnum: 'GRENE' --> 'GREEN'
Note
This is a useful example for subclassing Enum to add or change other
behaviors as well as disallowing aliases. If the only desired change is
disallowing aliases, the unique()
decorator can be used instead.
MultiValueEnum¶
Supports having more than one value per member:
>>> class MultiValueEnum(Enum):
... def __new__(cls, value, *values):
... self = object.__new__(cls)
... self._value_ = value
... for v in values:
... self._add_value_alias_(v)
... return self
...
>>> class DType(MultiValueEnum):
... float32 = 'f', 8
... double64 = 'd', 9
...
>>> DType('f')
<DType.float32: 'f'>
>>> DType(9)
<DType.double64: 'd'>
Planet¶
If __new__()
or __init__()
is defined, the value of the enum member
will be passed to those methods:
>>> class Planet(Enum):
... MERCURY = (3.303e+23, 2.4397e6)
... VENUS = (4.869e+24, 6.0518e6)
... EARTH = (5.976e+24, 6.37814e6)
... MARS = (6.421e+23, 3.3972e6)
... JUPITER = (1.9e+27, 7.1492e7)
... SATURN = (5.688e+26, 6.0268e7)
... URANUS = (8.686e+25, 2.5559e7)
... NEPTUNE = (1.024e+26, 2.4746e7)
... def __init__(self, mass, radius):
... self.mass = mass # in kilograms
... self.radius = radius # in meters
... @property
... def surface_gravity(self):
... # universal gravitational constant (m3 kg-1 s-2)
... G = 6.67300E-11
... return G * self.mass / (self.radius * self.radius)
...
>>> Planet.EARTH.value
(5.976e+24, 6378140.0)
>>> Planet.EARTH.surface_gravity
9.802652743337129
TimePeriod¶
An example to show the _ignore_
attribute in use:
>>> from datetime import timedelta
>>> class Period(timedelta, Enum):
... "different lengths of time"
... _ignore_ = 'Period i'
... Period = vars()
... for i in range(367):
... Period['day_%d' % i] = i
...
>>> list(Period)[:2]
[<Period.day_0: datetime.timedelta(0)>, <Period.day_1: datetime.timedelta(days=1)>]
>>> list(Period)[-2:]
[<Period.day_365: datetime.timedelta(days=365)>, <Period.day_366: datetime.timedelta(days=366)>]
Subclassing EnumType¶
While most enum needs can be met by customizing Enum
subclasses,
either with class decorators or custom functions, EnumType
can be
subclassed to provide a different Enum experience.


Descriptor Guide¶
- Author:
Raymond Hettinger
- Contact:
<python at rcn dot com>
Descriptors let objects customize attribute lookup, storage, and deletion.
This guide has four major sections:
The “primer” gives a basic overview, moving gently from simple examples, adding one feature at a time. Start here if you’re new to descriptors.
The second section shows a complete, practical descriptor example. If you already know the basics, start there.
The third section provides a more technical tutorial that goes into the detailed mechanics of how descriptors work. Most people don’t need this level of detail.
The last section has pure Python equivalents for built-in descriptors that are written in C. Read this if you’re curious about how functions turn into bound methods or about the implementation of common tools like
classmethod()
,staticmethod()
,property()
, and __slots__.
Primer¶
In this primer, we start with the most basic possible example and then we’ll add new capabilities one by one.
Simple example: A descriptor that returns a constant¶
The Ten
class is a descriptor whose __get__()
method always
returns the constant 10
:
class Ten:
def __get__(self, obj, objtype=None):
return 10
To use the descriptor, it must be stored as a class variable in another class:
class A:
x = 5 # Regular class attribute
y = Ten() # Descriptor instance
An interactive session shows the difference between normal attribute lookup and descriptor lookup:
>>> a = A() # Make an instance of class A
>>> a.x # Normal attribute lookup
5
>>> a.y # Descriptor lookup
10
In the a.x
attribute lookup, the dot operator finds 'x': 5
in the class dictionary. In the a.y
lookup, the dot operator
finds a descriptor instance, recognized by its __get__
method.
Calling that method returns 10
.
Note that the value 10
is not stored in either the class dictionary or the
instance dictionary. Instead, the value 10
is computed on demand.
This example shows how a simple descriptor works, but it isn’t very useful. For retrieving constants, normal attribute lookup would be better.
In the next section, we’ll create something more useful, a dynamic lookup.
Dynamic lookups¶
Interesting descriptors typically run computations instead of returning constants:
import os
class DirectorySize:
def __get__(self, obj, objtype=None):
return len(os.listdir(obj.dirname))
class Directory:
size = DirectorySize() # Descriptor instance
def __init__(self, dirname):
self.dirname = dirname # Regular instance attribute
An interactive session shows that the lookup is dynamic — it computes different, updated answers each time:
>>> s = Directory('songs')
>>> g = Directory('games')
>>> s.size # The songs directory has twenty files
20
>>> g.size # The games directory has three files
3
>>> os.remove('games/chess') # Delete a game
>>> g.size # File count is automatically updated
2
Besides showing how descriptors can run computations, this example also
reveals the purpose of the parameters to __get__()
. The self
parameter is size, an instance of DirectorySize. The obj parameter is
either g or s, an instance of Directory. It is the obj parameter that
lets the __get__()
method learn the target directory. The objtype
parameter is the class Directory.
Managed attributes¶
A popular use for descriptors is managing access to instance data. The
descriptor is assigned to a public attribute in the class dictionary while the
actual data is stored as a private attribute in the instance dictionary. The
descriptor’s __get__()
and __set__()
methods are triggered when
the public attribute is accessed.
In the following example, age is the public attribute and _age is the private attribute. When the public attribute is accessed, the descriptor logs the lookup or update:
import logging
logging.basicConfig(level=logging.INFO)
class LoggedAgeAccess:
def __get__(self, obj, objtype=None):
value = obj._age
logging.info('Accessing %r giving %r', 'age', value)
return value
def __set__(self, obj, value):
logging.info('Updating %r to %r', 'age', value)
obj._age = value
class Person:
age = LoggedAgeAccess() # Descriptor instance
def __init__(self, name, age):
self.name = name # Regular instance attribute
self.age = age # Calls __set__()
def birthday(self):
self.age += 1 # Calls both __get__() and __set__()
An interactive session shows that all access to the managed attribute age is logged, but that the regular attribute name is not logged:
>>> mary = Person('Mary M', 30) # The initial age update is logged
INFO:root:Updating 'age' to 30
>>> dave = Person('David D', 40)
INFO:root:Updating 'age' to 40
>>> vars(mary) # The actual data is in a private attribute
{'name': 'Mary M', '_age': 30}
>>> vars(dave)
{'name': 'David D', '_age': 40}
>>> mary.age # Access the data and log the lookup
INFO:root:Accessing 'age' giving 30
30
>>> mary.birthday() # Updates are logged as well
INFO:root:Accessing 'age' giving 30
INFO:root:Updating 'age' to 31
>>> dave.name # Regular attribute lookup isn't logged
'David D'
>>> dave.age # Only the managed attribute is logged
INFO:root:Accessing 'age' giving 40
40
One major issue with this example is that the private name _age is hardwired in the LoggedAgeAccess class. That means that each instance can only have one logged attribute and that its name is unchangeable. In the next example, we’ll fix that problem.
Customized names¶
When a class uses descriptors, it can inform each descriptor about which variable name was used.
In this example, the Person
class has two descriptor instances,
name and age. When the Person
class is defined, it makes a
callback to __set_name__()
in LoggedAccess so that the field names can
be recorded, giving each descriptor its own public_name and private_name:
import logging
logging.basicConfig(level=logging.INFO)
class LoggedAccess:
def __set_name__(self, owner, name):
self.public_name = name
self.private_name = '_' + name
def __get__(self, obj, objtype=None):
value = getattr(obj, self.private_name)
logging.info('Accessing %r giving %r', self.public_name, value)
return value
def __set__(self, obj, value):
logging.info('Updating %r to %r', self.public_name, value)
setattr(obj, self.private_name, value)
class Person:
name = LoggedAccess() # First descriptor instance
age = LoggedAccess() # Second descriptor instance
def __init__(self, name, age):
self.name = name # Calls the first descriptor
self.age = age # Calls the second descriptor
def birthday(self):
self.age += 1
An interactive session shows that the Person
class has called
__set_name__()
so that the field names would be recorded. Here
we call vars()
to look up the descriptor without triggering it:
>>> vars(vars(Person)['name'])
{'public_name': 'name', 'private_name': '_name'}
>>> vars(vars(Person)['age'])
{'public_name': 'age', 'private_name': '_age'}
The new class now logs access to both name and age:
>>> pete = Person('Peter P', 10)
INFO:root:Updating 'name' to 'Peter P'
INFO:root:Updating 'age' to 10
>>> kate = Person('Catherine C', 20)
INFO:root:Updating 'name' to 'Catherine C'
INFO:root:Updating 'age' to 20
The two Person instances contain only the private names:
>>> vars(pete)
{'_name': 'Peter P', '_age': 10}
>>> vars(kate)
{'_name': 'Catherine C', '_age': 20}
Closing thoughts¶
A descriptor is what we call any object that defines __get__()
,
__set__()
, or __delete__()
.
Optionally, descriptors can have a __set_name__()
method. This is only
used in cases where a descriptor needs to know either the class where it was
created or the name of class variable it was assigned to. (This method, if
present, is called even if the class is not a descriptor.)
Descriptors get invoked by the dot operator during attribute lookup. If a
descriptor is accessed indirectly with vars(some_class)[descriptor_name]
,
the descriptor instance is returned without invoking it.
Descriptors only work when used as class variables. When put in instances, they have no effect.
The main motivation for descriptors is to provide a hook allowing objects stored in class variables to control what happens during attribute lookup.
Traditionally, the calling class controls what happens during lookup. Descriptors invert that relationship and allow the data being looked-up to have a say in the matter.
Descriptors are used throughout the language. It is how functions turn into
bound methods. Common tools like classmethod()
, staticmethod()
,
property()
, and functools.cached_property()
are all implemented as
descriptors.
Complete Practical Example¶
In this example, we create a practical and powerful tool for locating notoriously hard to find data corruption bugs.
Validator class¶
A validator is a descriptor for managed attribute access. Prior to storing any data, it verifies that the new value meets various type and range restrictions. If those restrictions aren’t met, it raises an exception to prevent data corruption at its source.
This Validator
class is both an abstract base class and a
managed attribute descriptor:
from abc import ABC, abstractmethod
class Validator(ABC):
def __set_name__(self, owner, name):
self.private_name = '_' + name
def __get__(self, obj, objtype=None):
return getattr(obj, self.private_name)
def __set__(self, obj, value):
self.validate(value)
setattr(obj, self.private_name, value)
@abstractmethod
def validate(self, value):
pass
Custom validators need to inherit from Validator
and must supply a
validate()
method to test various restrictions as needed.
Custom validators¶
Here are three practical data validation utilities:
OneOf
verifies that a value is one of a restricted set of options.Number
verifies that a value is either anint
orfloat
. Optionally, it verifies that a value is between a given minimum or maximum.String
verifies that a value is astr
. Optionally, it validates a given minimum or maximum length. It can validate a user-defined predicate as well.
class OneOf(Validator):
def __init__(self, *options):
self.options = set(options)
def validate(self, value):
if value not in self.options:
raise ValueError(
f'Expected {value!r} to be one of {self.options!r}'
)
class Number(Validator):
def __init__(self, minvalue=None, maxvalue=None):
self.minvalue = minvalue
self.maxvalue = maxvalue
def validate(self, value):
if not isinstance(value, (int, float)):
raise TypeError(f'Expected {value!r} to be an int or float')
if self.minvalue is not None and value < self.minvalue:
raise ValueError(
f'Expected {value!r} to be at least {self.minvalue!r}'
)
if self.maxvalue is not None and value > self.maxvalue:
raise ValueError(
f'Expected {value!r} to be no more than {self.maxvalue!r}'
)
class String(Validator):
def __init__(self, minsize=None, maxsize=None, predicate=None):
self.minsize = minsize
self.maxsize = maxsize
self.predicate = predicate
def validate(self, value):
if not isinstance(value, str):
raise TypeError(f'Expected {value!r} to be a str')
if self.minsize is not None and len(value) < self.minsize:
raise ValueError(
f'Expected {value!r} to be no smaller than {self.minsize!r}'
)
if self.maxsize is not None and len(value) > self.maxsize:
raise ValueError(
f'Expected {value!r} to be no bigger than {self.maxsize!r}'
)
if self.predicate is not None and not self.predicate(value):
raise ValueError(
f'Expected {self.predicate} to be true for {value!r}'
)
Practical application¶
Here’s how the data validators can be used in a real class:
class Component:
name = String(minsize=3, maxsize=10, predicate=str.isupper)
kind = OneOf('wood', 'metal', 'plastic')
quantity = Number(minvalue=0)
def __init__(self, name, kind, quantity):
self.name = name
self.kind = kind
self.quantity = quantity
The descriptors prevent invalid instances from being created:
>>> Component('Widget', 'metal', 5) # Blocked: 'Widget' is not all uppercase
Traceback (most recent call last):
...
ValueError: Expected <method 'isupper' of 'str' objects> to be true for 'Widget'
>>> Component('WIDGET', 'metle', 5) # Blocked: 'metle' is misspelled
Traceback (most recent call last):
...
ValueError: Expected 'metle' to be one of {'metal', 'plastic', 'wood'}
>>> Component('WIDGET', 'metal', -5) # Blocked: -5 is negative
Traceback (most recent call last):
...
ValueError: Expected -5 to be at least 0
>>> Component('WIDGET', 'metal', 'V') # Blocked: 'V' isn't a number
Traceback (most recent call last):
...
TypeError: Expected 'V' to be an int or float
>>> c = Component('WIDGET', 'metal', 5) # Allowed: The inputs are valid
Technical Tutorial¶
What follows is a more technical tutorial for the mechanics and details of how descriptors work.
Abstract¶
Defines descriptors, summarizes the protocol, and shows how descriptors are called. Provides an example showing how object relational mappings work.
Learning about descriptors not only provides access to a larger toolset, it creates a deeper understanding of how Python works.
Definition and introduction¶
In general, a descriptor is an attribute value that has one of the methods in
the descriptor protocol. Those methods are __get__()
, __set__()
,
and __delete__()
. If any of those methods are defined for an
attribute, it is said to be a descriptor.
The default behavior for attribute access is to get, set, or delete the
attribute from an object’s dictionary. For instance, a.x
has a lookup chain
starting with a.__dict__['x']
, then type(a).__dict__['x']
, and
continuing through the method resolution order of type(a)
. If the
looked-up value is an object defining one of the descriptor methods, then Python
may override the default behavior and invoke the descriptor method instead.
Where this occurs in the precedence chain depends on which descriptor methods
were defined.
Descriptors are a powerful, general purpose protocol. They are the mechanism
behind properties, methods, static methods, class methods, and
super()
. They are used throughout Python itself. Descriptors
simplify the underlying C code and offer a flexible set of new tools for
everyday Python programs.
Descriptor protocol¶
descr.__get__(self, obj, type=None)
descr.__set__(self, obj, value)
descr.__delete__(self, obj)
That is all there is to it. Define any of these methods and an object is considered a descriptor and can override default behavior upon being looked up as an attribute.
If an object defines __set__()
or __delete__()
, it is considered
a data descriptor. Descriptors that only define __get__()
are called
non-data descriptors (they are often used for methods but other uses are
possible).
Data and non-data descriptors differ in how overrides are calculated with respect to entries in an instance’s dictionary. If an instance’s dictionary has an entry with the same name as a data descriptor, the data descriptor takes precedence. If an instance’s dictionary has an entry with the same name as a non-data descriptor, the dictionary entry takes precedence.
To make a read-only data descriptor, define both __get__()
and
__set__()
with the __set__()
raising an AttributeError
when
called. Defining the __set__()
method with an exception raising
placeholder is enough to make it a data descriptor.
Overview of descriptor invocation¶
A descriptor can be called directly with desc.__get__(obj)
or
desc.__get__(None, cls)
.
But it is more common for a descriptor to be invoked automatically from attribute access.
The expression obj.x
looks up the attribute x
in the chain of
namespaces for obj
. If the search finds a descriptor outside of the
instance __dict__
, its __get__()
method is
invoked according to the precedence rules listed below.
The details of invocation depend on whether obj
is an object, class, or
instance of super.
Invocation from an instance¶
Instance lookup scans through a chain of namespaces giving data descriptors
the highest priority, followed by instance variables, then non-data
descriptors, then class variables, and lastly __getattr__()
if it is
provided.
If a descriptor is found for a.x
, then it is invoked with:
desc.__get__(a, type(a))
.
The logic for a dotted lookup is in object.__getattribute__()
. Here is
a pure Python equivalent:
def find_name_in_mro(cls, name, default):
"Emulate _PyType_Lookup() in Objects/typeobject.c"
for base in cls.__mro__:
if name in vars(base):
return vars(base)[name]
return default
def object_getattribute(obj, name):
"Emulate PyObject_GenericGetAttr() in Objects/object.c"
null = object()
objtype = type(obj)
cls_var = find_name_in_mro(objtype, name, null)
descr_get = getattr(type(cls_var), '__get__', null)
if descr_get is not null:
if (hasattr(type(cls_var), '__set__')
or hasattr(type(cls_var), '__delete__')):
return descr_get(cls_var, obj, objtype) # data descriptor
if hasattr(obj, '__dict__') and name in vars(obj):
return vars(obj)[name] # instance variable
if descr_get is not null:
return descr_get(cls_var, obj, objtype) # non-data descriptor
if cls_var is not null:
return cls_var # class variable
raise AttributeError(name)
Note, there is no __getattr__()
hook in the __getattribute__()
code. That is why calling __getattribute__()
directly or with
super().__getattribute__
will bypass __getattr__()
entirely.
Instead, it is the dot operator and the getattr()
function that are
responsible for invoking __getattr__()
whenever __getattribute__()
raises an AttributeError
. Their logic is encapsulated in a helper
function:
def getattr_hook(obj, name):
"Emulate slot_tp_getattr_hook() in Objects/typeobject.c"
try:
return obj.__getattribute__(name)
except AttributeError:
if not hasattr(type(obj), '__getattr__'):
raise
return type(obj).__getattr__(obj, name) # __getattr__
Invocation from a class¶
The logic for a dotted lookup such as A.x
is in
type.__getattribute__()
. The steps are similar to those for
object.__getattribute__()
but the instance dictionary lookup is replaced
by a search through the class’s method resolution order.
If a descriptor is found, it is invoked with desc.__get__(None, A)
.
The full C implementation can be found in type_getattro()
and
_PyType_Lookup()
in Objects/typeobject.c.
Invocation from super¶
The logic for super’s dotted lookup is in the __getattribute__()
method for
object returned by super()
.
A dotted lookup such as super(A, obj).m
searches obj.__class__.__mro__
for the base class B
immediately following A
and then returns
B.__dict__['m'].__get__(obj, A)
. If not a descriptor, m
is returned
unchanged.
The full C implementation can be found in super_getattro()
in
Objects/typeobject.c. A pure Python equivalent can be found in
Guido’s Tutorial.
Summary of invocation logic¶
The mechanism for descriptors is embedded in the __getattribute__()
methods for object
, type
, and super()
.
The important points to remember are:
Descriptors are invoked by the
__getattribute__()
method.Classes inherit this machinery from
object
,type
, orsuper()
.Overriding
__getattribute__()
prevents automatic descriptor calls because all the descriptor logic is in that method.object.__getattribute__()
andtype.__getattribute__()
make different calls to__get__()
. The first includes the instance and may include the class. The second puts inNone
for the instance and always includes the class.Data descriptors always override instance dictionaries.
Non-data descriptors may be overridden by instance dictionaries.
Automatic name notification¶
Sometimes it is desirable for a descriptor to know what class variable name it
was assigned to. When a new class is created, the type
metaclass
scans the dictionary of the new class. If any of the entries are descriptors
and if they define __set_name__()
, that method is called with two
arguments. The owner is the class where the descriptor is used, and the
name is the class variable the descriptor was assigned to.
The implementation details are in type_new()
and
set_names()
in Objects/typeobject.c.
Since the update logic is in type.__new__()
, notifications only take
place at the time of class creation. If descriptors are added to the class
afterwards, __set_name__()
will need to be called manually.
ORM example¶
The following code is a simplified skeleton showing how data descriptors could be used to implement an object relational mapping.
The essential idea is that the data is stored in an external database. The Python instances only hold keys to the database’s tables. Descriptors take care of lookups or updates:
class Field:
def __set_name__(self, owner, name):
self.fetch = f'SELECT {name} FROM {owner.table} WHERE {owner.key}=?;'
self.store = f'UPDATE {owner.table} SET {name}=? WHERE {owner.key}=?;'
def __get__(self, obj, objtype=None):
return conn.execute(self.fetch, [obj.key]).fetchone()[0]
def __set__(self, obj, value):
conn.execute(self.store, [value, obj.key])
conn.commit()
We can use the Field
class to define models that describe the schema for
each table in a database:
class Movie:
table = 'Movies' # Table name
key = 'title' # Primary key
director = Field()
year = Field()
def __init__(self, key):
self.key = key
class Song:
table = 'Music'
key = 'title'
artist = Field()
year = Field()
genre = Field()
def __init__(self, key):
self.key = key
To use the models, first connect to the database:
>>> import sqlite3
>>> conn = sqlite3.connect('entertainment.db')
An interactive session shows how data is retrieved from the database and how it can be updated:
>>> Movie('Star Wars').director
'George Lucas'
>>> jaws = Movie('Jaws')
>>> f'Released in {jaws.year} by {jaws.director}'
'Released in 1975 by Steven Spielberg'
>>> Song('Country Roads').artist
'John Denver'
>>> Movie('Star Wars').director = 'J.J. Abrams'
>>> Movie('Star Wars').director
'J.J. Abrams'
Pure Python Equivalents¶
The descriptor protocol is simple and offers exciting possibilities. Several use cases are so common that they have been prepackaged into built-in tools. Properties, bound methods, static methods, class methods, and __slots__ are all based on the descriptor protocol.
Properties¶
Calling property()
is a succinct way of building a data descriptor that
triggers a function call upon access to an attribute. Its signature is:
property(fget=None, fset=None, fdel=None, doc=None) -> property
The documentation shows a typical use to define a managed attribute x
:
class C:
def getx(self): return self.__x
def setx(self, value): self.__x = value
def delx(self): del self.__x
x = property(getx, setx, delx, "I'm the 'x' property.")
To see how property()
is implemented in terms of the descriptor protocol,
here is a pure Python equivalent that implements most of the core functionality:
class Property:
"Emulate PyProperty_Type() in Objects/descrobject.c"
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
if doc is None and fget is not None:
doc = fget.__doc__
self.__doc__ = doc
def __set_name__(self, owner, name):
self.__name__ = name
def __get__(self, obj, objtype=None):
if obj is None:
return self
if self.fget is None:
raise AttributeError
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError
self.fset(obj, value)
def __delete__(self, obj):
if self.fdel is None:
raise AttributeError
self.fdel(obj)
def getter(self, fget):
return type(self)(fget, self.fset, self.fdel, self.__doc__)
def setter(self, fset):
return type(self)(self.fget, fset, self.fdel, self.__doc__)
def deleter(self, fdel):
return type(self)(self.fget, self.fset, fdel, self.__doc__)
The property()
builtin helps whenever a user interface has granted
attribute access and then subsequent changes require the intervention of a
method.
For instance, a spreadsheet class may grant access to a cell value through
Cell('b10').value
. Subsequent improvements to the program require the cell
to be recalculated on every access; however, the programmer does not want to
affect existing client code accessing the attribute directly. The solution is
to wrap access to the value attribute in a property data descriptor:
class Cell:
...
@property
def value(self):
"Recalculate the cell before returning value"
self.recalc()
return self._value
Either the built-in property()
or our Property()
equivalent would
work in this example.
Functions and methods¶
Python’s object oriented features are built upon a function based environment. Using non-data descriptors, the two are merged seamlessly.
Functions stored in class dictionaries get turned into methods when invoked. Methods only differ from regular functions in that the object instance is prepended to the other arguments. By convention, the instance is called self but could be called this or any other variable name.
Methods can be created manually with types.MethodType
which is
roughly equivalent to:
class MethodType:
"Emulate PyMethod_Type in Objects/classobject.c"
def __init__(self, func, obj):
self.__func__ = func
self.__self__ = obj
def __call__(self, *args, **kwargs):
func = self.__func__
obj = self.__self__
return func(obj, *args, **kwargs)
def __getattribute__(self, name):
"Emulate method_getset() in Objects/classobject.c"
if name == '__doc__':
return self.__func__.__doc__
return object.__getattribute__(self, name)
def __getattr__(self, name):
"Emulate method_getattro() in Objects/classobject.c"
return getattr(self.__func__, name)
def __get__(self, obj, objtype=None):
"Emulate method_descr_get() in Objects/classobject.c"
return self
To support automatic creation of methods, functions include the
__get__()
method for binding methods during attribute access. This
means that functions are non-data descriptors that return bound methods
during dotted lookup from an instance. Here’s how it works:
class Function:
...
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
if obj is None:
return self
return MethodType(self, obj)
Running the following class in the interpreter shows how the function descriptor works in practice:
class D:
def f(self):
return self
class D2:
pass
The function has a qualified name attribute to support introspection:
>>> D.f.__qualname__
'D.f'
Accessing the function through the class dictionary does not invoke
__get__()
. Instead, it just returns the underlying function object:
>>> D.__dict__['f']
<function D.f at 0x00C45070>
Dotted access from a class calls __get__()
which just returns the
underlying function unchanged:
>>> D.f
<function D.f at 0x00C45070>
The interesting behavior occurs during dotted access from an instance. The
dotted lookup calls __get__()
which returns a bound method object:
>>> d = D()
>>> d.f
<bound method D.f of <__main__.D object at 0x00B18C90>>
Internally, the bound method stores the underlying function and the bound instance:
>>> d.f.__func__
<function D.f at 0x00C45070>
>>> d.f.__self__
<__main__.D object at 0x00B18C90>
If you have ever wondered where self comes from in regular methods or where cls comes from in class methods, this is it!
Kinds of methods¶
Non-data descriptors provide a simple mechanism for variations on the usual patterns of binding functions into methods.
To recap, functions have a __get__()
method so that they can be converted
to a method when accessed as attributes. The non-data descriptor transforms an
obj.f(*args)
call into f(obj, *args)
. Calling cls.f(*args)
becomes f(*args)
.
This chart summarizes the binding and its two most useful variants:
Transformation
Called from an object
Called from a class
function
f(obj, *args)
f(*args)
staticmethod
f(*args)
f(*args)
classmethod
f(type(obj), *args)
f(cls, *args)
Static methods¶
Static methods return the underlying function without changes. Calling either
c.f
or C.f
is the equivalent of a direct lookup into
object.__getattribute__(c, "f")
or object.__getattribute__(C, "f")
. As a
result, the function becomes identically accessible from either an object or a
class.
Good candidates for static methods are methods that do not reference the
self
variable.
For instance, a statistics package may include a container class for
experimental data. The class provides normal methods for computing the average,
mean, median, and other descriptive statistics that depend on the data. However,
there may be useful functions which are conceptually related but do not depend
on the data. For instance, erf(x)
is handy conversion routine that comes up
in statistical work but does not directly depend on a particular dataset.
It can be called either from an object or the class: s.erf(1.5) --> 0.9332
or Sample.erf(1.5) --> 0.9332
.
Since static methods return the underlying function with no changes, the example calls are unexciting:
class E:
@staticmethod
def f(x):
return x * 10
>>> E.f(3)
30
>>> E().f(3)
30
Using the non-data descriptor protocol, a pure Python version of
staticmethod()
would look like this:
import functools
class StaticMethod:
"Emulate PyStaticMethod_Type() in Objects/funcobject.c"
def __init__(self, f):
self.f = f
functools.update_wrapper(self, f)
def __get__(self, obj, objtype=None):
return self.f
def __call__(self, *args, **kwds):
return self.f(*args, **kwds)
@property
def __annotations__(self):
return self.f.__annotations__
The functools.update_wrapper()
call adds a __wrapped__
attribute
that refers to the underlying function. Also it carries forward
the attributes necessary to make the wrapper look like the wrapped
function, including __name__
, __qualname__
,
and __doc__
.
Class methods¶
Unlike static methods, class methods prepend the class reference to the argument list before calling the function. This format is the same for whether the caller is an object or a class:
class F:
@classmethod
def f(cls, x):
return cls.__name__, x
>>> F.f(3)
('F', 3)
>>> F().f(3)
('F', 3)
This behavior is useful whenever the method only needs to have a class
reference and does not rely on data stored in a specific instance. One use for
class methods is to create alternate class constructors. For example, the
classmethod dict.fromkeys()
creates a new dictionary from a list of
keys. The pure Python equivalent is:
class Dict(dict):
@classmethod
def fromkeys(cls, iterable, value=None):
"Emulate dict_fromkeys() in Objects/dictobject.c"
d = cls()
for key in iterable:
d[key] = value
return d
Now a new dictionary of unique keys can be constructed like this:
>>> d = Dict.fromkeys('abracadabra')
>>> type(d) is Dict
True
>>> d
{'a': None, 'b': None, 'r': None, 'c': None, 'd': None}
Using the non-data descriptor protocol, a pure Python version of
classmethod()
would look like this:
import functools
class ClassMethod:
"Emulate PyClassMethod_Type() in Objects/funcobject.c"
def __init__(self, f):
self.f = f
functools.update_wrapper(self, f)
def __get__(self, obj, cls=None):
if cls is None:
cls = type(obj)
return MethodType(self.f, cls)
The functools.update_wrapper()
call in ClassMethod
adds a
__wrapped__
attribute that refers to the underlying function. Also
it carries forward the attributes necessary to make the wrapper look
like the wrapped function: __name__
,
__qualname__
, __doc__
,
and __annotations__
.
Member objects and __slots__¶
When a class defines __slots__
, it replaces instance dictionaries with a
fixed-length array of slot values. From a user point of view that has
several effects:
1. Provides immediate detection of bugs due to misspelled attribute
assignments. Only attribute names specified in __slots__
are allowed:
class Vehicle:
__slots__ = ('id_number', 'make', 'model')
>>> auto = Vehicle()
>>> auto.id_nubmer = 'VYE483814LQEX'
Traceback (most recent call last):
...
AttributeError: 'Vehicle' object has no attribute 'id_nubmer'
2. Helps create immutable objects where descriptors manage access to private
attributes stored in __slots__
:
class Immutable:
__slots__ = ('_dept', '_name') # Replace the instance dictionary
def __init__(self, dept, name):
self._dept = dept # Store to private attribute
self._name = name # Store to private attribute
@property # Read-only descriptor
def dept(self):
return self._dept
@property
def name(self): # Read-only descriptor
return self._name
>>> mark = Immutable('Botany', 'Mark Watney')
>>> mark.dept
'Botany'
>>> mark.dept = 'Space Pirate'
Traceback (most recent call last):
...
AttributeError: property 'dept' of 'Immutable' object has no setter
>>> mark.location = 'Mars'
Traceback (most recent call last):
...
AttributeError: 'Immutable' object has no attribute 'location'
3. Saves memory. On a 64-bit Linux build, an instance with two attributes
takes 48 bytes with __slots__
and 152 bytes without. This flyweight
design pattern likely only
matters when a large number of instances are going to be created.
4. Improves speed. Reading instance variables is 35% faster with
__slots__
(as measured with Python 3.10 on an Apple M1 processor).
5. Blocks tools like functools.cached_property()
which require an
instance dictionary to function correctly:
from functools import cached_property
class CP:
__slots__ = () # Eliminates the instance dict
@cached_property # Requires an instance dict
def pi(self):
return 4 * sum((-1.0)**n / (2.0*n + 1.0)
for n in reversed(range(100_000)))
>>> CP().pi
Traceback (most recent call last):
...
TypeError: No '__dict__' attribute on 'CP' instance to cache 'pi' property.
It is not possible to create an exact drop-in pure Python version of
__slots__
because it requires direct access to C structures and control
over object memory allocation. However, we can build a mostly faithful
simulation where the actual C structure for slots is emulated by a private
_slotvalues
list. Reads and writes to that private structure are managed
by member descriptors:
null = object()
class Member:
def __init__(self, name, clsname, offset):
'Emulate PyMemberDef in Include/structmember.h'
# Also see descr_new() in Objects/descrobject.c
self.name = name
self.clsname = clsname
self.offset = offset
def __get__(self, obj, objtype=None):
'Emulate member_get() in Objects/descrobject.c'
# Also see PyMember_GetOne() in Python/structmember.c
if obj is None:
return self
value = obj._slotvalues[self.offset]
if value is null:
raise AttributeError(self.name)
return value
def __set__(self, obj, value):
'Emulate member_set() in Objects/descrobject.c'
obj._slotvalues[self.offset] = value
def __delete__(self, obj):
'Emulate member_delete() in Objects/descrobject.c'
value = obj._slotvalues[self.offset]
if value is null:
raise AttributeError(self.name)
obj._slotvalues[self.offset] = null
def __repr__(self):
'Emulate member_repr() in Objects/descrobject.c'
return f'<Member {self.name!r} of {self.clsname!r}>'
The type.__new__()
method takes care of adding member objects to class
variables:
class Type(type):
'Simulate how the type metaclass adds member objects for slots'
def __new__(mcls, clsname, bases, mapping, **kwargs):
'Emulate type_new() in Objects/typeobject.c'
# type_new() calls PyTypeReady() which calls add_methods()
slot_names = mapping.get('slot_names', [])
for offset, name in enumerate(slot_names):
mapping[name] = Member(name, clsname, offset)
return type.__new__(mcls, clsname, bases, mapping, **kwargs)
The object.__new__()
method takes care of creating instances that have
slots instead of an instance dictionary. Here is a rough simulation in pure
Python:
class Object:
'Simulate how object.__new__() allocates memory for __slots__'
def __new__(cls, *args, **kwargs):
'Emulate object_new() in Objects/typeobject.c'
inst = super().__new__(cls)
if hasattr(cls, 'slot_names'):
empty_slots = [null] * len(cls.slot_names)
object.__setattr__(inst, '_slotvalues', empty_slots)
return inst
def __setattr__(self, name, value):
'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c'
cls = type(self)
if hasattr(cls, 'slot_names') and name not in cls.slot_names:
raise AttributeError(
f'{cls.__name__!r} object has no attribute {name!r}'
)
super().__setattr__(name, value)
def __delattr__(self, name):
'Emulate _PyObject_GenericSetAttrWithDict() Objects/object.c'
cls = type(self)
if hasattr(cls, 'slot_names') and name not in cls.slot_names:
raise AttributeError(
f'{cls.__name__!r} object has no attribute {name!r}'
)
super().__delattr__(name)
To use the simulation in a real class, just inherit from Object
and
set the metaclass to Type
:
class H(Object, metaclass=Type):
'Instance variables stored in slots'
slot_names = ['x', 'y']
def __init__(self, x, y):
self.x = x
self.y = y
At this point, the metaclass has loaded member objects for x and y:
>>> from pprint import pp
>>> pp(dict(vars(H)))
{'__module__': '__main__',
'__doc__': 'Instance variables stored in slots',
'slot_names': ['x', 'y'],
'__init__': <function H.__init__ at 0x7fb5d302f9d0>,
'x': <Member 'x' of 'H'>,
'y': <Member 'y' of 'H'>}
When instances are created, they have a slot_values
list where the
attributes are stored:
>>> h = H(10, 20)
>>> vars(h)
{'_slotvalues': [10, 20]}
>>> h.x = 55
>>> vars(h)
{'_slotvalues': [55, 20]}
Misspelled or unassigned attributes will raise an exception:
>>> h.xz
Traceback (most recent call last):
...
AttributeError: 'H' object has no attribute 'xz'


Argparse Tutorial

Argparse Tutorial¶
- author:
Tshepang Mbambo
This tutorial is intended to be a gentle introduction to argparse
, the
recommended command-line parsing module in the Python standard library.
Note
The standard library includes two other libraries directly related
to command-line parameter processing: the lower level optparse
module (which may require more code to configure for a given application,
but also allows an application to request behaviors that argparse
doesn’t support), and the very low level getopt
(which specifically
serves as an equivalent to the getopt()
family of functions
available to C programmers).
While neither of those modules is covered directly in this guide, many of
the core concepts in argparse
first originated in optparse
, so
some aspects of this tutorial will also be relevant to optparse
users.
Concepts¶
Let’s show the sort of functionality that we are going to explore in this introductory tutorial by making use of the ls command:
$ ls
cpython devguide prog.py pypy rm-unused-function.patch
$ ls pypy
ctypes_configure demo dotviewer include lib_pypy lib-python ...
$ ls -l
total 20
drwxr-xr-x 19 wena wena 4096 Feb 18 18:51 cpython
drwxr-xr-x 4 wena wena 4096 Feb 8 12:04 devguide
-rwxr-xr-x 1 wena wena 535 Feb 19 00:05 prog.py
drwxr-xr-x 14 wena wena 4096 Feb 7 00:59 pypy
-rw-r--r-- 1 wena wena 741 Feb 18 01:01 rm-unused-function.patch
$ ls --help
Usage: ls [OPTION]... [FILE]...
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
...
A few concepts we can learn from the four commands:
The ls command is useful when run without any options at all. It defaults to displaying the contents of the current directory.
If we want beyond what it provides by default, we tell it a bit more. In this case, we want it to display a different directory,
pypy
. What we did is specify what is known as a positional argument. It’s named so because the program should know what to do with the value, solely based on where it appears on the command line. This concept is more relevant to a command like cp, whose most basic usage iscp SRC DEST
. The first position is what you want copied, and the second position is where you want it copied to.Now, say we want to change behaviour of the program. In our example, we display more info for each file instead of just showing the file names. The
-l
in that case is known as an optional argument.That’s a snippet of the help text. It’s very useful in that you can come across a program you have never used before, and can figure out how it works simply by reading its help text.
The basics¶
Let us start with a very simple example which does (almost) nothing:
import argparse
parser = argparse.ArgumentParser()
parser.parse_args()
Following is a result of running the code:
$ python prog.py
$ python prog.py --help
usage: prog.py [-h]
options:
-h, --help show this help message and exit
$ python prog.py --verbose
usage: prog.py [-h]
prog.py: error: unrecognized arguments: --verbose
$ python prog.py foo
usage: prog.py [-h]
prog.py: error: unrecognized arguments: foo
Here is what is happening:
Running the script without any options results in nothing displayed to stdout. Not so useful.
The second one starts to display the usefulness of the
argparse
module. We have done almost nothing, but already we get a nice help message.The
--help
option, which can also be shortened to-h
, is the only option we get for free (i.e. no need to specify it). Specifying anything else results in an error. But even then, we do get a useful usage message, also for free.
Introducing Positional arguments¶
An example:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("echo")
args = parser.parse_args()
print(args.echo)
And running the code:
$ python prog.py
usage: prog.py [-h] echo
prog.py: error: the following arguments are required: echo
$ python prog.py --help
usage: prog.py [-h] echo
positional arguments:
echo
options:
-h, --help show this help message and exit
$ python prog.py foo
foo
Here is what’s happening:
We’ve added the
add_argument()
method, which is what we use to specify which command-line options the program is willing to accept. In this case, I’ve named itecho
so that it’s in line with its function.Calling our program now requires us to specify an option.
The
parse_args()
method actually returns some data from the options specified, in this case,echo
.The variable is some form of ‘magic’ that
argparse
performs for free (i.e. no need to specify which variable that value is stored in). You will also notice that its name matches the string argument given to the method,echo
.
Note however that, although the help display looks nice and all, it currently
is not as helpful as it can be. For example we see that we got echo
as a
positional argument, but we don’t know what it does, other than by guessing or
by reading the source code. So, let’s make it a bit more useful:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("echo", help="echo the string you use here")
args = parser.parse_args()
print(args.echo)
And we get:
$ python prog.py -h
usage: prog.py [-h] echo
positional arguments:
echo echo the string you use here
options:
-h, --help show this help message and exit
Now, how about doing something even more useful:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", help="display a square of a given number")
args = parser.parse_args()
print(args.square**2)
Following is a result of running the code:
$ python prog.py 4
Traceback (most recent call last):
File "prog.py", line 5, in <module>
print(args.square**2)
TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'
That didn’t go so well. That’s because argparse
treats the options we
give it as strings, unless we tell it otherwise. So, let’s tell
argparse
to treat that input as an integer:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", help="display a square of a given number",
type=int)
args = parser.parse_args()
print(args.square**2)
Following is a result of running the code:
$ python prog.py 4
16
$ python prog.py four
usage: prog.py [-h] square
prog.py: error: argument square: invalid int value: 'four'
That went well. The program now even helpfully quits on bad illegal input before proceeding.
Introducing Optional arguments¶
So far we have been playing with positional arguments. Let us have a look on how to add optional ones:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--verbosity", help="increase output verbosity")
args = parser.parse_args()
if args.verbosity:
print("verbosity turned on")
And the output:
$ python prog.py --verbosity 1
verbosity turned on
$ python prog.py
$ python prog.py --help
usage: prog.py [-h] [--verbosity VERBOSITY]
options:
-h, --help show this help message and exit
--verbosity VERBOSITY
increase output verbosity
$ python prog.py --verbosity
usage: prog.py [-h] [--verbosity VERBOSITY]
prog.py: error: argument --verbosity: expected one argument
Here is what is happening:
The program is written so as to display something when
--verbosity
is specified and display nothing when not.To show that the option is actually optional, there is no error when running the program without it. Note that by default, if an optional argument isn’t used, the relevant variable, in this case
args.verbosity
, is givenNone
as a value, which is the reason it fails the truth test of theif
statement.The help message is a bit different.
When using the
--verbosity
option, one must also specify some value, any value.
The above example accepts arbitrary integer values for --verbosity
, but for
our simple program, only two values are actually useful, True
or False
.
Let’s modify the code accordingly:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--verbose", help="increase output verbosity",
action="store_true")
args = parser.parse_args()
if args.verbose:
print("verbosity turned on")
And the output:
$ python prog.py --verbose
verbosity turned on
$ python prog.py --verbose 1
usage: prog.py [-h] [--verbose]
prog.py: error: unrecognized arguments: 1
$ python prog.py --help
usage: prog.py [-h] [--verbose]
options:
-h, --help show this help message and exit
--verbose increase output verbosity
Here is what is happening:
The option is now more of a flag than something that requires a value. We even changed the name of the option to match that idea. Note that we now specify a new keyword,
action
, and give it the value"store_true"
. This means that, if the option is specified, assign the valueTrue
toargs.verbose
. Not specifying it impliesFalse
.It complains when you specify a value, in true spirit of what flags actually are.
Notice the different help text.
Short options¶
If you are familiar with command line usage, you will notice that I haven’t yet touched on the topic of short versions of the options. It’s quite simple:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-v", "--verbose", help="increase output verbosity",
action="store_true")
args = parser.parse_args()
if args.verbose:
print("verbosity turned on")
And here goes:
$ python prog.py -v
verbosity turned on
$ python prog.py --help
usage: prog.py [-h] [-v]
options:
-h, --help show this help message and exit
-v, --verbose increase output verbosity
Note that the new ability is also reflected in the help text.
Combining Positional and Optional arguments¶
Our program keeps growing in complexity:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbose", action="store_true",
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbose:
print(f"the square of {args.square} equals {answer}")
else:
print(answer)
And now the output:
$ python prog.py
usage: prog.py [-h] [-v] square
prog.py: error: the following arguments are required: square
$ python prog.py 4
16
$ python prog.py 4 --verbose
the square of 4 equals 16
$ python prog.py --verbose 4
the square of 4 equals 16
We’ve brought back a positional argument, hence the complaint.
Note that the order does not matter.
How about we give this program of ours back the ability to have multiple verbosity values, and actually get to use them:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbosity", type=int,
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity == 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity == 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
And the output:
$ python prog.py 4
16
$ python prog.py 4 -v
usage: prog.py [-h] [-v VERBOSITY] square
prog.py: error: argument -v/--verbosity: expected one argument
$ python prog.py 4 -v 1
4^2 == 16
$ python prog.py 4 -v 2
the square of 4 equals 16
$ python prog.py 4 -v 3
16
These all look good except the last one, which exposes a bug in our program.
Let’s fix it by restricting the values the --verbosity
option can accept:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2],
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity == 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity == 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
And the output:
$ python prog.py 4 -v 3
usage: prog.py [-h] [-v {0,1,2}] square
prog.py: error: argument -v/--verbosity: invalid choice: 3 (choose from 0, 1, 2)
$ python prog.py 4 -h
usage: prog.py [-h] [-v {0,1,2}] square
positional arguments:
square display a square of a given number
options:
-h, --help show this help message and exit
-v, --verbosity {0,1,2}
increase output verbosity
Note that the change also reflects both in the error message as well as the help string.
Now, let’s use a different approach of playing with verbosity, which is pretty
common. It also matches the way the CPython executable handles its own
verbosity argument (check the output of python --help
):
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display the square of a given number")
parser.add_argument("-v", "--verbosity", action="count",
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity == 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity == 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
We have introduced another action, “count”, to count the number of occurrences of specific options.
$ python prog.py 4
16
$ python prog.py 4 -v
4^2 == 16
$ python prog.py 4 -vv
the square of 4 equals 16
$ python prog.py 4 --verbosity --verbosity
the square of 4 equals 16
$ python prog.py 4 -v 1
usage: prog.py [-h] [-v] square
prog.py: error: unrecognized arguments: 1
$ python prog.py 4 -h
usage: prog.py [-h] [-v] square
positional arguments:
square display a square of a given number
options:
-h, --help show this help message and exit
-v, --verbosity increase output verbosity
$ python prog.py 4 -vvv
16
Yes, it’s now more of a flag (similar to
action="store_true"
) in the previous version of our script. That should explain the complaint.It also behaves similar to “store_true” action.
Now here’s a demonstration of what the “count” action gives. You’ve probably seen this sort of usage before.
And if you don’t specify the
-v
flag, that flag is considered to haveNone
value.As should be expected, specifying the long form of the flag, we should get the same output.
Sadly, our help output isn’t very informative on the new ability our script has acquired, but that can always be fixed by improving the documentation for our script (e.g. via the
help
keyword argument).That last output exposes a bug in our program.
Let’s fix:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbosity", action="count",
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
# bugfix: replace == with >=
if args.verbosity >= 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity >= 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
And this is what it gives:
$ python prog.py 4 -vvv
the square of 4 equals 16
$ python prog.py 4 -vvvv
the square of 4 equals 16
$ python prog.py 4
Traceback (most recent call last):
File "prog.py", line 11, in <module>
if args.verbosity >= 2:
TypeError: '>=' not supported between instances of 'NoneType' and 'int'
First output went well, and fixes the bug we had before. That is, we want any value >= 2 to be as verbose as possible.
Third output not so good.
Let’s fix that bug:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbosity", action="count", default=0,
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity >= 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity >= 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
We’ve just introduced yet another keyword, default
.
We’ve set it to 0
in order to make it comparable to the other int values.
Remember that by default,
if an optional argument isn’t specified,
it gets the None
value, and that cannot be compared to an int value
(hence the TypeError
exception).
And:
$ python prog.py 4
16
You can go quite far just with what we’ve learned so far,
and we have only scratched the surface.
The argparse
module is very powerful,
and we’ll explore a bit more of it before we end this tutorial.
Getting a little more advanced¶
What if we wanted to expand our tiny program to perform other powers, not just squares:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
parser.add_argument("-v", "--verbosity", action="count", default=0)
args = parser.parse_args()
answer = args.x**args.y
if args.verbosity >= 2:
print(f"{args.x} to the power {args.y} equals {answer}")
elif args.verbosity >= 1:
print(f"{args.x}^{args.y} == {answer}")
else:
print(answer)
Output:
$ python prog.py
usage: prog.py [-h] [-v] x y
prog.py: error: the following arguments are required: x, y
$ python prog.py -h
usage: prog.py [-h] [-v] x y
positional arguments:
x the base
y the exponent
options:
-h, --help show this help message and exit
-v, --verbosity
$ python prog.py 4 2 -v
4^2 == 16
Notice that so far we’ve been using verbosity level to change the text that gets displayed. The following example instead uses verbosity level to display more text instead:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
parser.add_argument("-v", "--verbosity", action="count", default=0)
args = parser.parse_args()
answer = args.x**args.y
if args.verbosity >= 2:
print(f"Running '{__file__}'")
if args.verbosity >= 1:
print(f"{args.x}^{args.y} == ", end="")
print(answer)
Output:
$ python prog.py 4 2
16
$ python prog.py 4 2 -v
4^2 == 16
$ python prog.py 4 2 -vv
Running 'prog.py'
4^2 == 16
Specifying ambiguous arguments¶
When there is ambiguity in deciding whether an argument is positional or for an
argument, --
can be used to tell parse_args()
that
everything after that is a positional argument:
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-n', nargs='+')
>>> parser.add_argument('args', nargs='*')
>>> # ambiguous, so parse_args assumes it's an option
>>> parser.parse_args(['-f'])
usage: PROG [-h] [-n N [N ...]] [args ...]
PROG: error: unrecognized arguments: -f
>>> parser.parse_args(['--', '-f'])
Namespace(args=['-f'], n=None)
>>> # ambiguous, so the -n option greedily accepts arguments
>>> parser.parse_args(['-n', '1', '2', '3'])
Namespace(args=[], n=['1', '2', '3'])
>>> parser.parse_args(['-n', '1', '--', '2', '3'])
Namespace(args=['2', '3'], n=['1'])
Conflicting options¶
So far, we have been working with two methods of an
argparse.ArgumentParser
instance. Let’s introduce a third one,
add_mutually_exclusive_group()
. It allows for us to specify options that
conflict with each other. Let’s also change the rest of the program so that
the new functionality makes more sense:
we’ll introduce the --quiet
option,
which will be the opposite of the --verbose
one:
import argparse
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
group.add_argument("-q", "--quiet", action="store_true")
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
args = parser.parse_args()
answer = args.x**args.y
if args.quiet:
print(answer)
elif args.verbose:
print(f"{args.x} to the power {args.y} equals {answer}")
else:
print(f"{args.x}^{args.y} == {answer}")
Our program is now simpler, and we’ve lost some functionality for the sake of demonstration. Anyways, here’s the output:
$ python prog.py 4 2
4^2 == 16
$ python prog.py 4 2 -q
16
$ python prog.py 4 2 -v
4 to the power 2 equals 16
$ python prog.py 4 2 -vq
usage: prog.py [-h] [-v | -q] x y
prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose
$ python prog.py 4 2 -v --quiet
usage: prog.py [-h] [-v | -q] x y
prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose
That should be easy to follow. I’ve added that last output so you can see the sort of flexibility you get, i.e. mixing long form options with short form ones.
Before we conclude, you probably want to tell your users the main purpose of your program, just in case they don’t know:
import argparse
parser = argparse.ArgumentParser(description="calculate X to the power of Y")
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
group.add_argument("-q", "--quiet", action="store_true")
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
args = parser.parse_args()
answer = args.x**args.y
if args.quiet:
print(answer)
elif args.verbose:
print(f"{args.x} to the power {args.y} equals {answer}")
else:
print(f"{args.x}^{args.y} == {answer}")
Note that slight difference in the usage text. Note the [-v | -q]
,
which tells us that we can either use -v
or -q
,
but not both at the same time:
$ python prog.py --help
usage: prog.py [-h] [-v | -q] x y
calculate X to the power of Y
positional arguments:
x the base
y the exponent
options:
-h, --help show this help message and exit
-v, --verbose
-q, --quiet
How to translate the argparse output¶
The output of the argparse
module such as its help text and error
messages are all made translatable using the gettext
module. This
allows applications to easily localize messages produced by
argparse
. See also Internationalizing your programs and modules.
For instance, in this argparse
output:
$ python prog.py --help
usage: prog.py [-h] [-v | -q] x y
calculate X to the power of Y
positional arguments:
x the base
y the exponent
options:
-h, --help show this help message and exit
-v, --verbose
-q, --quiet
The strings usage:
, positional arguments:
, options:
and
show this help message and exit
are all translatable.
In order to translate these strings, they must first be extracted
into a .po
file. For example, using Babel,
run this command:
$ pybabel extract -o messages.po /usr/lib/python3.12/argparse.py
This command will extract all translatable strings from the argparse
module and output them into a file named messages.po
. This command assumes
that your Python installation is in /usr/lib
.
You can find out the location of the argparse
module on your system
using this script:
import argparse
print(argparse.__file__)
Once the messages in the .po
file are translated and the translations are
installed using gettext
, argparse
will be able to display the
translated messages.
To translate your own strings in the argparse
output, use gettext
.
Custom type converters¶
The argparse
module allows you to specify custom type converters for
your command-line arguments. This allows you to modify user input before it’s
stored in the argparse.Namespace
. This can be useful when you need to
pre-process the input before it is used in your program.
When using a custom type converter, you can use any callable that takes a single string argument (the argument value) and returns the converted value. However, if you need to handle more complex scenarios, you can use a custom action class with the action parameter instead.
For example, let’s say you want to handle arguments with different prefixes and process them accordingly:
import argparse
parser = argparse.ArgumentParser(prefix_chars='-+')
parser.add_argument('-a', metavar='<value>', action='append',
type=lambda x: ('-', x))
parser.add_argument('+a', metavar='<value>', action='append',
type=lambda x: ('+', x))
args = parser.parse_args()
print(args)
Output:
$ python prog.py -a value1 +a value2
Namespace(a=[('-', 'value1'), ('+', 'value2')])
In this example, we:
Created a parser with custom prefix characters using the
prefix_chars
parameter.Defined two arguments,
-a
and+a
, which used thetype
parameter to create custom type converters to store the value in a tuple with the prefix.
Without the custom type converters, the arguments would have treated the -a
and +a
as the same argument, which would have been undesirable. By using custom
type converters, we were able to differentiate between the two arguments.
Conclusion¶
The argparse
module offers a lot more than shown here.
Its docs are quite detailed and thorough, and full of examples.
Having gone through this tutorial, you should easily digest them
without feeling overwhelmed.