Decorators in Python
—
- python, descriptors, decorators
- 7
- 7
- finished
Meet a decorator:
def decorator(fn):
return fn
def foo():
...
foo = decorator(foo)
It may look differently than your ordinary @decorator, but trust me: it
is the same thing. It’s a callable (like a function, or a functor),
(sidenote: Functor: an object which implements a __call__ method.)
which takes some entity, usually class or function, and returns a
value which replaces it. Usually, when we decorate a function, decorator
returns some other function, which adds some functionality. But it isn’t
necessary to return the same thing. We can return something entirely
different and there’s no decorator police to stop us. Madness and anarchy.
>>> def decorator(fn):
... return "abc"
...
>>> @decorator
... def foo():
... return 123
...
>>> print(type(foo), foo)
<class 'str'> abc
The best decorators do just one thing and then leave the decorated entity alone. That’s for the sake of maintainability and debuggability, but in theory decorator can totally replace whatever it decorates.
Writing decorators as functions is burdensome and not cool at all. They look
like ordinary functions (which they are) and its becomes extra hard to make
cool syntax highlighting just for them. This is why
Python Gods we have introduced a syntactic sugar for boring function
calls: @decorator, which we place just above a thing which we’d like to
change.
@decorator
def foo():
...
Unfortunately, @decorator syntax breaks people’s brains. With foo =
decorator(foo) you immediately know what to expect. When it comes to
@decorator… this StackOverflow’s question question accumulated 3156
points since 2009 and is still growing.
My First Decorator
Let’s make one of the most typical decorators. It will log all calls to the decorated entity:
def decorator(fn):
def wrapper():
print("Call:", fn.__name__)
return fn()
return wrapper
How exactly does it work? When Python encounters a @decorator part, it
takes a function named decorator, takes an entity from line below (our
decorated function), and calls decorator(decorated_function). It then
replaces decorated function with whatever that call returns. Presto.
Decorators aren’t lazy: python executes and evaluates decorators code immediately when it reaches a decorator. Remember: it’s only a syntactic sugar for function call and assign. This is why it’s better if decorators don’t have any heavy side effects, or their execution might gravely slow down startup time of our script or module.
Consider this decorator, which doesn’t even call the decorated foo,
function, to illustrate the idea:
>>> def decorator(fn):
... print("decorator")
... return fn
...
>>> @decorator
... def foo():
... return 1
...
decorator
Because of this, to do anything useful at a runtime, decorators usually
return a closure – a nested callable which references data passed to
decorator: our decorated function. Let’s call it with a nice generic name,
like wrapper.
>>> def decorator(fn):
... def wrapper():
... print("Call:", fn.__name__)
... return fn()
... return wrapper
...
>>> @decorator
... def foo():
... print("foo()")
...
>>> foo()
Call: foo
foo()
Now because of the decorator we’ll automatically get informed about all calls
to foo().
Decorators and Arguments
So far so good, but what happens when the decorated callable accepts
arguments? We can capture them and do whatever we want with them in a
wrapper. Signature of wrapper should match the signature of decorated
callable, or otherwise we’ll silently change its interface, and utterly
confuse our future selves. A common technique is to forward all positional
and keyword arguments to the decorated entity.
(sidenote: An
interesting fact about this technique is that it actually does change the
interface of decorated entity, but because we pass everything to the original
function (stored in a closure), Python will still produce the same results
and errors.)
>>> def decorator(fn):
... def wrapper(*a, **kw):
... print("positional arguments:", *a)
... print("keyword arguments:", kw)
... return fn(*a, **kw)
... return wrapper
...
>>> @decorator
... def foo(i, j, k=None):
... print("-----")
...
>>> foo(1, "abc")
positional arguments: 1 abc
keyword arguments: {}
-----
>>> foo(1, "abc", k="bar")
positional arguments: 1 abc
keyword arguments: {'k': 'bar'}
-----
Decorators can have arguments themselves as well. Parametrizing a decorator can be understood as a way of creating a new instance of decorator every time.
In other words, when we say this:
@decorator(print_calls=True, print_args=False)
def foo():
pass
@decorator(print_calls=True, print_args=True)
def bar():
pass
we mean:
- please create a new decorator for function
foowhich prints its calls, but doesn’t print its arguments; - please create a new decorator for function
barwhich prints both its calls and its arguments.
“Please” is important. It’s always good to be nice to the computers, in case they rule the world one day.
For comparison, without decorator’s syntactic sugar they’d look like this:
foo = decorator(print_calls=True, print_args=False)(foo)
bar = decorator(print_calls=True, print_args=True)(bar)
Because we know what we mean, we can now easily implement a simple function (often called factory) which returns some other function which has a very decorator-specific interface.
def decorator(print_calls=False, print_args=False):
def actual_decorator(fn):
def wrapper(*a, **kw):
if print_calls:
print("Call:", fn.__name__)
if print_args:
print("positional arguments:", *a)
print("keyword arguments:", kw)
return fn(*a, **kw)
return wrapper
return actual_decorator
Don’t be scared by deeply nested functions. It’s just a usual decorator
wrapped by yet another function. This is arguably the easiest method of
creating a closure in Python, so we don’t have to explicitly pass
print_calls and print_args to the wrapper.
Optional arguments
It’s possible to write a decorator which takes both forms: parameter-less
@decorator or @decorator() and decorator with parameters
@decorator(parameters). I think it’s nice touch for user-facing interfaces.
This technique isn’t something they teach at schools, but it’s really, really
simple. All we have to do is to detect in decorator() function what is its
first argument and act accordingly. When it’s set to the callable, then it’s
a good indication that we’re using @decorator form and we should return a
wrapper directly. Otherwise, we’re using one of decorator factories.
def decorator(fn=None, *, parameter=None):
def deco(wfn):
def wrapper(*a, **kw):
do_something(parameter)
return wfn(*a, **kw)
return wrapper
if fn is None: # @decorator(), @decorator(parameter=...)
return deco
return deco(fn) # @decorator
@functools.lru_cache employs this technique.
Using Classes as Decorators
Classes can be used as decorators as well. Arguably, they might be easier to
understand as such, because they have clear distinction between creating a
closure (inside the constructor) and calling it (inside a __call__ dunder
method).
class decorator:
def __init__(self, fn):
self.fn = fn
def __call__(self, *a, **kw):
return self.fn(*a, **kw)
@decorator
def foo():
...
Decorators Stacking
Decorators can be “stacked” on each other, which is an elegant design pattern for the decorator-based composition. The below snippet:
@decorator1
@decorator2
def foo():
...
is equivalent to:
foo = decorator1(decorator2(foo))
The interesting fact is that decorators are bound to the name of callable from the bottom to the top (in reverse order). It totally makes sense: just imagine opening a bracket after each decorator:
# Invalid syntax, for illustrative purposes only
@decorator1(
@decorator2(
def foo():
...
)
)
Now, when we call foo(), we really decorator’s inner wrappers will call in
their natural order: wrapper1 → wraper2 → foo. This is less obvious,
but still the only natural thing which may happen.
>>> def decorator1(fn1):
... print("Applying decorator1")
... def wrapper1(s: str) -> str:
... return fn1(s + "_d1")
... return wrapper1
...
>>> def decorator2(fn2):
... print("Applying decorator2")
... def wrapper2(s: str) -> str:
... return fn2(s + "_d2")
... return wrapper2
...
>>> @decorator1
... @decorator2
... def foo(s: str):
... return s
...
Applying decorator2
Applying decorator1
>>> foo("f")
'f_d1_d2'
It is important distinction. Remember that Python evaluates decorators as soon as code execution reaches them? If decorators have any side effects at that stage, these side effects will be applied in reverse order onto different entities.
A classical example which highlights the importance of decorators order is
stacking cache and property decorators. Correct way to do it is
to stack @property on top of @cache. This way property decorator
doesn’t interfere with memoization implementation, which caches the arguments
passed to decorated function (in this case the self reference).
class Foo:
@property
@cache
def bar(self):
...
Decorators of Classes
Classes are callables too: you call a class to create an instance of object. It means that we can decorate classes as well. This is an opportunity to not only wrap a constructor, but also to modify the class itself. I find this technique a good, easier to understand alternative to metaclasses (discussed below).
def hellostr(cls):
def __str__(self):
return f"Hello, {cls.__name__}, i: {self.i}"
cls.__str__ = __str__
def wrapper(i: int):
return cls(i + 100)
return wrapper
@hellostr
class Foo:
def __init__(self, i: int):
self.i = i
print(Foo(1)) # Hello, Foo, i: 101
In this example we change 2 things: First, we add the __str__ method and
then bind the name Foo to callable which returns the original class, but
called with incremented argument
Metaclass Counterpart
Metaclasses, are sometimes refered to as class factories. They
provide similar possibilities to the decorator above. By defining a custom
callable returned by hellostr decorator we replicated the idea behind
__call__ method of metaclass. Above example could be implemented like this:
class HelloStr(type):
def __new__(metacls, name, bases, ns, **kw):
ns["__str__"] = lambda self: f"Hello, {name}, i: {self.i}"
return super().__new__(metacls, name, bases, ns, **kw)
def __call__(cls, i: int):
return super().__call__(i + 100)
class Bar(metaclass=HelloStr):
def __init__(self, i: int):
self.i = i
print(Bar(1)) # Hello, Bar, i: 101
One of the most prominent class decorators is @dataclass,
which automatically generates a lot of boilerplate code which you’re likely
to write anyway.
Differences to Metaclasses
Metaclasses are a core mechanism to produce classes. Compared to that, decorators are more limited, but still powerful and might be just enough to get the job done. Here are the most prominent differences between metaclasses and decorators regarding the modification of classes.
Composability
Thanks to stacking, we can create many, small decorators which apply changes one after another. On the other hand, class can have only one metaclass, which must do everything. Thus, decorators approach can be cleaner and more maintainable.
Changing Class Before It’s Created
Decorators have access to already created class and can modify it, or even
return a totally different one. But metaclasses, through their __new__
method can alter it even before it is created.
Such alterations usually lie in the realm of hardcore trickery, but if you need to do something before class is even created, then you should use metaclasses.
Base Classes
Inheritance-wise, many magic methods of standard metaclass interface
(__new__, __init__, __prepare__) accept bases parameter, which is a
tuple of base classes for a class. Thus, metaclasses can depend their
behaviour on inheritance. They can even modify this list. It is impossible
for decorator-based approach, because class is already constructed at the
point when decorators start their work.
Classes as Decorators
Any callable can be a decorator. This includes methods of existing objects.
class API:
def decorator(self, fn):
def wrapper(*a, **kw):
return fn(*a, **kw)
return wrapper
api = API()
@api.decorator
def foo():
...
This technique is wide-sperad among many Python libraries and frameworks. A
very popular one is Flask web framework, which uses a global app
object to configure certain aspects of HTTP endpoints. For example decorating
a function with @app.route("/") registers it as a handler for network
requests.
Important Built-in Decorators
@wraps
All above examples have a flaw: by rebinding names to arbitrary callables we
lose their names (fn.__name__) and docstrings (fn.__doc__). We could of
course manually attach them to callables returned by decorators, but
fortunately there is a nifty decorator (sic!) in standard library which does
thas for us. We just have to attach it to the wrapper function.
from functools import wraps
def deco(fn):
@wraps(fn)
def wrapper(*a, **kw):
...
return wrapper
@deco
def foo():
"""Hello"""
...
print(foo.__name__) # foo
print(foo.__doc__) # Hello
@property
@property is a decorator which turns a method of class into a read-only
property.
(sidenote: Descriptor, really.)
From that
point we’ll access values returned by the function through obj.func instead
of obj.func(). Typically it is used to either make a real property (which
is often pseudo-private) immutable or to introduce some logic when modifying
it via a setter.
class Foo:
def __init__(self):
self._prop = 1
@property
def prop(self):
return self._prop
@prop.setter
def prop(self, val):
assert val > 0
self._prop = val
@property
def other_prop(self):
db = Database()
return db.get("other property")
Although property() is a built-in implemented in C, there’s an example of
how pure Python equivalent could look like. @property creates
a descriptor object, which implements special __get__ and __set__ dunder
methods. Apart from that, it’s an ordinary object whose instance is bound to
the prop name. That’s why we can create getters, setters and deleters
separately for each property.
@staticmethod, @classmethod
@staticmethod and @classmethod decorators convert methods to
static and class methods (i.e. methods, which aren’t called on initialized
objects, but on classes themselves).
(sidenote: Technically, class and static methods can be used on objects as well: both
Foo.cmethod() and Foo().cmethod() will work the same.)
The distinction between the two is subtle and often there will be no practical difference between them. Class methods receive class as the implicit first argument and static methods do not.
class Foo:
@classmethod
def cmethod(cls):
...
@staticmethod
def smethod():
...
Class methods are often used to create alternative constructors:
class Foo:
def __init__(self, arg1, arg2, arg3, arg4, arg5):
...
@classmethod
def from_fibonacci(cls):
return cls(1, 1, 2, 3, 5)
@contextmanager
One of my favourite decorators is @contextmanager. It abuses yield
keyword inside a generator function to turn it into a context manager used
with with statements. With @contextmanager Python executes everything
until yield statement, then enters the with block, then executes
everything after yield. This is so much easier than writing ordinary
context managers with __enter__ and __exit__ methods.
One of my favourite context managers created this way temporarily changes the current working directory. I write it for most of my programs which need to operate on a file system. One of its best properties: it can be nested.
from contextlib import contextmanager
@contextmanager
def chdir(path):
cwd = os.getcwd()
try:
os.chdir(path)
yield
finally:
os.chdir(cwd)
with chdir(".."):
do_sth()
with chdir("/var/log"):
do_sth_else()
@cache
@cache decorator is useful for computionally-heavy and frequently called
functions. It remembers the arguments passed to the function and its return
value. Next time the function is called with cached set of arguments,
@cache will return stored return value instead of running the function once
again.
@cache is more modern version of @lru_cache. The latter, however, allows
choosing how many values should be stored, which should prevent unbounded
memory usage.
from functools import cache
@cache
def foo(arg1, arg2):
ret = heavy_computations(arg1, arg2)
return ret
foo(1, 2) # 20 seconds
foo(1, 2) # 0 seconds
foo(1, 3) # 20 seconds
foo(1, 3) # 0 seconds
Ideas for Decorators
Argparse Wrapper
I’ve been using decorators to implement a wrapper for argparse library to simplify creating applications with subcommands.
# SPDX-License-Identifier: GPL-3.0-only
# Copyright (C) 2022 Michał Góral
import argparse
import textwrap
class ArgumentParser:
def __init__(self, *a, **kw):
self.parser = argparse.ArgumentParser(*a, **kw)
self._subparsers = None
self._subpmap = {}
def parse_args(self, *a, **kw):
return self.parser.parse_args(*a, **kw)
def subcommand(self, name: str = None):
def _deco(fn):
if not self._subparsers:
self._subparsers = self.parser.add_subparsers()
subpname = name if name else fn.__name__
parser = self._subparsers.add_parser(
subpname,
description=textwrap.dedent(fn.__doc__),
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.set_defaults(func=fn)
self._subpmap[fn] = parser
return fn
return _deco
def arg(self, *a, **kw):
def _deco(fn):
subp = self._subpmap[fn]
subp.add_argument(*a, **kw)
return fn
return _deco
# Usage:
cli = args.ArgumentParser()
@cli.arg("foo", default=42, help="some help")
@cli.arg("bar", help="some help")
@cli.subcommand()
def mycommand():
pass
It is very important to remember about the order in which decorators are applied and called (see: Decorators Stacking): from the bottom to the top. In this case it means that decorator which creates a subcommand should be just above the decorated function and the order of subcommand’s arguments is reversed as well.
Repeating Unsuccessful Calls
This technique is useful for non-CPU bound operations which program can’t control, like network calls or I/O. Decorator simply repeats the last function when it fails. Just for fun I implemented it as a exponential backoff, with rate limiting in mind.
from time import sleep
def repeat(fn):
t = 1
stop = 32
def wrapper(*a, **kw):
while True:
ret = fn(*a, **kw)
if ret:
return ret
sleep(t)
if t < stop:
t *= 2
return ret
return wrapper
@repeat
def foo():
assert False
Function Registry
This is a big but simple idea. We implement a central (usually global) registry of functions which we can use any way we want. We may implement a plugin system of automatic dispatching of filters and vlidators for registered functions.
Keep in mind that you see only a skeleton. The true fun begins with how you use it.
REGISTRY = []
def register(fn):
REGISTRY.append(fn)
return fn
@register
def foo():
...
@register
def bar():
...
Annotating Callables
Pytest uses this technique to implement test marks: labels attached to test functions which allow users to filter tests which Pytest runs.
In below example we select all registered functions (see above) which are really important to us.
def mark(name: str)
def deco(fn):
setattr(fn, name, True)
return fn
return deco
@register
@mark("important")
def foo():
...
importants = [f for f in REGISTRY if getattr(f, "important", False)]