Decorators in Python

Table of Contents

Python has a useful concept of decorator - a callable which returns another callable, usually changing it or extending.

Decorator Syntax and Internals

Meet a decorator:

def decorator(fn):
  return fn

def foo():
  ...

foo = decorator(foo)

Here we have a function which changes some other function and immediately re-binds the new entity to the old name. But decorator doesn’t have to be a function. Any callable works both for decorator and its argument1.

Writing decorators like this would be be burdensome and hard to follow. Re-bound callables could be easily missed, especially when the original ones are long. So instead, we can use a @decorator syntax, which is a syntactic sugar and means exactly the same:

@decorator
def foo():
  ...

Decorators are evaluated at entitie’s definition, so their bodies are executed immediately (not during the actual call). Consider this interactive Python session, which doesn’t even call the decorated foo function, to illustrate the idea:

>>> def decorator(fn):
...   print("decorator")
...   return fn
...
>>> @decorator
... def foo():
...   return 1
...
decorator

Because of this, to do anything useful in the runtime, decorators usually return a closure – a nested callable which references a callable passed to the decorator. Let’s call it wrap:

def decorator(fn):
  def wrap():
    print("decorator")
    return fn()
  return wrap

@decorator
def foo():
  ...

Now, whenever we call foo(), a “decorator” string will be printed to the terminal.

Decorators and Arguments

So far so good, but what happens when the decorated callable accepts arkuments? In this case, the callable returned by the decorator should accept them as well. Otherwise we won’t be able to properly call it. A common techique is to simply forward all positional and keyword arguments to the original entity.

def decorator(fn):
  def wrap(*a, **kw):
    print("decorator")
    return fn(*a, **kw)
  return wrap

@decorator
def foo(i, j):
  ...

Decorators can have arguments themselves as well. This is “emulated” by creating yet another closure. We wrap the decorator in yet another callable, which accepts arguments and returns a “parametrized decorator”. For example, to decorate a function like this:

@parametrized(a=11, b=12)
def foo(bar):
  print(bar)

We must create a callable named parametrized which returns a real decorator, which in turn returns a callable which is bound to the foo name. In this context we can treat @parametrized(a=11, b=12) as an ordinary call made in the place of definition of function foo.

Function which implements this idea will look like this:

def parametrized(a, b):
  def decorator(fn):
    def wrap(*a, **kw):
      print(f"decorator with parameters a={a}, b={b}")
      return fn(*a, **kw)
    return wrap
  return decorator

Decorators Stacking

Decorators can be “stacked” on each other, which is an elegant design pattern for the kind of decorator-based composition. The below snippet:

@decorator1
@decorator2
def foo():
  ...

is equivalent to:

foo = decorator1(decorator2(foo))

The interesting fact is that decorators are bound to the name of callable from the bottom to the top (in reverse order), but in the runtime the reference to decorated callable will be passed from the top to the bottom.

def decorator1(fn1):
  print("Applying decorator1")
  def wrap1(s: str) -> str:
    return fn1(s + "_d1")
  return wrap1

def decorator2(fn2):
  print("Applying decorator2")
  def wrap2(s: str) -> str:
    return fn2(s + "_d2")
  return wrap2

@decorator1
@decorator2
def foo(s: str):
  return s

# Applying decorator2
# Applying decorator1

foo("f")  # returns "f_d1_d2"

The reason for this is that after decorator application name foo is bound to wrap1. When we call foo(), in reality we call wrap1, which adds _d1 to the input string first before calling the inner fn1. fn1 is bound to wrap2, which is called with f_dn1. It adds _d2 to the input string and calls the original foo, which now simply returns the string.

Of course this behaviour depends on decorator’s implementation. Nothing in the world prevents our decorators to not call the original callable at all. Decorators can modify them in other ways or just run some additional code, without defining their own wrappers. In this case the decorator order is even more important.

A classical example which highlights the importance of decorators order is using built-in cache and property decorators The correct way to do it is to stack @property on top of @cache. This way property decorator doesn’t interfere with memoization implementation, which caches the arguments passed to decorated function (in this case the self reference).

class Foo:
  @property
  @cache
  def bar(self):
    ...

Decorators of Classes

Classes are callables as well, so decorating them works exatly the same.

def hellostr(cls):
  def __str__(self):
    return f"Hello, {cls.__name__}, i: {self.i}"

  cls.__str__ = __str__

  def wrap(i: int):
    return cls(i + 100)

  return wrap

@hellostr
class Foo:
  def __init__(self, i: int):
    self.i = i

print(Foo(1))  # Hello, Foo, i: 101

In this example we change 2 things in decorated class. First, we add (or change) the __str__ method and then bind the name Foo to callable which returns the original class, but called with increased argument

Metaclass Counterpart

Metaclasses, sometimes called class factories, provide similar possibilities to the decorator above. By defining a custom callable returned by hellostr decorator we replicated the idea behind __call__ method of metaclass. Above example could be implemented like this:

class HelloStr(type):
  def __new__(metacls, name, bases, ns, **kw):
    ns["__str__"] = lambda self: f"Hello, {name}, i: {self.i}"
    return super().__new__(metacls, name, bases, ns, **kw)

  def __call__(cls, i: int):
    return super().__call__(i + 100)

class Bar(metaclass=HelloStr):
  def __init__(self, i: int):
    self.i = i

print(Bar(1))  # Hello, Bar, i: 101

Differences to Metaclasses

Metaclasses is a core mechanism devised to produce classes and has unlimited possibilities in this area. Compared to that, decorators are more limited, but still powerful and might be just enough to get the job done. Here are the most prominent differences between metaclasses and decorators regarding the modification of classes.

Composability

Thanks to stacking, we can create many, small decorators which apply changes one after another. On the other hand, class can have only one metaclass, which would have to contain all of the code. Thus, decorators approach can be cleaner and more maintainable.

Changing Class Before It’s Created

Decorators have access to already created class and can modify it, or even return a totally different one. But metaclasses, through their __new__ method can alter it even before it is created.

Such alterations usually lie in the realm of hardcore trickery, but if you need to do something before class is even created, then you should use metaclasses.

Base Classes

Inheritance-wise, many magic methods of standard metaclass interface (__new__, __init__, __prepare__ accept bases parameter, which is a tuple of base classes for a class. Thus, metaclasses can depend their behaviour on class’ base clases. They can even modify this list. It is impossible for decorator-based approach, because class is already constructed at the point when decorators start their work.

Classes as Decorators

Any callable can be a decorator. This includes object methods.

class API:
  def decorator(self, fn):
    def wrap(*a, **kw):
      return fn(*a, **kw)
    return wrap

api = API()

@api.decorator
def foo():
  ...

This technique wide-sperad among many Python libraries and frameworks. A very popular example is Flask web framework, which uses a global app objet to configure certain aspects of HTTP endpoints. For example decorating a function with @app.route("/") registers it as a handler for network requests.

Important Built-in Decorators

@wraps

All above examples have a flaw: by rebinding names to arbitrary callables we lose their names (fn.__name__) and docstrings (fn.__doc__). We could of course manually attach them to callables returned by decorators, but fortunately there is a nifty decorator in standard library which does thas for us:

from functools import wraps

def deco(fn):
  @wraps(fn)
  def wrap(*a, **kw):
    ...
  return wrap

@deco
def foo():
  """Hello"""
  ...

print(foo.__name__)  # foo
print(foo.__doc__)   # Hello

@property, @staticmethod, @classmethod

Python standard library defines a couple of decorators intended for class methods.

First is @property - a decorator which turns a function into a “property attribute”, meaning that we now access values returned by the function through obj.func instead of obj.func(). Typically it is used to either make a real property (often private) immutable or to introduce some logic when obtaining property.

class Foo:
  def __init__(self):
    self._prop = 1

  @property
  def prop(self):
    return self._prop  # we can modify obj.prop

  @property
  def other_prop(self):
    db = Database()
    return db.get("other property")

Next, there are @staticmethod and @classmethod which convert methods to static and class methods (i.e. methods, which aren’t called on initialized objects, but on classes themselves2).

The distinction between the two is subtle and often there will be no practical difference between them. Class methods receive class as the implicit first argument and static methods do not.

class Foo:
  @classmethod
  def cmethod(cls):
    ...

  @staticmethod
  def smethod():
    ...

Class methods are often used to create alternative constructors for the class:

class Foo:
  def __init__(self, arg1, arg2, arg3, arg4, arg5):
    ...

  @classmethod
  def from_fibonacci(cls):
    return cls(1, 1, 2, 3, 5)

@contextmanager

One of my favourite decorators is @contextmanager which changes a generator function which yields exactly one value into a context manager for with statements. With @contextmanager Python executes everything until yield statement, then enters the with block, then executes everything after yield. This is so much easier than writing ordinary context manager with __enter__ and __exit__ methods.

One of my favourite context managers created this way temporarily changes the current working directory. I write it for most of my programs which need to operate on a file system. One of its best properties: it can be nested.

from contextlib import contextmanager


@contextmanager
def chdir(path):
  cwd = os.getcwd()
  try:
    os.chdir(path)
    yield
  finally:
    os.chdir(cwd)


with chdir(".."):
  do_sth()
  with chdir("/var/log"):
    do_sth_else()

@cache

@cache decorator is useful for computionally-heavy and frequently called functions. It remembers the arguments passed to the function and its return value. Next time the function is called with cached set of arguments, @cache will return stored return value instead of running the function once again.

@cache is more modern version of @lru_cache. The latter, however, allows choosing how many values should be stored, which should prevent unbounded memory usage.

from functools import cache

@cache
def foo(arg1, arg2):
  ret = heavy_computations(arg1, arg2)
  return ret

foo(1, 2)  # 20 seconds
foo(1, 2)  # 0 seconds
foo(1, 3)  # 20 seconds
foo(1, 3)  # 0 seconds

Ideas for Decorators

Argparse Wrapper

I’ve been using decorators to implement a wrapper for argparse library to simplify creating applications with subcommands.

The source code of the snippet below is provided under the terms of GPLv3, contrary to the general license of the snippets presented on this page.
# SPDX-License-Identifier: GPL-3.0-only
# Copyright (C) 2022 Michał Góral

import argparse
import textwrap


class ArgumentParser:
    def __init__(self, *a, **kw):
        self.parser = argparse.ArgumentParser(*a, **kw)
        self._subparsers = None
        self._subpmap = {}

    def parse_args(self, *a, **kw):
        return self.parser.parse_args(*a, **kw)

    def subcommand(self, name: str = None):
        def _deco(fn):
            if not self._subparsers:
                self._subparsers = self.parser.add_subparsers()

            subpname = name if name else fn.__name__
            parser = self._subparsers.add_parser(
                subpname,
                description=textwrap.dedent(fn.__doc__),
                formatter_class=argparse.RawDescriptionHelpFormatter,
            )
            parser.set_defaults(func=fn)
            self._subpmap[fn] = parser
            return fn

        return _deco

    def arg(self, *a, **kw):
        def _deco(fn):
            subp = self._subpmap[fn]
            subp.add_argument(*a, **kw)
            return fn

        return _deco


# Usage:
cli = args.ArgumentParser()

@cli.arg("foo", default=42, help="some help")
@cli.arg("bar", help="some help")
@cli.subcommand()
def mycommand():
    pass

It is very important to remember about the order in which decorators are applied and called (see: Decorators Stacking): from the bottom to the top. In this case it means that decorator which creates a subcommand should be just above the decorated function and the order of subcommand’s arguments is reversed as well.

Repeating Unsuccessful Calls

This technique is useful for non-CPU bound operations which program can’t control, like network calls or I/O. Predicate simply repeats the last function when it fails. Just for fun I implemented it as a exponential backoff, with rate limiting in mind.

from time import sleep

def repeat(fn):
  t = 1
  stop = 32
  def wrap(*a, **kw):
    while True:
      ret = fn(*a, **kw)
      if ret:
        return ret
      sleep(t)
      if t < stop:
        t *= 2
    return ret
  return wrap


@repeat
def foo():
  assert False

Function Registry

This is a big but simple idea. We implement a central (usually global) registry of functions which can be used in any way we want. Some ideas include implementing plugin systems or automatic dispatching of filters, validators etc.

Keep in mind that you see only a skeleton. The true fun begins with how you use it.

REGISTRY = []

def register(fn):
  REGISTRY.append(fn)
  return fn

@register
def foo():
  ...

@register
def bar():
  ...

Annotating Callables

This technique is used by pytest to implement test marks which allow running tests based on some user-defined labels. In below example we select all registered functions (see above) which are really important to us.

def mark(name: str)
  def deco(fn):
    setattr(fn, name, True)
    return fn
  return deco

@register
@mark("important")
def foo():
  ...

importants = [f for f in REGISTRY if hasattr(f, "important")]

  1. Technically, decorators don’t have to return functions. They can return any object. They also don’t have to take callables as their arguments – any object will work. However, due to Python syntax requirements, we can only decorate callables, so a decorator which takes, say, string, would have to be stacked over a decorator which transforms a callable to the string. 

  2. technically, class and static methods can be used on objects as well: both Foo.cmethod() and Foo().cmethod() will work the same.