Let’s get exceptions right

Correct exception definition and handling is easy and actually helps write correct programs. But first, the right way has to be understood. None of the advice given here is without justification. I do not recommend practices because they appeal to my aesthetic sense. While things shouldn’t look wrong, confusing, or ugly, what we find proper or “elegant” should be derived from real-life implications of a given practice, not intuition or “common sense”.

We’ll also see how Python itself makes mistakes using exceptions.

Note

To qualify: we’re talking about Python which uses exceptions for anything that isn’t a correct result - otherwise anything that sways from the happy path. Some exceptions are expected. Exceptions are used for control flow regardless of anyone’s opinion of it. Maybe result types are better, but this is Python and we have to work with it.

Another disclaimer: we’re not talking about perfectionism like never passing formatted messages to exceptions. Only pragmatic advice to make exception handling solid.

Know your error boundaries

An error boundary is a place in the code (a stack frame in an executing program) where all possible expected or unexpected exceptions are handled. Depending on a specific application, upon handling an exception, the error boundary can report an error, send an error response to the proper destination, terminate or restart the process, retry the request, or even reboot the machine.

An error boundary is where handling all possible unexpected exceptions (except Exception) is OK.

In CLI applications error boundaries usually handle unexpected errors by printing an error message, a traceback, possibly sending an error report to a central location, and exiting with an error code. It may looks something like:

def main():
    try:
        run(sys.argv)
    except SomeExpectedError as e:
        print(f"Some expected error {e}", file=sys.stderr)
        sys.exit(1)
    except SomeOtherExpectedError as e:
        print(f"Some other expected error {e}", file=sys.stderr)
        sys.exit(2)
    except Exception as e:
        # sending errors/stats to a central location - useful for internal scripts, probably
        # dodgy in scripts used by 3rd party users
        record_exception_somehow(e)
        stats.increment("script.unexpected_exception", 1, tags=[f"type:{type(e)}"])
        raise  # Will terminate the program and print the traceback.

There are many more fancy things to do in a generic exception handler, for example invoking post-mortem debugging with pdb, however that should only happen if the user asked for it, usually through an env var or a parameter.

In web backends error boundaries are typically located within web frameworks and upon catching an unexpected exception, respond with a 500 Internal Server Error response. They also log the exception with a traceback, roll back the DB transaction, and if a proper 3rd party tool is enabled, send metrics to an aggregator like a statsd-enabled platform and report the error to a platform like Sentry/Rollbar.

If you are unsure if you need an error boundary, you probably don’t. Exceptions can be allowed to bubble up to the program entry point. Python will then print a traceback and exit with an error code.

Error handling strategies are domain-specific

A script may apply the “let it fail” strategy - it will not handle any unexpected exceptions, which means that when they occur, the user is tasked with understanding the traceback. For example a script expecting an ISO-formatted UTC timestamp may format its argument with datetime.fromisoformat without a try block and if the string passed to it isn’t correct, let the exception terminate the program.

A web API path handler (a view) running within a web framework should turn all input format errors into 400 Bad Request, all permission errors into 401 Unauthorized, and only let actual unexpected errors coming from the system become 500 Internal Server Error.

except Exception is not a good coping strategy for dealing with uncertainty

There are 2 valid strategies for dealing with exceptions raised from called code:

• handle specific errors you expect

• “let it fail”

Catching generic exceptions because you feel too anxious about not handling something you should handle is generally NOT a good strategy.

One reason is that it’s much worse to suppress an error and falsely complete successfully than to respond with an error.

Another reason is because some unexpected exceptions may render the current operation purposeless or worse. If writing one file resulted in a “no space left on device” OSError, it doesn’t make sense to try to write another. Or, if working in a DB transaction, it doesn’t make sense to suppress a database error and continue using the same transaction, since the transaction will be aborted at that point (if configured well).

That’s why instead of suppressing exceptions in a generic way via except Exception, you should know what errors you expect from the code you call. This can be difficult, since Python and even its type annotations don’t help us know the types of exceptions raised from a function. It’s unfortunately still up to the developer to document what possible exceptions a function or its callers can raise.

Any unexpected exceptions should bubble up to the error boundary which will act appropriately to the needs of the program.

Distinguish between error- and non-error exceptions

“Errors” are exceptions which are not expected to be caught and should terminate the program or the current request. Those can be AssertionError, SystemExit, or anything you don’t expect. In many cases, it depends on a specific program what’s expected or not. For example you probably don’t want to catch werkzeug.exceptions.BadRequest in Flask app view code.

Other exceptions are expected to be handled, e.g. in a web application an OS-level FileNotFoundError exception may be handled by the app code to become werkzeug.exceptions.NotFound, or in a CLI application, become a SystemExit with a non-zero code.

One exceptional condition, one exception type

Exceptions are discriminated by type so do not reuse exceptions for more than one error type. When in doubt, define another exception type.

Why? To prevent confusion as to where the error came from.

class AppError(Exception):
    pass


def operation_1(element):
    if ...:
        raise AppError("Error in operation 1")
    ...


def operation_2(element):
    if ...:
        raise AppError("Error in operation 2")
    ...


def main():
    unprocessed_elements = []
    for element in elements:
        try:
            operation_1(element)
            operation_2(element)
        except AppError:
            log.exception("Skipped element %r", element)
            unprocessed_elements.append(element)

How do we tell which operation failed? What if we only want to tolerate one of the operations failing, not the other? We can do it like this:

def main():
    unprocessed_elements = []
    for element in elements:
        try:
            operation_1(element)
        except AppError:
            log.exception("Skipped element %r", element)
            unprocessed_elements.append(element)
        else:
            operation_2(element)

But it gets impossible with indirection:

def do_operations():
    with db_transaction:
        operation_1(element)
        operation_2(element)

def main():
    unprocessed_elements = []
    for element in elements:
        try:
            do_operations(element)
        except AppError:
            # Which one?
            log.exception("Skipped element %r", element)
            unprocessed_elements.append(element)

Defining one exception type per error communicates errors better - more information in the type, less information in the string. This is especially important in libraries, since a library user cannot easily change library code to throw more specific exceptions.

The Python stdlib is a horrible offender when it comes to reusing exception types and it won’t change. Since Python throws ValueError from a multitude of places, handling them outside of the very place they emerge from risks suppressing unrelated ValueErrors. See Be super careful when catching commonly raised exceptions

Exception hierarchies are fine

It lets programs tell between validation or permission errors, actually unexpected errors caused by bugs or conditions, or exceptions cause by unavailability of external dependencies. It’s one of the few cases where using inheritance to build a hierarchy is OK.

def work(record_id):
    # may raise network errors or HTTP errors
    resp = requests.get(EXTERNAL_RESOURCE_URL, data={"record_id": record_id}, timeout=30)

    # Will raise HTTP status error
    resp.raise_for_status()

    likelihood = resp.json["likelyhood"]  # Bug - typo, should be "likelihood"

    # May raise if the DB connection is interrupted.
    db.session.execute(
        update(Record).set(likelihood=likelihood).where(record_id=record_id)
    )

Not just the work function but also its callers may choose to handle any subset of the raised exceptions or none of them. This is given to us for free because the libraries in use define their own exception hierarchies rather than reusing exceptions from the stdlib.

Be super careful when catching commonly raised exceptions

E.g. ValueError, KeyError.

Offending example:

try:
    return HANDLERS[route_name](request_data)
except KeyError:
    raise NotFound(route_name)

How it fails? It catches KeyError exceptions thrown from the handler.

Instead, make the try block more slender:

try:
    handler = HANDLERS[route_name]
except KeyError:
    raise NotFound(route_name)
else:
    return handler(request_data)

Don’t reuse exceptions that belong to other projects

Including the Python language.

Offending example:

import re


newline = re.compile(r"\n|\r\n")


# this function has a bug
def decode_list_of_ids(ids_list: str) -> list[int]:
    lines = newline.split(ids_list)

    int_ids = []
    for line in lines:
        line_id = int(line.strip())
        if line_id <= 0:
            raise ValueError("IDs must be positive integers, {} given")

        int_ids.append(line_id)

    return int_ids

How it fails:

def handle_request() -> Response:
    ...

    try:
        ids = decode_list_of_ids(ids_list)
    except ValueError as e:
        return Response(code=400, message=f"Bad input: {e}")

    ...

The function decode_list_of_ids has a bug: if an empty string is passed to it, rather than returning an empty list, it raises ValueError: invalid literal for int() with base 10: ''. Since the type of legitimate validation errors is also ValueError, by handling it, we will suppress the bug and respond with an incorrect error.

You can reuse external exceptions if you’re implementing a protocol, e.g. to define your own collection.

Documenting exceptions

Type annotations don’t include exceptions, which means we’re still in the bike shed. For decades, Sphinx has provided a way to document functions via its autodoc module conventions. While parameter and return types have been superseded by type annotations, raised exception types remain needed.

Sphinx also does a great job of burying the description of its type annotation convention deep in the docs: https://www.sphinx-doc.org/en/master/usage/domains/python.html#info-field-lists. And even then this is confusing. There is no need to include the .. py:function:: directive - this is applied automatically in docstrings.

def function(input: str) -> list[str]:
    """
    Do something.

    :raises SomeError: if the input is kinda ugly or something
    """
    ...

It is only expected to document the exceptions raised by the function or the callees which are not type errors caused by calling the function with incorrect arguments.

Other ways to document exceptions are intentionally excluded, since this article is biased towards Sphinx. Anyway, the legend says that someone once used Google’s conventions outside of Google.