Have you been hurt by lambdas in Python? | Deephaven

Using lambdas in Python can save developers time and energy by creating higher-order thinking without needing to define a whole new function. However, in some cases, lambdas behave in a confusing manner and can even lead to results that are unexpected or incorrect.

In Gremlins in Python default arguments, we looked at the Principle of Least Astonishment (POLA) and how it relates to default arguments in Python. In this follow-up, let’s look at an interesting case with Python lambdas and show how to debug an anonymous function’s behavior.

What do you think this code prints?

def f():
  lambdas = []

  for i in range(5):
    lambdas.append(lambda x: x+i)

  for l in lambdas:
    print(l(3))

f()

Most users expect to see:

In reality, you see:

This is strange. When evaluated, all of the lambdas have i = 4, even though they are created with i values of 0, 1, 2, 3, and 4.

Maybe Python is creating one lambda for the whole loop. A quick test shows that this is not the case.

def f():
  lambdas = []

  for i in range(5):
    lambdas.append(lambda x: x+i)

  for l in lambdas:
    print(l)
    print(l(3))

f()

What could possibly be happening?

The lambdas in this example are, in computer science lingo, closures. The lambda function is using bound variables from the scope where it is defined – in this case, the i variable.

To understand more, let’s peer into the variables bound to the lambdas. In Python’s data model, __closure__ is a tuple containing the variables a function closes over. Here, I print the address of the closure variables for the lambdas plus the value of i from __closure__.

def f():
  lambdas = []

  for i in range(5):
    lambdas.append(lambda x: x+i)

  for l in lambdas:
    print(l)
    print(f"i_addr={hex(id(l.__closure__[0]))} i={l.__closure__[0].cell_contents}")
    print(l(3))

f()

You will very quickly notice that all of the lambdas share the same tuple of closure variables, since the closure contents have the same address, and all of the lambdas have i = 4! So, all of the lambdas are different, but they all share the same variables. The lambda is associated with the variable name i and not the variable value of i from when it is created. This explains why only the value 7 is ever printed.

Closures can also be created when inner functions are defined. Do they also have the same strange behavior? Yes, they do.

def f():
  funcs = []

  for i in range(5):
    def inner(x):
      return x+i

    funcs.append(inner)

  for l in funcs:
    print(l(3))

f()

We can work around this problem by using default values to bind a new variable for each lambda. In this case, ii.

def f():
  lambdas = []

  for i in range(5):
    lambdas.append(lambda x, ii=i: x+ii)

  for l in lambdas:
    print(l(3))

f()

Under the covers, the lambda now no longer has values in __closure__, because no variables are being enclosed, but it now has the expected values in __defaults__.

def f():
  lambdas = []

  for i in range(5):
    lambdas.append(lambda x, ii=i: x+ii)

  for l in lambdas:
    print(l)
    print(f"closure={l.__closure__} defaults={l.__defaults__}")
    print(l(3))

f()

Hopefully this provides some insight into a situation that can be difficult to debug. Beware of lambdas and inner functions with loops.

Do you think that this violates the Principle of Least Astonishment (POLA)? Let us know in our Slack community.