The surprise
I haven't been surprised by Python in a long time, but today I got surprised by lambdas and how do they capture their surrounding state.
>>> def set(i):
... print 'SET', i
...
>>> l = []
>>> for i in range(3):
... l.append(lambda: set(i))
...
>>> l[0]
<function <lambda> at 0x7b0b0>
>>> l[0]()
SET 2
The explanation
I would have expected the value of i to be captured there and then, but it turns out that lambda only stores the name i. This is easily demonstrated by the following:
>>> i = 123
>>> l[0]()
SET 123
However, if this is done inside a function:
>>> def aaaa():
... i = 321
... l[0]()
...
>>> aaaa()
SET 123
So lambdas capture the scope dictionary as well.
The solution
I found two workarounds. Both amount to storing the value of the required scope somewhere else:
def f(i):
... return lambda: set(i)
from functools import partial
partial(set, i)
I think I'll go for the partial solution, just because I don't want to declare a new function just for this.
Comments
Comment by Orestis Markou , 9 months, 3 weeks ago :
That makes sense although it looks ugly IMO.
I knew about how loops don't create a new scope, but this is the first time I'm actually bitten by that - I guess I don't use loops and closures together that much :)
Comment by Jonathan Hartley , 9 months, 3 weeks ago :
Ah-ha! I didn't realise that either. Thanks for the write-up!
Comment by Paul Boddie , 9 months, 3 weeks ago :
This behaviour changed when closures were added to Python. Previously, lambdas had their own strict local scope and the only way to capture state from the function in which the lambda is defined was to define default parameters involving the "outer" objects, as Michael demonstrates.
I'm not a huge fan of closures, myself. There are supposedly more compelling reasons for having them in certain languages, but the moaning of advocates of such languages combined with the usual levels of language envy amongst Python developers conspired to push this feature in.
Not that anyone really started using Python purely because this feature got added, but I guess it has its uses and some people seem happy with it.
Comment by Michael Foord , 9 months, 3 weeks ago :
Sorry, I should have said *another* way of capturing the value bound to a name inside a loop is to use the default argument. Your ways are completely valid of course.
Personally I love closures and use them regularly. I recently had to do some programming in a language without closures (VB.NET) and having to create classes just to capture state (and workaround the lack of closures) made me sad.
Comment by rgz , 9 months, 3 weeks ago :
Me too, creating anonymous classes in Java is depressingly verbose.
I don't think Python users suffer excessive language envy, we don't see much clamor for automatic variables or private attributes. Interpolation, a little, but not enough.
Code blocks/closures are the only features that is constantly proposed... the trinary operator too...
Mind you, it seems most the features that are constantly envied are about using statements as values...
Comment by Lennart Regebro , 9 months, 3 weeks ago :
Lambdas are just anonymous functions. It gets easier when you realize that
>>> for i in range(3):
... l.append(lambda: set(i))
Can be written
>>> def foo():
... return set(i)
>>> for i in range(3):
... l.append(lambda: foo())
In which case it's obvious i has a late binding.
I wish lambda would have been renamed "def" in Python 3, and these things would have been obvious. Ah well.
This post is older than 30 days and comments have been turned off.
Comment by Michael Foord , 9 months, 3 weeks ago :
The normal way of capturing the current value of a name in a lambda is:
lambda i=i: set(i)
The default value is bound immediately and doesn't have the late bound problems with functions. Note that the behaviour of lambdas here is identical to named functions - name lookup is done dynamically in the enclosing scope where the function is defined.
Loops (etc) *don't* create a new scope.