Variable scope in list comprehension vs. generator expression

I just got bitten by a subtle difference between Python's list comprehensions and generator expressions.

First let me explain what those are, in case you haven't heard those terms before.

This is a list comprehension:

[a * 2 for a in (1, 2, 3)]

This means, create a list where for every item a in the sequence (1, 2, 3), the corresponding value will be a * 2. So when you run this, you'll get:

>>> [a * 2 for a in (1, 2, 3)]
[2, 4, 6]

In Python 2, this is implemented very much like a for..in loop. The loop variable a is in scope even after the list comprehension is finished:

>>> a
3

This, on the other hand, is a generator expression:

(b * 2 for b in (1, 2, 3))

(The parenthesis aren't part of the generator expression per se, but as a generator expression has to be surrounded by parenthesis, you certainly get that impression.)

The result of this is not a list, but a generator. You can get the values out of a generator by applying the list function on it:

>>> list(b * 2 for b in (1, 2, 3))
[2, 4, 6]

The result seems to be the same, but there is a subtle difference: When you use a generator expression, the loop variable does not "leak" into the current scope:

>>> b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'b' is not defined

I was surprised to find out this after trying to figure out why I got this strange error in a Django application:

TypeError: 'class Meta' got invalid attribute(s): p

The code causing this was an attempt at using Django 1.7's new feature where you can specify a set of default permissions. The code went something like this:

class MyModel(models.Model):
    class Meta:
        # Declare the available permissions
        permissions = (
            ('update_project', 'Can update project'),
            ('invite_to_project', 'Can invite members to project'),
            ('delete_project', 'Can delete project'),

        # Include all of the permissions in the defaults
        default_permissions = [p[0] for p in permissions]

The error was caused by the variable p leaking into the scope of the Meta class and appearing as an attribute of it. As Django expects all attributes of the Meta class to have some meaning, and it doesn't understand the meaning of p, I got the error above. It took me some time to figure out why my little variable name inside my little list comprehension appeared in an error message.

The solution? Use a generator expression instead of a list comprehension.

        default_permissions = list(p[0] for p in permissions)

Alternatively, switch to Python 3, where loop variables never leak like this, even in list comprehensions.

Comments disabled on this post.