Default arguments in Python: two easy blunders

by garth on May 9, 2008

I’m glad I stumbled across Patrick Altman’s tweet about a “default bug in Django“. I’d never have guessed you can pass a callable to a field’s default= argument, otherwise. That’s quite a powerful idiom, and I think I’ll use it a lot.

To balance the karma, I’d like to post a quick reminder to everyone else that expressions in default arguments are calculated when the function is defined, not when it’s called. In Patrick’s code, for example, all objects created in the same running session got the same timestamp. Try this in the Python interactive prompt:

>>>import time
>>> def report(when=time.time()): 
...     print when
... 
>>> report()
1210294387.19
>>> time.sleep(5)
>>> report()
1210294387.19

Until the interpreter quits, you’ll always get the same timestamp. The correct way to go about this is to default to None or some other sentinel, then replace it inside the function:

>>> def report(when=None): 
...     if when is None:
...         when = time.time()
...     print when
... 
>>> report()
1210294762.29
>>> time.sleep(5)
>>> report()
1210294772.23

Now that you know about that blunder, you should be able to figure out what’s going on with this second classic blunder when using default arguments in Python:

>>> def spam(eggs=[]): 
...     eggs.append("spam")
...     return eggs
... 
>>> spam()
['spam']
>>> spam()
['spam', 'spam']
>>> spam()
['spam', 'spam', 'spam']
>>> spam()
['spam', 'spam', 'spam', 'spam']

{ 1 trackback }

Python pitfall: Passing Mutable Objects as Default Args « Parerga und Paralipomena
05.19.09 at 1:18 am

{ 20 comments }

Will 05.09.08 at 11:40 am

Why not something like this:

import time
def report(when=time.time):
if callable(when):
when=when()
print when

Now you have a dynamic default, and you can pass a callable.

Jay 05.09.08 at 12:57 pm

The standard Python idiom is to do this:

>>> def report(when=None):
… when = when or time.time()
… print when

See help(‘or’) for details.

You can use `and` similarly for side effects:

file = None

try:
file = open(‘/tmp/somefile’)
finally:
file and file.close()

(obviously using `with` is easier but this is the clearest example I could give)

Mike Pirnat 05.09.08 at 1:28 pm

This is one of those real eye-openers about Python. It can make for some exceptionally subtle and hard-to-spot bugs, and is especially fun if your mutable default (list, dictionary, whatever) is exposed to more than one user (for example, in a web application), as the actions of one user can pollute the experience of others.

Parand 05.09.08 at 3:08 pm

Second one got me good in a web app, took quite a while to figure it out.

LMZ 05.09.08 at 3:13 pm

the last example is bug or *feature* ?!

k3rni 05.09.08 at 4:30 pm

And the last one (using a list as default argument) is a trick question we used at job interviews for Python developers. Most people got it wrong, and if someone did know the answer, that usually meant he had all the other needed skills capabilities, among them a Python-focused mind.

Tom 05.09.08 at 8:07 pm

The solution is to “pass a callable to a field’s default= argument,”

>>> def report(when=time.time):
… print when()
>>> report()
1210327491.01
>>> report()
1210327492.29

right?

Tom 05.09.08 at 9:43 pm

I was hit by number 2 a few days ago on a project, took me a long time to figure out what was going on.

Nick 05.10.08 at 12:55 am

Just watch this useful video, you’ll learn a lot of python concepts.
Understanding Python
http://video.google.com/videoplay?docid=7760178035196894549&q=python&ei=eGUkSLqADpeG2wL1lZyfDA

Fredrik 05.10.08 at 1:20 am

Patrick’s example has nothing to do with Python’s default argument handling, though. His mistake was thinking that if you write

x = expression

and then evaluate “x” some time in the future, Python would by some reason evaluate the expression again. I’m sure he knows that things doesn’t work this way, and that it was just the unusual context (defining models by setting class attributes) that was confusing him.

saluk 05.10.08 at 12:14 pm

Alternately, this (mis?)feature can be used to your advantage in certain situations:

def load_file(filename,cache={}):
if filename not in cache:
cache[filename] = open(filename)
return cache[filename]

Still, this does cause more bugs than I would like. Knowing a little bit of how python works I can’t think of a different way to do this to keep this from happening. Another idiom perhaps is this:

def getval(f):
if callable(f):
return f()
return f
def report(when=time.time):
print getval(when)

The only case where this doesn’t work of course, is if the arg is meant to hold a function and not a value, but this is rare (and doesn’t suffer from the default problem). I like this better than having a bunch of None default values in my constructors.

I tried to think of a decorator solution, but it seems like you will still have the early execution thing, and what we are after is lazy execution. But it can’t happen every time, because you do want to be able to set a default value based on the output of a function in some situations.

Do we need a function that evaluates to itself the first time, and then generates values after that? Maybe a generator solution?

Mutables are an even harder issue though, and I think that is a much more common bug as well than the functional one. I wouldn’t mind warnings for assigning mutables to default arguments.

David 05.10.08 at 11:06 pm

For the report function, I would use this implementation :
>>> def report(when=time.time):
… print when()

Note that no parenthesis are used in the default parameter.

me 05.11.08 at 8:58 pm

the report example would be simpler with

def report(when=None):
when = when or time.time()
print when

Jasko 05.13.08 at 3:26 am

How about something like this to handle both cases?

def report(when=time.time):
… if hasattr(when, “__call__”):
… print when()
… else:
… print when

>>> report()
1210613185.87
>>> report(“Now”)
Now

Though regarding your second example — yowza! I can see a sleepless night worrying about that one!

garth 05.15.08 at 11:24 am

I just batch-approved all these excellent comments. Thanks in particular for all the reminders to use the ‘or’ operator. Cheers!

swr 06.02.08 at 6:50 am

Jay said,
ON MAY 9TH, 2008 AT 12:57 PM

file = None

try:
file = open(’/tmp/somefile’)
finally:
file and file.close()

why putting ‘open’ into the try-finally block?
If it fails there is nothing to close anyway.
I was repeating such construction many times until I realized it’s a bit illogical.

garth 06.12.08 at 3:51 pm

I put the open() with the rest of the code that does the work; I find it easier to read. Great use of an “and” in that last line, too.

Denis 07.02.08 at 4:38 am

I just hit Blunder#2. I’m pretty sure it isn’t the first time, but I felt compelled to get to the root of it once and for all. In the end, I feel disappointed with this violation of the “you get what you expect” principle that generally pervades the realm of Pythonese.

If I had wanted a class instance variable, I would have declared one, I’m not shy about it, honest. Is this truly not considered a bug in the interpreter?

Omer 09.14.08 at 12:07 pm

Omg. Thanks so much! I’ve been struggling with this same problem for like an hour!

SEO 09.18.08 at 7:59 am

I had similar problems to the second one using delayed javascript functions. Thanks for the heads up.

Comments on this entry are closed.