[Nallo]

This is not a question, just a little puzzle for (beginning)/intermediate python programers

I got a little inspired by atraubs 'spot the error' topic in the advanced discussion forum. So here is a little bug finding problem for intermediate python programmers (experts please keep your tongue in check). The first and second print statement show the expected result. The third does not! Can you explain the bug?

def append_u_words(words, return_list=[]):
    """Supposed to return a list where all elements in words starting with 'u'
    have been added to return_list. And if return_list is not given as an
    argument returns a list of the strings in words starting 'u'.
    """
    u_words = [word for word in words if word.startswith('u')]
    return_list += u_words
    return return_list

print append_u_words(["user", "tomorrow"]) #should print ['user']
print append_u_words(["unity", "yabbayabba"], ["still_works"]) #should print ['still_works', 'unity']
print append_u_words(["uncool"]) #should print ['uncool'] , but does something else

[jon.kiparsky]

I'm not exactly an advanced python user, but it's clear what's happening. The question is, why?

This is clearly intentional behavior (at least, I'd think it could hardly arise by accident) so what's the advantage? The function definition looks like it defines return_list to be an empty list - why does it instead define it to be "the previous value"?

If I didn't know better, I'd say the bug is in the behavior of python!

This post has been edited by jon.kiparsky: 06 August 2012 - 05:17 PM

[sepp2k]

jon.kiparsky, on 07 August 2012 - 02:16 AM, said:

The function definition looks like it defines return_list to be an empty list - why does it instead define it to be "the previous value"?

It doesn't. If you call the function with an explicit second argument foo and then afterwards call it without one, the value of return_list won't be foo. Also if the function reassigned the value of return_list instead of mutating it, subsequent calls would be unaffected.

What is happening is that default arguments in Python are not re-evaluated each time the function is called. Instead they're evaluated once when the function definition is parsed. So every time the function is called without a second argument, a reference to the same list will be used as the default argument. So any changes done to that list will be visible in subsequent calls that also don't supply a second argument - though changes to the variable itself would not be.

[Nallo]

Yes, sepp2k you are right on the spot. That's why I pleaded experienced python programmers not to answer so soon (and you are certainly on that list). You spoiled it :-(

[sepp2k]

Sorry, I didn't mean to spoil anything. Jon more or less already had it anyway, I just corrected his details. I thought the challenge was over.

PS: I wouldn't really consider myself a Python programmer (experienced or otherwise). I've never worked on a Python project in my life.

[jon.kiparsky]

Quote

It doesn't.

This is a matter of perspective, but I think "keeping the same value" and "defining it to the previous value" are identical in their effects, no? The implementations specified by one phrasing or another are distinct, but from my point of view as the caller of the code, the two have the same outcome. I phrased it as I did to highlight the apparent absurdity of

return_list = []

actually meaning

return_list = [] or whatever it was the last time you called it or whatever value you give it this time, depending

So since I'm new to the language, is there a reason for this to be implemented this way? Or is this just just a python "gotcha" that we should surround with yellow warning tape and orange cones?
I'm really curious about this - since python is generally a pretty well-made language, it seems hard for me to imagine that something like this would be done by accident, but if I came across this in another language, say, PHP or something, I'd by pointing and laughing and mocking it. So what gives? What makes this a good decision?

[Nallo]

well, the point was to understand, that default values get assigned only once. When python runs the def the first time. And if it was a mutable ... of course it would retain everything put into it.

Well, nice doing guys. I just would have liked more time for others to scratch their heads about it.

[sepp2k]

jon.kiparsky, on 07 August 2012 - 02:54 AM, said:

This is a matter of perspective, but I think "keeping the same value" and "defining it to the previous value" are identical in their effects, no? The implementations specified by one phrasing or another are distinct, but from my point of view as the caller of the code, the two have the same outcome.

It doesn't keep the same value. To illustrate what I meant in my first post with an actual code example:

def f(x=[]):
  x.append(0)
  return x

f() # Returns [0]
f([1,2,3]) # Returns [1,2,3,0]
f() # Return [0,0]

If it just used the same value as the last call, the last one should return [1,2,3,0,0]. Likewise if you reassigned x inside the function.

Quote

So since I'm new to the language, is there a reason for this to be implemented this way?

Performance, I suppose (not that Python is a very performance-aware language)? Or possibly Guido (or whoever first implemented the feature) just didn't think much about the consequences of that choice and just did what first came to mind or was easiest to implement. Frankly I've got no clue.

Quote

Or is this just just a python "gotcha" that we should surround with yellow warning tape and orange cones?

It is. This gotcha is the very reason why it's an idiom in Python to set None as the default value for arguments and then set them to the real default value inside the function body if it's None. Which is a very unwieldy way to work around this.

Quote

What makes this a good decision?

It's not.

This post has been edited by sepp2k: 06 August 2012 - 06:06 PM

[Nallo]

To Jon:

No dont put a red tape around it. The way I wrote the function is completely not Pythonic. It was just a pitfall you can get into when using a mutable as a deflaut argument. You don't want that in Python.

One would rather have:

def f(l=None):
    if l is None:
        l = []
    #something

instead of

def f(l=[]):
    #something

[Nallo]

Quote

Quote

What makes this a good decision?

It's not.

Ok, I get it. Bye bye everyone. I wont come back.

Anyone knows, how I can delete my account here?

This post has been edited by Nallo: 06 August 2012 - 06:20 PM

[jon.kiparsky]

sepp2k, on 06 August 2012 - 08:05 PM, said:

It doesn't keep the same value. To illustrate what I meant in my first post with an actual code example:

If it just used the same value as the last call, the last one should return [1,2,3,0,0]. Likewise if you reassigned x inside the function.

Sorry, I meant "it sometimes retains the same value... or not" Very much like your example illustrates. Maybe it's the same... maybe not...

Quote

Quote

What makes this a good decision?

It's not.

Okay, that's a relief actually. 'Cause I was torturing my brain trying to figure out how it was a good idea, and nothing was coming to mind... and I'm not usually that slow on the uptake!


@Nallo - I'm sorry, but your examples don't really set my mind at ease I'm afraid, precisely because the function definition, to the naive user like myself, looks so much like it does this for you - that's what makes this a gotcha. This is definitely a red-flag issue for me.


All that being said, I can understand how the language could have come to be this way, and I suspect that if I were to delve into the internals of python this would not only make sense but fit in with other parts of the language that I am not aware of - but from the outside, especially if you've studied any other language, this looks very wrong.

Nallo, on 06 August 2012 - 08:19 PM, said:

Quote

Quote

What makes this a good decision?

It's not.

Ok, I get it. Bye bye everyone. I wont come back.

Anyone knows, how I can delete my account here?

I surely hope you're not going to leave just because I don't understand what it is you're trying to get across here. If there's something correct about this, I'd very much like to know what it is, because then I'll be a better python programmer. When I say why it looks wrong to me, the point is not that it is in fact wrong. The point is that I don't see how it's right!
You could at least do us (or me at least) the courtesy of explaining your example before you go!

[atraub]

I actually once emailed Guido Van Rossum about this behavior. He was very familiar with it and said that it wouldn't be fixed due to backward compatibility reasons. I wasn't very satisfied with his answer, but what are ya gonna do?


ALSO:
I love the idea of a "spot the bug" trend in posts. I think this could be a pretty fun way to get more community involvement. I would say that purplse, blues, teals, reds, etc... should probably take a backseat though


EDIT 2:
In Python, EVERYTHING is an object, even functions. At runtime, all the function objects are created and stored in memory, and that means their default values are created and stored in memory too. Each time you make a function call, you're actually reusing that same function object. Thus, each time you use a default parameter, you're using the same one initially created at runtime. I personally think it's a bad implementation, and probably in my top 3 biggest gripes about Python.

EDIT 3:
Nallo, I looked at your profile and saw that it was wiped clean. I really hope you're joking about quitting. Earlier today I was gushing about how much respect I have for you and how brilliant I think you are here. And believe me, there are very few people I give such strong recommendations for.

This post has been edited by atraub: 06 August 2012 - 08:18 PM

[jon.kiparsky]

atraub, on 06 August 2012 - 10:05 PM, said:

I actually once emailed Guido Van Rossum about this behavior. He was very familiar with it and said that it wouldn't be fixed due to backward compatibility reasons. I wasn't very satisfied with his answer, but what are ya gonna do?

So, not until Python 4, at least, is what I'm hearing.

Quote

ALSO:
I love the idea of a "spot the bug" trend in posts. I think this could be a pretty fun way to get more community involvement. I would say that purplse, blues, teals, reds, etc... should probably take a backseat though

Oh, I want to play too. I promise I know nothing!

Quote

EDIT 2:
In Python, EVERYTHING is an object, even functions. At runtime, all the function objects are created and stored in memory, and that means their default values are created and stored in memory too. Each time you make a function call, you're actually reusing that same function object. Thus, each time you use a default parameter, you're using the same one initially created at runtime. I personally think it's a bad implementation, and probably in my top 3 biggest gripes about Python.

It starts to make sense now. Not in the sense that "it starts to seem like a good idea" but in the sense that "that's why it's done that way".

Quote

EDIT 3:
Nallo, I looked at your profile and saw that it was wiped clean. I really hope you're joking about quitting. Earlier today I was gushing about how much respect I have for you and how brilliant I think you are here. And believe me, there are very few people I give such strong recommendations for.

I can't say that I've run into Nallo's posts before, but anyone who could post such an interesting puzzle is someone we should all hate to lost as a member of the community - this quite apart from the fact that (s)he evidently left feeling quite put upon, which is quite regrettable, if also inexplicable.

This post has been edited by jon.kiparsky: 07 August 2012 - 08:16 AM