4 Replies - 636 Views - Last Post: 05 August 2012 - 03:01 PM Rate Topic: -----

#1 darek9576  Icon User is online

  • D.I.C Lover

Reputation: 198
  • View blog
  • Posts: 1,689
  • Joined: 13-March 10

List comprehension

Posted 05 August 2012 - 01:21 PM

Lets say we have the piece of code:

lst = ["asd","sdf","asdaaaa"]
lst = [x.capitalize() for x in lst]

Now, my question is regarding the inner working of list comprehensions. Does python create a new list and put i back to lst when the second statement is executed or it is being smart and efficient and is working on the old one.

Thanks
Is This A Good Question/Topic? 0
  • +

Replies To: List comprehension

#2 sepp2k  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 2117
  • View blog
  • Posts: 3,242
  • Joined: 21-June 11

Re: List comprehension

Posted 05 August 2012 - 01:33 PM

It creates a new one.

PS:

My first instinct was to say that this optimization would be quite hard to do correctly for cases that aren't as trivial as your example because it would be hard for Python to proof that the reference that's currently being reassigned is the only reference to the list you're iterating over. But on second thought that's not actually true. Since (C-)Python is reference counted, that could be checked pretty easily.

That said it'd still be a pretty specific optimization and would probably not even safe that much time. So it probably wouldn't be worth it.

Either way Python doesn't do it.
Was This Post Helpful? 0
  • +
  • -

#3 Lemur  Icon User is offline

  • Pragmatism over Dogma
  • member icon


Reputation: 1368
  • View blog
  • Posts: 3,455
  • Joined: 28-November 09

Re: List comprehension

Posted 05 August 2012 - 02:14 PM

The reason Python tends to do that is that it follows a slight functional lean. The reasoning behind it is that data should stay pure. In this case you would have potentially clobbered the original data and it could make some nasty messes if a wrench finds its way in.

I'm always inclined to keep things on the functional side when it comes to mapping an array like this, otherwise you're bound to end up in debug limbo.
Was This Post Helpful? 1
  • +
  • -

#4 sepp2k  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 2117
  • View blog
  • Posts: 3,242
  • Joined: 21-June 11

Re: List comprehension

Posted 05 August 2012 - 02:38 PM

View PostLemur, on 05 August 2012 - 11:14 PM, said:

The reason Python tends to do that is that it follows a slight functional lean. The reasoning behind it is that data should stay pure. In this case you would have potentially clobbered the original data and it could make some nasty messes if a wrench finds its way in.


What? No. The whole point of the proposed optimization was that it wouldn't change the semantics of the code at all because the old value of lst is no longer accessible and can thus be safely overridden.

If Python did do this optimization, there'd be no downside in regards to safety or debugability.

Note that some languages that are way more functional in nature than Python, do perform this kind of optimization a lot. If you want to write functional code and still have excellent performance, you actually need this kind of optimization. For example the Data.Vector datatype in Haskell heavily relies on stream fusion (which is basically a more general of what we're talking about here) for its performance.

This post has been edited by sepp2k: 03 December 2012 - 02:24 PM

Was This Post Helpful? 0
  • +
  • -

#5 sepp2k  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 2117
  • View blog
  • Posts: 3,242
  • Joined: 21-June 11

Re: List comprehension

Posted 05 August 2012 - 03:01 PM

Okay, the above might have come out a bit less coherent than I intended, so let me try to restate in a more organized fashion:

In functional languages you rely heavily on immutable data structures. However some things are simply impossible to implement efficiently if every modification of a list or array requires you to copy the whole thing.

In Python that's not a problem because in the cases where it is a problem, you can simply rewrite the code to use mutation. Python lists are perfectly mutable after all. Of course that's not at all necessary in the OP's example, but in general there are cases where you'd want to use mutation in Python and in those cases doing so would be perfectly Pythonic.

However the situation is different in languages that have more than just a slight functional lean (as you put it quite nicely ;-) ). In those languages mutation is more heavily discouraged (or in some cases simply not allowed). Also many statically typed functional languages actually have a heavier focus on performance than Python does. So those languages need a way to take code written in a functional style (i.e. without mutation) and still make it run with an acceptable speed (preferably as fast as the equivalent imperative code would run).

That's where optimization techniques like the one the OP asked about come into play. If we want functional code to run as fast as imperative code, we need optimizations that take code that would normally create a lot of copies and turn it into code that only creates few copies (or none at all).

So saying that that kind of optimization runs counter to the functional style is not accurate. The opposite is the case: the more functional a language is, the more it needs that kind of optimization.

Of course as far as Python is concerned the situation still is simply as I said: Python doesn't do this kind of optimization - nor does it really need to.

This post has been edited by sepp2k: 03 December 2012 - 02:26 PM

Was This Post Helpful? 1
  • +
  • -

Page 1 of 1