8 Replies - 1092 Views - Last Post: 03 January 2018 - 09:30 AM

#1 bobsmith76  Icon User is offline

  • D.I.C Head

Reputation: 9
  • View blog
  • Posts: 159
  • Joined: 14-February 17

I don't understand memory

Posted 02 January 2018 - 02:43 PM

Here's a quote from Python Central:
"""
Generators are especially useful for memory-intensive tasks, where there is no need to keep all of the elements of a memory-heavy list accessible at the same time. Calculating a series of values one-by-one can also be useful in situations where the complete result is never needed, yielding intermediate results to the caller until some requirement is satisfied and further processing stops.
"""

I don't understand what are the drawbacks of memory intensive tasks. Does it slow things down? It seems like my computer has more than enough memory so what are the advantages of using less memory?

I'm guessing that you keep memory low by having as few global and local variables as possible and keeping what is in the variables as short as possible. Am I right?

Is This A Good Question/Topic? 1
  • +

Replies To: I don't understand memory

#2 modi123_1  Icon User is offline

  • Suitor #2
  • member icon



Reputation: 13954
  • View blog
  • Posts: 55,700
  • Joined: 12-June 08

Re: I don't understand memory

Posted 02 January 2018 - 02:53 PM

The advantages of a small foot print is typically snappier returns, room to grow, not crashing a system, being a better fit for different systems with different memory, etc.
Was This Post Helpful? 3
  • +
  • -

#3 NeoTifa  Icon User is offline

  • NeoTifa Codebreaker, the Scourge of Devtester
  • member icon





Reputation: 4186
  • View blog
  • Posts: 18,388
  • Joined: 24-September 08

Re: I don't understand memory

Posted 02 January 2018 - 03:11 PM

*
POPULAR

Large datasets will slow down your system massively, especially when sorting and searching. Finding efficient algorithms is key when handling such. I did an assignment with big data sets and searching for a worst case scenario (last index) on a set of about 5 million entries on my system took about 20 mins. Given, my machine is old-ish. Although a few lines of code extra wouldn't hurt in the long run on a big system, but if you're working on smaller systems (like a microprocessor), you need all the space you can get. Even handling large data with bigger machines, extra overhead is extra overhead.

This post has been edited by NeoTifa: 02 January 2018 - 03:14 PM

Was This Post Helpful? 5
  • +
  • -

#4 Radius Nightly  Icon User is offline

  • D.I.C Regular

Reputation: 30
  • View blog
  • Posts: 276
  • Joined: 07-May 15

Re: I don't understand memory

Posted 02 January 2018 - 03:46 PM

I call it "konzerva memory", because "konzerva" at my language means "can" (cylindrical metal container, like a garbage one) and its pronounced similarly to "conserve", like "conserve memory" = "konzerva memory". :nuke:
Was This Post Helpful? 0
  • +
  • -

#5 Skydiver  Icon User is offline

  • Code herder
  • member icon

Reputation: 6112
  • View blog
  • Posts: 21,040
  • Joined: 05-May 12

Re: I don't understand memory

Posted 02 January 2018 - 05:48 PM

View Postbobsmith76, on 02 January 2018 - 04:43 PM, said:

It seems like my computer has more than enough memory so what are the advantages of using less memory?

Two counter questions:
How much of that memory is dedicated for the use of your program?
What happens if every programmer took on the same attitude of just using memory willy-nilly?
Was This Post Helpful? 0
  • +
  • -

#6 jon.kiparsky  Icon User is offline

  • Beginner
  • member icon


Reputation: 11022
  • View blog
  • Posts: 18,804
  • Joined: 19-March 11

Re: I don't understand memory

Posted 02 January 2018 - 07:25 PM

The simplest argument for seeking efficiency of memory usage is this: the more efficiently you use memory, the bigger the problems you can solve on a given machine. In general, we'd like to solve bigger problems on smaller machines, because in general we'd like to be able to solve bigger problems.

That being said, it's important to remember Knuth's Admonition. Start out by writing something that works, and don't worry too much about optimizing until you have some reason to think that your code has come out pessimized.
Was This Post Helpful? 2
  • +
  • -

#7 jon.kiparsky  Icon User is offline

  • Beginner
  • member icon


Reputation: 11022
  • View blog
  • Posts: 18,804
  • Joined: 19-March 11

Re: I don't understand memory

Posted 02 January 2018 - 07:44 PM

View Postbobsmith76, on 02 January 2018 - 04:43 PM, said:

I don't understand what are the drawbacks of memory intensive tasks.


It's not so much that memory-intensive tasks are bad, it's more that when find yourself in memory trouble, there are some good tools for dealing with them, and generators are one of those tools in python. If you're writing something which uses a lot of memory in proportion to the size of the problem, and you know that you're only ever going to solve small problems, then you shouldn't worry about memory. If you're writing a general-purpose tool that might run into large problems then you might want to see how big a problem you want to solve.

Quote

I'm guessing that you keep memory low by having as few global and local variables as possible and keeping what is in the variables as short as possible. Am I right?


Not quite. The number of variables you declare is almost never going to be a problem simply because you can't declare all that many variables. I mean, you could declare ints and strings until next Tuesday and you'd not come up with enough to cause a hiccup on a modern machine. What can be a problem is the unconstrained growth of your data structures. For example, if you're sieving primes, you'll eventually find that you have a large list of primes to deal with*. This list will be only one variable, but since it grows without bound, it is guaranteed to eventually swamp your memory**.
This is why generators are handy: they allow you to avoid keeping a list around if all you need is the next item at any given time.

* just to be clear, "eventually" can happen in well under a second - I mean "eventually" as in "if you go up high enough", not as in "it might take a while.
** conveniently, there are useful results in number theory which can tell you pretty closely how many primes you will find less than some number N, so at least in this case you can predict where you'll run into trouble - this is not always the case!
Was This Post Helpful? 2
  • +
  • -

#8 ndc85430  Icon User is online

  • I think you'll find it's "Dr"
  • member icon

Reputation: 781
  • View blog
  • Posts: 3,191
  • Joined: 13-June 14

Re: I don't understand memory

Posted 02 January 2018 - 10:17 PM

Of course, these issues are very relevant today as we live in a world where we're generating a lot of data that we need to process. See, for example, this article about the need for technology in the environment. The second paragraph under "Earth applications" briefly mentions a somewhat relevant example to this thread.
Was This Post Helpful? 2
  • +
  • -

#9 Skydiver  Icon User is offline

  • Code herder
  • member icon

Reputation: 6112
  • View blog
  • Posts: 21,040
  • Joined: 05-May 12

Re: I don't understand memory

Posted 03 January 2018 - 09:30 AM

Or if that enviromntal impact is too distant from something that you are doing now as you write your code, consider this: let's say you wrote code that is a memory hog. To accommodate this, the OS keeps giving you virtual memory for your use at the cost of needing to swap your virtual memory space in and out of disk. That requires spinning up the disk motor, or activation more circuits for a SSD. That requires more energy. More energy usage means needing to cool the components which requires more energy to power the cooling system.
Was This Post Helpful? 2
  • +
  • -

Page 1 of 1