8 Replies - 745 Views - Last Post: 03 August 2010 - 05:13 AM Rate Topic: -----

#1 brianmen  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 39
  • Joined: 14-June 09

Can this code be written more efficiently?

Posted 02 August 2010 - 12:32 PM

So I have this code in python that writes some values to a Dictionary where each key is a student ID number and each value is a Class (of type student) where each Class has some variables associated with it.
#Assign var2
            try:
                    if ((str(i) in row_num_id.iterkeys()) and (row_num_id[str(i)]==varschosen[1])):
                            valuetowrite=str(row[i])
                            if students[str(variablekey)].var2 != []:
                                    students[str(variablekey)].var2.append(valuetowrite)
                            else:
                                    students[str(variablekey)].var2=([valuetowrite])
            except:
                    two=1#This is just a dummy assignment because I #can't leave it empty... I don't need my program to do anything if the "try" doesn't work. I just want to prevent a crash.

            #Assign var3
            try:
                    if ((str(i) in row_num_id.iterkeys()) and (row_num_id[str(i)]==varschosen[2])):
                            valuetowrite=str(row[i])
                            if students[str(variablekey)].var3 != []:
                                    students[str(variablekey)].var3.append(valuetowrite)
                            else:
                                    students[str(variablekey)].var3=([valuetowrite])
            except:
                    two=1

            #Assign var4
            try:
                    if ((str(i) in row_num_id.iterkeys()) and (row_num_id[str(i)]==varschosen[3])):
                            valuetowrite=str(row[i])
                            if students[str(variablekey)].var4 != []:
                                    students[str(variablekey)].var4.append(valuetowrite)
                            else:
                                    students[str(variablekey)].var4=([valuetowrite])
            except:
                    two=1



The same code repeats many, many times for each variable that the student has (var5, var6,....varX). However, the RAM spike in my program comes up as I execute the function that does this series of variable assignments.

I wish to find out a way to make this more efficient in speed or more memory efficient because running this part of my program takes up around half a gig of memory. :(

Thanks for your help!

Is This A Good Question/Topic? 0
  • +

Replies To: Can this code be written more efficiently?

#2 moopet  Icon User is offline

  • binary decision maker
  • member icon

Reputation: 339
  • View blog
  • Posts: 1,185
  • Joined: 02-April 09

Re: Can this code be written more efficiently?

Posted 02 August 2010 - 02:00 PM

Presumably this is all part of some outer loop?
I don't see why you're not doing it as a for loop, because each test/assignment is the same except for one incrementing value.
Btw, instead of your dummy two=1 just use pass
Was This Post Helpful? 0
  • +
  • -

#3 brianmen  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 39
  • Joined: 14-June 09

Re: Can this code be written more efficiently?

Posted 02 August 2010 - 02:09 PM

View Postmoopet, on 02 August 2010 - 01:00 PM, said:

Presumably this is all part of some outer loop?
I don't see why you're not doing it as a for loop, because each test/assignment is the same except for one incrementing value.
Btw, instead of your dummy two=1 just use pass

Yes, this is all within another loop.
I'm not sure how to do a for loop when dealing with Class variable names and their assignments.
Was This Post Helpful? 0
  • +
  • -

#4 Nallo  Icon User is offline

  • D.I.C Regular
  • member icon

Reputation: 163
  • View blog
  • Posts: 255
  • Joined: 19-July 09

Re: Can this code be written more efficiently?

Posted 02 August 2010 - 02:51 PM

To loop through variables you can make use of getattr() and setattr():

varname = "var1"
x = getattr(some_object, varname)
setattr(some_object, varname, some_value)

#does the same as:
x = some_object.var1
some_object.var1 = some_value



So you can use a list of variablenames (as strings) to loop through the variables:
varnames = ["var1", "var2", "var3"]
for varname in varnames:
    setattr(some_object, varname, some_value)


This post has been edited by Nallo: 02 August 2010 - 02:54 PM

Was This Post Helpful? 0
  • +
  • -

#5 Simown  Icon User is offline

  • Blue Sprat
  • member icon

Reputation: 319
  • View blog
  • Posts: 650
  • Joined: 20-May 10

Re: Can this code be written more efficiently?

Posted 02 August 2010 - 04:06 PM

As the "varchosen" indexes are consecutive you could use a range() loop instead of typing the values explicitly.

For example you have:


(row_num_id[str(i)]==varschosen[1])):

// Then later


(row_num_id[str(i)]==varschosen[2])):

//Up to supposedly

(row_num_id[str(i)]==varschosen[n])):




Instead of typing it out each time, use a loop to condense the code:

// n = the number of vars
    for j in range(1, n+1):
        if .... and(row_num_id[str(i)] == varschosen[j])):                               
        // rest of code here



Combined with what Nallo and moopet said, your code will be significantly condensed.

This post has been edited by Simown: 02 August 2010 - 04:17 PM

Was This Post Helpful? 0
  • +
  • -

#6 brianmen  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 39
  • Joined: 14-June 09

Re: Can this code be written more efficiently?

Posted 02 August 2010 - 04:20 PM

Okay let me simplify my question: In my case, I have a dictionary of about 6000 instantiated classes, where each class has 1000 attributed variables all of type string or list of strings. I don't really care about the number of lines my code is or the speed at which it runs (Right now, my code is at almost 20,000 lines and is about a 1 MB .py file!). What I am concerned about is the amount of memory it is taking up because this is the culprit in throttling my CPU. The ultimate question is: does the number of code lines by which I build up this massive dictionary matter so much in terms of RAM usage?

My original code functions fine, but the RAM usage is high. I'm not sure if that is "normal" with the amount of data I am collecting. Does writing the code in a condensed fashion (as shown by the people who helped me below) actually make a noticeable difference in the amount of RAM I am going to eat up? Sure there are X ways to build a dictionary, but does it even affect the RAM usage in this case?
Was This Post Helpful? 0
  • +
  • -

#7 baavgai  Icon User is offline

  • Dreaming Coder
  • member icon

Reputation: 5796
  • View blog
  • Posts: 12,631
  • Joined: 16-October 07

Re: Can this code be written more efficiently?

Posted 02 August 2010 - 05:05 PM

I suppose you could write the code you first posted like this:
for n in range(1,4):
	attr = 'var' + str(n+1)
	iStr = str(i)
	try:
		if iStr in row_num_id.iterkeys():
			if row_num_id[iStr]==varschosen[n]:
				valuetowrite=str(row[i])
				student = students[str(variablekey)]
				if student.__dict__[attr] != []:
					student.__dict__[attr].append(valuetowrite)
				else:
					student.__dict__[attr] = [valuetowrite]
	except:
		None



Honestly, if your code is repeating and millions of lines, you're clearly doing something wrong.
Was This Post Helpful? 1
  • +
  • -

#8 Nallo  Icon User is offline

  • D.I.C Regular
  • member icon

Reputation: 163
  • View blog
  • Posts: 255
  • Joined: 19-July 09

Re: Can this code be written more efficiently?

Posted 03 August 2010 - 04:48 AM

The way you are doing things Ram usage will be high. You reference 6000 class instances with 1000 variables containing some strings and lists. Those objects live somewhere in memory! You will be using at least 6000 * 1000 * average_variabledata_length of Ram. So with an average data length of about 50 you get to that halv GByte. So it is normal (and the code length doesnt matter for this question).

To cut down ram usage you could store those instances on your hard drive (probably in a database) and only load those you currently need.
Was This Post Helpful? 1
  • +
  • -

#9 chemicalfan  Icon User is offline

  • D.I.C Head

Reputation: 4
  • View blog
  • Posts: 88
  • Joined: 16-October 09

Re: Can this code be written more efficiently?

Posted 03 August 2010 - 05:13 AM

Storing them in RAM will be miles faster though. I don't think using half a gig of RAM is an issue, you're hard pushed to buy a computer with less than 1 gig now anyway, most laptops come with at least 2 gig. I wouldn't panic over RAM usage in all honesty.

If it's a performance issue, you might have to look towards multi-processing or implementing these parts as a C extension
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1