[brianmen]

So I have this code in python that writes some values to a Dictionary where each key is a student ID number and each value is a Class (of type student) where each Class has some variables associated with it.

#Assign var2
            try:
                    if ((str(i) in row_num_id.iterkeys()) and (row_num_id[str(i)]==varschosen[1])):
                            valuetowrite=str(row[i])
                            if students[str(variablekey)].var2 != []:
                                    students[str(variablekey)].var2.append(valuetowrite)
                            else:
                                    students[str(variablekey)].var2=([valuetowrite])
            except:
                    two=1#This is just a dummy assignment because I #can't leave it empty... I don't need my program to do anything if the "try" doesn't work. I just want to prevent a crash.

            #Assign var3
            try:
                    if ((str(i) in row_num_id.iterkeys()) and (row_num_id[str(i)]==varschosen[2])):
                            valuetowrite=str(row[i])
                            if students[str(variablekey)].var3 != []:
                                    students[str(variablekey)].var3.append(valuetowrite)
                            else:
                                    students[str(variablekey)].var3=([valuetowrite])
            except:
                    two=1

            #Assign var4
            try:
                    if ((str(i) in row_num_id.iterkeys()) and (row_num_id[str(i)]==varschosen[3])):
                            valuetowrite=str(row[i])
                            if students[str(variablekey)].var4 != []:
                                    students[str(variablekey)].var4.append(valuetowrite)
                            else:
                                    students[str(variablekey)].var4=([valuetowrite])
            except:
                    two=1

The same code repeats many, many times for each variable that the student has (var5, var6,....varX). However, the RAM spike in my program comes up as I execute the function that does this series of variable assignments.

I wish to find out a way to make this more efficient in speed or more memory efficient because running this part of my program takes up around half a gig of memory.

Thanks for your help!

[moopet]

Presumably this is all part of some outer loop?
I don't see why you're not doing it as a for loop, because each test/assignment is the same except for one incrementing value.
Btw, instead of your dummy two=1 just use pass

[brianmen]

moopet, on 02 August 2010 - 01:00 PM, said:

Presumably this is all part of some outer loop?
I don't see why you're not doing it as a for loop, because each test/assignment is the same except for one incrementing value.
Btw, instead of your dummy two=1 just use pass

Yes, this is all within another loop.
I'm not sure how to do a for loop when dealing with Class variable names and their assignments.

[Nallo]

To loop through variables you can make use of getattr() and setattr():

varname = "var1"
x = getattr(some_object, varname)
setattr(some_object, varname, some_value)

#does the same as:
x = some_object.var1
some_object.var1 = some_value

So you can use a list of variablenames (as strings) to loop through the variables:

varnames = ["var1", "var2", "var3"]
for varname in varnames:
    setattr(some_object, varname, some_value)

This post has been edited by Nallo: 02 August 2010 - 02:54 PM

[Simown]

As the "varchosen" indexes are consecutive you could use a range() loop instead of typing the values explicitly.

For example you have:

(row_num_id[str(i)]==varschosen[1])):

// Then later


(row_num_id[str(i)]==varschosen[2])):

//Up to supposedly

(row_num_id[str(i)]==varschosen[n])):

Instead of typing it out each time, use a loop to condense the code:

// n = the number of vars
    for j in range(1, n+1):
        if .... and(row_num_id[str(i)] == varschosen[j])):                               
        // rest of code here

Combined with what Nallo and moopet said, your code will be significantly condensed.

This post has been edited by Simown: 02 August 2010 - 04:17 PM

[brianmen]

Okay let me simplify my question: In my case, I have a dictionary of about 6000 instantiated classes, where each class has 1000 attributed variables all of type string or list of strings. I don't really care about the number of lines my code is or the speed at which it runs (Right now, my code is at almost 20,000 lines and is about a 1 MB .py file!). What I am concerned about is the amount of memory it is taking up because this is the culprit in throttling my CPU. The ultimate question is: does the number of code lines by which I build up this massive dictionary matter so much in terms of RAM usage?

My original code functions fine, but the RAM usage is high. I'm not sure if that is "normal" with the amount of data I am collecting. Does writing the code in a condensed fashion (as shown by the people who helped me below) actually make a noticeable difference in the amount of RAM I am going to eat up? Sure there are X ways to build a dictionary, but does it even affect the RAM usage in this case?

[baavgai]

I suppose you could write the code you first posted like this:

for n in range(1,4):
	attr = 'var' + str(n+1)
	iStr = str(i)
	try:
		if iStr in row_num_id.iterkeys():
			if row_num_id[iStr]==varschosen[n]:
				valuetowrite=str(row[i])
				student = students[str(variablekey)]
				if student.__dict__[attr] != []:
					student.__dict__[attr].append(valuetowrite)
				else:
					student.__dict__[attr] = [valuetowrite]
	except:
		None

Honestly, if your code is repeating and millions of lines, you're clearly doing something wrong.

[Nallo]

The way you are doing things Ram usage will be high. You reference 6000 class instances with 1000 variables containing some strings and lists. Those objects live somewhere in memory! You will be using at least 6000 * 1000 * average_variabledata_length of Ram. So with an average data length of about 50 you get to that halv GByte. So it is normal (and the code length doesnt matter for this question).

To cut down ram usage you could store those instances on your hard drive (probably in a database) and only load those you currently need.

[chemicalfan]

Storing them in RAM will be miles faster though. I don't think using half a gig of RAM is an issue, you're hard pushed to buy a computer with less than 1 gig now anyway, most laptops come with at least 2 gig. I wouldn't panic over RAM usage in all honesty.

If it's a performance issue, you might have to look towards multi-processing or implementing these parts as a C extension