5 Replies - 1818 Views - Last Post: 13 April 2011 - 11:21 AM Rate Topic: -----

#1 theholygod  Icon User is offline

  • D.I.C Regular

Reputation: 7
  • View blog
  • Posts: 342
  • Joined: 05-February 06

Store as 1 byte integer?

Posted 12 April 2011 - 07:38 PM

Hi

I'm dealing with a list containing a LOT of data. This is eating a lot of memory that is just being wasted.

Each value in my list is an integer between 0 and 110. Is there any way each object in my list can be a single byte?

Thanks
Is This A Good Question/Topic? 1
  • +

Replies To: Store as 1 byte integer?

#2 nandureddy  Icon User is offline

  • D.I.C Head

Reputation: 28
  • View blog
  • Posts: 129
  • Joined: 31-January 11

Re: Store as 1 byte integer?

Posted 12 April 2011 - 08:00 PM

Use the struct.pack module.
Refer to http://docs.python.o...ary/struct.html
Was This Post Helpful? 0
  • +
  • -

#3 theholygod  Icon User is offline

  • D.I.C Regular

Reputation: 7
  • View blog
  • Posts: 342
  • Joined: 05-February 06

Re: Store as 1 byte integer?

Posted 12 April 2011 - 08:13 PM

Have I completely misunderstood something here?

>>> print sys.getsizeof(100)
12
>>> print sys.getsizeof(struct.pack("B", 100))
25

Was This Post Helpful? 0
  • +
  • -

#4 theholygod  Icon User is offline

  • D.I.C Regular

Reputation: 7
  • View blog
  • Posts: 342
  • Joined: 05-February 06

Re: Store as 1 byte integer?

Posted 13 April 2011 - 12:13 AM

Am now using array.arrays instead of lists. Seems to have helped considerably.
Was This Post Helpful? 1
  • +
  • -

#5 baavgai  Icon User is offline

  • Dreaming Coder
  • member icon

Reputation: 5846
  • View blog
  • Posts: 12,705
  • Joined: 16-October 07

Re: Store as 1 byte integer?

Posted 13 April 2011 - 05:37 AM

When I first read this I thought, but python doesn't have arrays! Of course, it does have a butt load of packages, so I'm guessing this one.

I was curious how a string might do. This was my first test:
>>> a = [ random.randint(65,100) for i in range(50) ]
>>> sys.getsizeof(a)
536
>>> sys.getsizeof([ struct.pack("B", v) for v in a ])
536
>>> sys.getsizeof(''.join([chr(i) for i in a]))
90
>>> 



Encouraged, I wrote a test suite. Both to validate that I could get back to the int list from the string and also test the array.
The initial run:
>>> def testSize(elementCount):
...     test = [ ('Just ints', [ random.randint(65,100) for i in range(elementCount) ]) ]
...     test.append( ('Packed ints',[ struct.pack("B", v) for v in test[0][1] ]) )
...     test.append( ('String', ''.join([chr(i) for i in test[0][1]]) ) )
...     test.append( ('Str revert', [ ord(c) for c in test[2][1] ] ) )
...     test.append( ('Array', array.array('B', test[0][1]) ) )
...     test.append( ('Array Revert', [ b for b in test[4][1] ] ) )
...     print "testSize", elementCount
...     for item in test:
...             print sys.getsizeof(item[1]), item[0], (test[0][1]==item[1])
...     print
... 
>>> 
>>> for size in (10,100,200): testSize(size)
... 
testSize 10
200 Just ints True
200 Packed ints False
50 String False
200 Str revert True
56 Array False
200 Array Revert True

testSize 100
920 Just ints True
920 Packed ints False
140 String False
920 Str revert True
56 Array False
920 Array Revert True

testSize 200
1680 Just ints True
1680 Packed ints False
240 String False
1680 Str revert True
56 Array False
1680 Array Revert True

>>> 



At first I was impressed with the array, but then, you look at them... clearly sys.getsizeof isn't being honest here. Looking at the spec, "buffer_info" seems to solve the mystery.


>>> def testSize2(counts):
...     print "                                                Real"
...     print "Items   Int Size    Str Size  Array Size  Array Size"
...     for elementCount in counts:
...             intList = [ random.randint(65,100) for i in range(elementCount) ]
...             stringStore = ''.join([chr(i) for i in intList])
...             arrayStore = array.array('B', intList)
...             data = (elementCount, 
...                     sys.getsizeof(intList), 
...                     sys.getsizeof(stringStore),
...                     sys.getsizeof(arrayStore), 
...                     arrayStore.buffer_info()[1] )
...             print "%4d   %9d   %9d   %9d   %9d" % data
...     print
... 
>>> 
>>> testSize2((10,100,200,500,1000))
                                                Real
Items   Int Size    Str Size  Array Size  Array Size
  10         200          50          56          10
 100         920         140          56         100
 200        1680         240          56         200
 500        4280         540          56         500
1000        9032        1040          56        1000

>>> 



Looks like a standard python string has 40 bytes overhead, which is interesting. But it's only 40 bytes and I might consider it because it's a fundamental type in the language.
Was This Post Helpful? 3
  • +
  • -

#6 theholygod  Icon User is offline

  • D.I.C Regular

Reputation: 7
  • View blog
  • Posts: 342
  • Joined: 05-February 06

Re: Store as 1 byte integer?

Posted 13 April 2011 - 11:21 AM

That's really interesting. Thanks for going to so much trouble!

I'm going to stick with array.arrays for now since they're entirely compatible with all the functions I have written so far that use lists.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1