[Nekroze]

Ok so yesterday I thought it would be cool to make a python module that allows developers to contain all sensitive project data (logic scripts, models, textures etc.) in special containers similar to bethesda's .BSA files.

These containers will use bz2 compression on all files and if the developer wants to and have pycrypto installed they may enable encryption and set a key which will also encrypt each file in these containers.

So far things have gone rather well considering as with most things i do, i have never done something like this. However I am now testing its ability to use the files loaded from an archive into pygame, specifically an image, and then display that image.

however when i pass my loaded file, a cStringIO file, to pygame.image.load it errors as unsupported image format. Ok so i try to pipe it the data directly by giving it the cStringIO.getvalue() and it errors again at pygame.image.load saying it must be a string without null bytes, no str.

I am kind of lost as to where to go from here to fix it.

The module is simple and i have attached it and and the test.py which replicates this problem using the module. I would like to get this fixed and then polish it up some more freely distribute it but like i said, only started this yesterday so its got a bit more work needed i think. please excuse the state of some of it as i have been spent the last few hours hacking it to peices to fix the little bugs but this one i cannot kill.

Thanks guys for any help you can offer!

Attached File(s)

•  darc.zip (12.9K)
  Number of downloads: 32

[Motoma]

There are a couple of problems you are encountering.

First, you are passing binary data (your image content) to pygame.image.load(), which is expecting a file name. If you have the image data in memory, you will want to use pygame.image.fromstring() or pygame.image.frombuffer() to load the data.

Unfortunately, both of these functions require you to have used pygame to create the image file to begin with; you would use pygame.image.load() to open your desired image file, then call pygame.image.tostring() to turn it into data which then could be loaded via pygame.image.fromstring() or pygame.image.frombuffer().

In your case, you would use pygame.image.tostring() to get the string which you are then encrypting/compressing. In this process, you will need to keep track of the metadata of the file--the file size and the format--and store this in your archive as well.

An alternate option is to write the decrypted/extracted image to a temporary directory, and then use pygame.image.load() to open that. This would be the easier solution to your problem, but I don't know how it fits in with your plans.

[Nekroze]

Ok, a bit to go through there.

First as detailed here http://www.pygame.or...game.image.load

pygame.image.load can take a fileobject so i thought if i give it something similar it may work or i may be able to make it interface somehow but i just don't know how.

as far as having to create the image with pygame THEN store it for later retrieval... darc is not intended to work only for pygame it is supposed to work to just load any file to be used, so later videos, sounds even text for scripts may be stored. so using pygame to rebuild the items before storage is kind of out of the question.

as for the last part, i always thought that this would solve my problems however the idea here is to, if the developer wishes it, provide security for their projects assets as well as a convenient container. by extracting the final un-encrypted copy of a file to a hard location and THEN passing it in it defeats the purpose of almost everything darc is for.

I am not sure how to address these issues, the most method that i believe most likely to be possible and still maintain darc's goals would be the first and original idea to pass .load a fileobject or something that can interface with it like one and be loaded like the pygame docs say it can. I just dont know how. it is a shame the image.from* methods require the image to be pre-formated for pygame.

EDIT: The only reason the test uses pygame is because i am much more familiar with it then other python graphics systems and it is very commonly used by myself and many others so compatibility must include pygame but not be exclusive to it.

This post has been edited by Nekroze: 21 February 2012 - 02:04 PM

[Motoma]

Hey, thanks for a more thorough explanation of what you are aiming for. I think that this is definitely doable, the key resides in your ability to emulate the file object. Depending on the interface you are using this is either nearly impossible (need to implement fileno()) or simple (need merely to implement read()).

As a place to start, you could create a new file object and run dir() on it. You will wantread()to replicate as many of the public functions as possible in order to create a file-like object. In the simplest case, you would want read(), read(size), write(bytes), and seek(offset).

If you use this in conjunction with the namehint keyword argument, you should be able to load this into Pygame with pygame.image.load(). The key will be how closely you can emulate the file object.

Here is a quick example of what I mean, this may work for you in PyGame, but it will certainly need more to work in general:

class DummyFile:
    def __init__(self, data):
        self.data = data
        self.pos = 0

    def read(self, size=None):
        start = self.pos
        end = len(self.data) - 1

        if size is not None:
            end = min(len(self.data), self.pos + size)

        self.pos = end
        return self.data[start:end]

    def seek(self, offset):
        self.pos = offset
    
    def write(self):
        pass

You could add these functions to your encapsulating class, or wrap the returned data in this before passing it on. An ideal implementation would allow you to open a wrapped file and seamlessly use it as if it were a real file.

[Nekroze]

ok so what im getting is this is failing because the cStringIO object i am giving it is lacking the important file-like functions such as read and write.

I can see how using this dummy file class can work, but what data must it have, would i give it the data from the cStringIO on creation like this:

pygame.image.load(DummyFile(cStringFile.getvalue()))

or would i need to alter the data first. Also i am not at all clear on the use of the namehint thing. I understand that it should contain the files original name (doesn't need the path though?) but what is it for?

Apart from the issues with darc, what do you think of the code at the moment, it needs some cleaning after i finish this bug hunting but i have tried to make it as well documented and useable as possible so i can publish this, maybe get it on pypi or something eventually.

[Motoma]

Your use of the DummyFile was exactly as I intended it.

The namehint is so that PyGame can determine the file format (GIF vs JPEG vs PNG) trivially.

I haven't taken a deep look at the code, but I would warn against putting cryptographic code in any authoritative repository without having a crypto expert look it over. It is deceptively hard to get right, even for the professionals. The recent RSA debacle is one such example of just how difficult good crypto is.

There are plenty of well meaning coders out there that want to make cryptography available to the masses, but these are littered with cases of flawed implementations that provide little to no protection; if relied on in cases of repressive governments (pick any of the handful littering the news), lives could be at stake.

[Nekroze]

True, although intended mostly for games and other media heavy projects, i don't know why but it could be used for data hiding i guess.

My cryptographic code is from pycrypto tutorials of which there are few, so i do have to take it at face value. However i do not know anyone who is skilled in cryptography so i thought to get it to the best state i can and then open it to the public to suggest alterations and such.

I have been a bit touchy on the idea of using the same password for every single file in each archive, i just cannot think of any more secure way at the moment other then adding the file name to the key for each file encryption.

Thanks for the help and insight i will try to work that DummyFile system out.

NOTE: preliminary use of the DummyFile has yealded another error on the image.load:

pygame.error: Can't seek in this data source

but there is a seek method there so why is this, i am responding before searching but i have yet to find much help from google for this whole project so far.

This post has been edited by Nekroze: 21 February 2012 - 02:47 PM

[Motoma]

This could be because my definition of seek is not the same implementation as a file objects, which has an optional whence argument. Take a look at the functions provided by the Built-in File Object.

[Nekroze]

ah so its just that it needs better emulation of the fileObjects methods. Perfect, will get on it mate.

Thanks a heap for the help again.

EDIT: the test worked after changing the seek method to make full use of whence (i think) with the following:

def seek(self, offset, whence=0):
    if whence == 0:
        self.pos = offset
    elif whence == 1:
        self.pos = self.pos + offset
    elif whence == 2:
        self.pos = len(data) + offset

This post has been edited by Nekroze: 21 February 2012 - 03:07 PM