6 Replies - 178 Views - Last Post: 04 July 2013 - 02:01 PM Rate Topic: -----

#1 Wnt2bsleepin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 24
  • Joined: 04-July 13

Detect garbled but valid images

Posted 04 July 2013 - 11:25 AM

I have a large amount of garbled images. I would like to write something that will go through them and delete any file that is garbled or corrupted. The problem is that the images seems to be valid, and PIL doesn't pick up on that.

The images look like this.

Posted Image

I've already written something using PIL that worked for some images.

def checkColor():
    mostCommonColor = ()
    logFile = open("log.txt","w")
    currentFolder = 100
    #List each directory in root
    for i,dir_file in enumerate(os.listdir(os.getcwd())):
         if "SORTED" not in dir_file[1]:
            fileNumber = dir_file.partition('r')[2]
             
            if fileNumber.isdigit():

               
                if os.path.isdir(dir_file) and (int(fileNumber) == int(currentFolder)):
                    #Follow it down one level
                    os.chdir(dir_file)
                    logFile.write("Current Directory: " + dir_file)
                    print("Current Dir: " + dir_file)

                    #Run through each file
                    totalFiles = len(os.listdir(os.getcwd()))
                    for i,file_dir in enumerate(os.listdir(os.getcwd())):

                        print("\r{:.2%}".format(i/totalFiles))
                        if os.path.isfile(file_dir) and isImage(file_dir):
                            image_file = open(file_dir,'rb')
                            img = Image.open(image_file)
                            colorTypes = img.getcolors(img.size[0] * img.size[1])
                            image_file.close()
                            highest = -1 
                            oldHigh = -1
                            index = 0
                            #Find the most common color
                            for i,stuff in enumerate(colorTypes):
                                highest = max(stuff[0],highest)

                                if highest != oldHigh:
                                    index = i

                                oldHigh = highest

                            mostCommonColor = colorTypes[index][1]
                            logFile.write("File: "+ file_dir + " Most Common Color: " + str(mostCommonColor) + "\n")
                            #print("File: "+ file_dir + " Most Common Color: " + str(mostCommonColor))
                            if mostCommonColor == (128,128,128):
                                if os.path.exists(damagedFilesDirectory) is False:
                                    os.mkdir(damagedFilesDirectory)
                                        

                                #Move the file to damaged folder
                                #print("Moving file " + file_dir)
                                logFile.write("Moving File " + file_dir + "\n") 
                                os.rename(file_dir,os.path.join(damagedFilesDirectory,file_dir))

                    logFile.flush()
                                    

                    os.chdir("..")

                    currentFolder+=1 



It worked well. Moving some of the files over, but it doesn't catch an images like the one above. Thank you for any help.

Is This A Good Question/Topic? 0
  • +

Replies To: Detect garbled but valid images

#2 tlhIn`toq  Icon User is offline

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5436
  • View blog
  • Posts: 11,658
  • Joined: 02-June 10

Re: Detect garbled but valid images

Posted 04 July 2013 - 12:55 PM

Its pretty obvious that a block of data is missing. Possibly a pattern of missing data blocks in a re-occurring pattern. Like a packet of data gets lost in transmission during upload. But there are decernable rows about 8-10 pixels tall that all line up... then the stride of the image goes off. and the next block about 10 pixels high is shoved to the right.

What would be even cooler, and maybe more useful would be to add padding in the data (dull gray pixels for example) so as to move the rest of the data to the right and eventually re-align.

If you were to add the right number of pixels, at the right point, this could be salvaged.

Otherwise an edge-detection filter should go crazy here and show hundreds of perfectly horizontal edges. I would think you could use a filter like that, and set a threshold. For example, if more than 2000 horizontal edges then it must be garbled.

You could also look for the absence of something. For example if all the photos should have people in them, do a face detection pass. If there is no face then it must be garbled.
Was This Post Helpful? 0
  • +
  • -

#3 Wnt2bsleepin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 24
  • Joined: 04-July 13

Re: Detect garbled but valid images

Posted 04 July 2013 - 01:15 PM

These were recovered from a partition that was completely wiped. I don't know if I can recover any image data.

Some of the images also look like this

Posted Image

These are the ones that are caught by detecting the most common occurring color, and comparing it to (128,128,128). Face detection wouldn't work as not all of them have faces, but it could help eliminate a large percentage of the photos. Are there are face detection libraries available?

I am not too familiar with image manipulation or how image data is stored. How would an edge detection filter work? From what I understand of your first paragraph, you're saying that the data is out of order. In fact, some of the images have the bottoms and tops reversed. So the bottom half of the picture shows up on the top. We are seeing the same thing here, except with horizontal slices. Correct me if I am wrong, I am just happy that I was able to recover this much.
Was This Post Helpful? 0
  • +
  • -

#4 tlhIn`toq  Icon User is offline

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5436
  • View blog
  • Posts: 11,658
  • Joined: 02-June 10

Re: Detect garbled but valid images

Posted 04 July 2013 - 01:29 PM

Quote

These were recovered from a partition that was completely wiped.


I'm going to have to drop out of this conversation. I'm not at all comfortable helping someone trying to restore images from a drive that was wiped. It sounds very malicious. If the drive was wiped (though not securely, DoD level) then you shouldn't be trying to recover these images.
Was This Post Helpful? 0
  • +
  • -

#5 Wnt2bsleepin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 24
  • Joined: 04-July 13

Re: Detect garbled but valid images

Posted 04 July 2013 - 01:46 PM

It was a reinstall of Windows that I did. I was told that the images were backed up beforehand, only to find out that they weren't. I should have backed them up anyway, but I was short on time and I am kicking myself for my lack of foresight. The majority of the images are fine, but I wanted to automate the detection of the ones that are corrupted.
Was This Post Helpful? 0
  • +
  • -

#6 tlhIn`toq  Icon User is offline

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5436
  • View blog
  • Posts: 11,658
  • Joined: 02-June 10

Re: Detect garbled but valid images

Posted 04 July 2013 - 01:54 PM

We all learn our lessons about backing up. It certainly shouldn't happen only if you are installing an OS. What if the drive had simply died one day?

The simple fact is that we aren't there to know the truth. For all we know you were issued a new laptop at work and want to recover someone else's files. Or bought it on ebay and what their images and data. We simply don't know.

And the recovery of data from formatted drives is a sensitive matter. And you're a newbie here. All of that together makes this a project that I'm not going to be apart of any further.
Was This Post Helpful? 0
  • +
  • -

#7 Wnt2bsleepin  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 24
  • Joined: 04-July 13

Re: Detect garbled but valid images

Posted 04 July 2013 - 02:01 PM

That's fine. I understand and support your reasoning. Thanks for your help.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1