File Archiver Application

  • (2 Pages)
  • +
  • 1
  • 2

23 Replies - 683 Views - Last Post: 12 June 2013 - 03:00 PM Rate Topic: -----

#16 tlhIn`toq  Icon User is offline

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5676
  • View blog
  • Posts: 12,194
  • Joined: 02-June 10

Re: File Archiver Application

Posted 12 June 2013 - 01:16 PM

Morgan, I think you're getting caught up in the single-minded pursuit of one aspect of one feature of your overall application.

What is the total purpose of databasing this? What is the end goal of the application? Do you want to be able to do a search for a file by description? Do you want to put this out as a web service to track the most current version and all the versions in the evolution?

Don't get so hung up on trying to use the DB as a compressor that you loose sight of all the other needs and features. For example, how is this scheme going to handle an update in one of the records? What is the performance impact of saving 3 bytes?

Have you even compared a dozen files at byte level to see if there are enough identical byte-patterns to make it worth while? Can you find enough identical data in a dozen installers to make all this a reasonable course of action? As modi asked, at what level does it become profitable? Do you have to find groups of 4 bytes, 8 bytes, 12 bytes? In other words, did you just jump into the code before you did some research to prove the need and viability?
Was This Post Helpful? 0
  • +
  • -

#17 tlhIn`toq  Icon User is offline

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5676
  • View blog
  • Posts: 12,194
  • Joined: 02-June 10

Re: File Archiver Application

Posted 12 June 2013 - 01:26 PM

View Postmadmorgan, on 12 June 2013 - 01:04 PM, said:

File A = [112][12][1][2][32][56][245][90]
File B = [1] [2] [90] [66] [80] [91] [78] [99]

the database will have all of File A bits with in it I then add File B and in stead of adding the following bytes [1] [2] it would refance the File A indexs.


Let's think about the logic you want to impliment. YOu have two bytes that are the same in these files. How do you reference it in the listing of file b?

You have to reference the source record in some way
You have to reference the starting index
You have to reference the length of common bytes

So to save the two bytes [1][2] you need to keep
  • The UniqueID of the source record (at least 8 bytes)
  • The index of where the matching pattern in the source begins (8 bytes for a 64bit integer since the files are to be of 4 gig in size as you mentioned)
  • The length of the matching pattern (in this case 2) (another 8 bytes for a 64bit int)

So the reference to the match has to be at least 32bytes as I see it.
If you can't find matching patterns of greater than 32bytes then you are storing more data than you are saving.
Was This Post Helpful? 0
  • +
  • -

#18 madmorgan  Icon User is offline

  • D.I.C Head

Reputation: 1
  • View blog
  • Posts: 104
  • Joined: 07-May 10

Re: File Archiver Application

Posted 12 June 2013 - 01:37 PM

tlhIn`toq,

the point of the application is to work in the same what that MAC time machine would work having version control on each file also in a simulator way to how TFS works (TFS keeps all files in a DB and each file is in a bit format).
Was This Post Helpful? 0
  • +
  • -

#19 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 9574
  • View blog
  • Posts: 36,262
  • Joined: 12-June 08

Re: File Archiver Application

Posted 12 June 2013 - 01:42 PM

TFS and a backup program is not quite what this guy has in mind - or at least communicated to us.
Was This Post Helpful? 0
  • +
  • -

#20 madmorgan  Icon User is offline

  • D.I.C Head

Reputation: 1
  • View blog
  • Posts: 104
  • Joined: 07-May 10

Re: File Archiver Application

Posted 12 June 2013 - 01:49 PM

the work and feel like MAC Time Machine collect and store data and like a TFS back end that keeps track of all files and raw data in to a DB
Was This Post Helpful? 0
  • +
  • -

#21 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 9574
  • View blog
  • Posts: 36,262
  • Joined: 12-June 08

Re: File Archiver Application

Posted 12 June 2013 - 01:59 PM

Why would you back up whole files and also maintain some sort of version control on them?

Here's how Time Machine works:

Quote

Time Machine automatically backs up your entire Mac, including system files, applications, accounts, preferences, email messages, music, photos, movies, and documents.
[...] Time Machine keeps hourly backups for the past 24 hours, daily backups for the past month, and weekly backups until your backup drive is full.

https://support.apple.com/kb/HT1427

or..

Quote

Time Machine creates a folder on the designated Time Machine volume that's named the current date and time. It then copies all locally attached drives (except for files and directories that it has specifically been told not to copy, including the Time Machine volume itself) to the folder. Every hour thereafter, it creates a new folder on the remote drive using the same naming scheme. However, instead of making another complete copy of the primary hard drive, Time Machine instead only backs up files that have changed and creates hard links to files that already exist on the remote drive. A user can browse these "versions" of the primary drive and see each file as if it were right where it was left.[3]

Some other backup utilities save "deltas" for file changes, much like version control systems. Such an approach permits more frequent backups of minor changes, but can often complicate the interaction with the backup volume. By contrast, it's possible to manually browse a Time Machine backup volume without using the Time Machine interface; the software's use of hard links makes each backup session appear to the user like a full backup, rather than an incremental or "delta" backup.

https://en.wikipedia...ne_%28Mac_OS%29
Was This Post Helpful? 0
  • +
  • -

#22 Curtis Rutland  Icon User is offline

  • (╯□)╯︵ (~ .o.)~
  • member icon


Reputation: 4577
  • View blog
  • Posts: 8,019
  • Joined: 08-June 10

Re: File Archiver Application

Posted 12 June 2013 - 02:06 PM

Yeah, Time Machine doesn't do some funky byte-pattern-matching-normalization. They do a simple incremental backup, in which they only back up files that have changed since the last backup.
Was This Post Helpful? 0
  • +
  • -

#23 modi123_1  Icon User is online

  • Suitor #2
  • member icon



Reputation: 9574
  • View blog
  • Posts: 36,262
  • Joined: 12-June 08

Re: File Archiver Application

Posted 12 June 2013 - 02:20 PM

Just a stab in the dark - but are you looking at doing something like Boar?
Was This Post Helpful? 0
  • +
  • -

#24 tlhIn`toq  Icon User is offline

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5676
  • View blog
  • Posts: 12,194
  • Joined: 02-June 10

Re: File Archiver Application

Posted 12 June 2013 - 03:00 PM

This is becoming silly in my opinion. There is too much lack of understanding about files, and data and version control, compression algorithms, to make this something we can help with. There are entire textbooks that cover these topics because of their complexity. I'm bowing out of the conversation.
Was This Post Helpful? 0
  • +
  • -

  • (2 Pages)
  • +
  • 1
  • 2