Page 1 of 1

java.utils.zip, sharpziplib, shell32.dll compression Rate Topic: -----

#1 ricardosms  Icon User is offline

  • D.I.C Regular
  • member icon

Reputation: 73
  • View blog
  • Posts: 301
  • Joined: 02-April 10

Posted 09 August 2011 - 08:38 AM

Hello Everybody:

This is not really a tutorial, but just some notes about compression situations I had to deal with recently:
I have a windows application that is like a dashboard where I keep my customers lists, input an output different kind of data and call other programs from it. It is not a standard application, but serves me good for what I need. I has gone through different versions, starting as a DOS program, and evolving through windows 3, 95,98,xp,vista and 7.
The last version came as a need when my computer whith windows vista broke up and I had to buy one with Windows 7. I make my own programs, so I decided to compile it with Visual Basic 2010 Express. And it didn't compile due to few errors.
Anyway, one of the subroutines I call from within my program is for file compression, that I call occasionally to backup the new information obtained. This allows me to compress my files without exiting my program.
At the beginning I had a batch file that utilized pkzip, it was a 16 bit program and worked Ok up to windows XP, and I kept it as an option, but I started using java.utils.zip that could be called from VB.Net 2008 and Net framework 3.5:
    Private Sub ToolStripMenuItem11_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ToolStripMenuItem11.Click
        Dim obj As New clsZip
        Dim folder As String = System.AppDomain.CurrentDomain.BaseDirectory

        If Not folder Is Nothing AndAlso IO.Directory.Exists(folder) Then

            Dim di As New IO.DirectoryInfo(System.AppDomain.CurrentDomain.BaseDirectory)
            Dim diar1 As IO.FileInfo() = di.GetFiles("*.re*", SearchOption.TopDirectoryOnly)
            Dim diar2 As IO.FileInfo() = di.GetFiles("*.od*", SearchOption.TopDirectoryOnly)
            Dim diar3 As IO.FileInfo() = di.GetFiles("*.doc*", SearchOption.TopDirectoryOnly)
            Dim diar4 As IO.FileInfo() = di.GetFiles("*.txt", SearchOption.TopDirectoryOnly)

            Dim dra As IO.FileInfo
            Y.Clear()
            For Each dra In diar1
                Y.Add(dra.Name)
            Next
            For Each dra In diar2
                Y.Add(dra.Name)
            Next
            For Each dra In diar3
                Y.Add(dra.Name)
            Next
            For Each dra In diar4
                Y.Add(dra.Name)
            Next
        End If

        Y.Sort()

        obj.CreateZipFile(System.AppDomain.CurrentDomain.BaseDirectory)
        MsgBox(Format(Now, "MMMddyy") & ".zip" & " Was Created", MsgBoxStyle.Information, "Zip File")
    End Sub



And the zip class:
Imports System.IO
Imports java.io
Imports java.util
Imports java.util.zip


Public Class clsZip
    Dim y As New ArrayList()
    Public Sub CreateZipFile(ByVal sPath As String)
        Dim fos As java.io.FileOutputStream
        Dim zos As java.util.zip.ZipOutputStream
        Dim di As System.IO.DirectoryInfo

        If System.IO.Directory.Exists(sPath) Then
            'create a zip file with same name in the same path
            fos = New java.io.FileOutputStream(Format(Now, "MMMddyy") & ".zip")
            zos = New java.util.zip.ZipOutputStream(fos)
            di = New System.IO.DirectoryInfo(sPath)

            'procedure to zip a directory
            ZipDirectory(fos, zos, di, sPath)
            zos.close()
            fos.close()
            zos.flush()
            fos.flush()

        End If


    End Sub

    Private Sub ZipDirectory(ByVal fos As java.io.FileOutputStream, ByVal zos As java.util.zip.ZipOutputStream, ByVal di As System.IO.DirectoryInfo, ByVal SRootDir As String)
        Dim fis As java.io.FileInputStream
        Dim ze As java.util.zip.ZipEntry
        Dim j As Integer
     
        For j = 0 To Form1.Y.Count - 1
            ze = New java.util.zip.ZipEntry(Form1.Y.Item(j).ToString)
            ze.setMethod(ze.DEFLATED)
            zos.putNextEntry(ze)
            fis = New java.io.FileInputStream(System.AppDomain.CurrentDomain.BaseDirectory & Form1.Y.Item(j).ToString)
            CopyStream(fis, zos) 'Copy stream is a simple method to read a file input stream (file to zip) and write it to a file output stream(new zip file)
            zos.closeEntry()
            fis.close()
        Next

    End Sub

    Private Sub ZipOneFile(ByVal fos As java.io.FileOutputStream, ByVal zos As java.util.zip.ZipOutputStream, ByVal sFullName As String)
        Dim fis As java.io.FileInputStream
        Dim ze As java.util.zip.ZipEntry
        'give the zip entry or the folder arrangement for the file
        ze = New java.util.zip.ZipEntry(sFullName.Substring(sFullName.LastIndexOf("\")))
        'The DEFLATED method is the one of the methods to zip a file
        ze.setMethod(ze.DEFLATED)
        zos.putNextEntry(ze)
        'Input stream for the file to zip
        fis = New java.io.FileInputStream(sFullName)
        'Copy stream is a simple method to read a file input stream (file to zip) and write it to a file output stream(new zip file)
        CopyStream(fis, zos)
        zos.closeEntry()
        fis.close()
    End Sub

    Private Sub CopyStream(ByVal src As java.io.FileInputStream, ByVal dest As java.util.zip.ZipOutputStream)
        Dim reader As New java.io.InputStreamReader(src)
        Dim writer As New java.io.OutputStreamWriter(dest)
        While reader.ready
            writer.write(reader.read)
        End While
        writer.flush()
    End Sub


End Class



It did work very good and I never worried about it. It could compress open files as well as hidden files, so I tought this was standard. But I was wrong.
Net framework 4.0 dropped support of java. So my program was kind of broken, so temorarily, I did my backups manually, and started checking around. The buit-in classes on the framework, didn't do what I needed, and some of them were not on a format that external programs could read, besides my system is 64 bit and won't nun 16 bit programs.
I found Sharpziplib, it was easy to program, and I worked on it for a little bit, but I found a limitation: it gave me access errors when trying to compress files open on another applicaton.
Called like this:
     Private Sub ToolStripMenuItem11_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ToolStripMenuItem11.Click
          ZipClass.CreateSample(String.Format("{0:MMMddyy}", DateTime.Now) & ".zip", "", System.Environment.CurrentDirectory)
         MessageBox.Show(String.Format("{0:MMMddyy}", DateTime.Now) & ".zip was created")
     End Sub
    


And the module:

Imports System
Imports System.IO
Imports ICSharpCode.SharpZipLib.Core
Imports ICSharpCode.SharpZipLib.Zip
Module ZipClass
    
    Public Sub CreateSample(ByVal outPathname As String, ByVal password As String, ByVal folderName As String)
        ' Dim ExtensionsToCompress As String = ".TXT_.REC_.REP_.ODS_.ODT_.DOC_.XLS_.RTF_.PDF_.HTM"
        Dim ExtensionsToCompress As String = ".TXT_.REC_.REP_.ODS_.ODT_.XLS_.RTF"
        Dim Filter As String
        Dim fsOut As FileStream = File.Create(outPathname)
        Dim zipStream As New ZipOutputStream(fsOut)

        zipStream.SetLevel(3)   '0-9, 9 being the highest level of compression
        zipStream.Password = password   ' optional. Null is the same as not setting.
        ' If folderName = "" Then folderName = Directory.GetCurrentDirectory() & "\"
        Dim files As String() = Directory.GetFiles(folderName)

        For Each filename As String In files

            Dim fi As New FileInfo(filename)
            Filter = UCase(Path.GetExtension(filename))

            ' If UCase(filename).Contains(".ZIP") Or UCase(filename).Contains(".DOC") Then
            ' Else
            If ExtensionsToCompress.Contains(Filter) Then

                Dim entryName As String = ZipEntry.CleanName(filename) ' Removes drive from name and fixes slash direction
                ' MessageBox.Show(entryName)


                ' Dim newEntry As New ZipEntry(entryName) ''All The Subolders Path
                Dim newentry As New ZipEntry(Path.GetFileName(filename)) '' No Path, Only FileName

                newentry.DateTime = fi.LastWriteTime    ' Note the zip format stores 2 second granularity
                ' Specifying the AESKeySize triggers AES encryption. Allowable values are 0 (off), 128 or 256.
                '   newEntry.AESKeySize = 256;

                ' To permit the zip to be unpacked by built-in extractor in WinXP and Server2003, WinZip 8, Java, and other older code,
                ' you need to do one of the following: Specify UseZip64.Off, or set the Size.
                ' If the file may be bigger than 4GB, or you do not need WinXP built-in compatibility, you do not need either,
                ' but the zip will be in Zip64 format which not all utilities can understand.
                '   zipStream.UseZip64 = UseZip64.Off;
                newentry.Size = fi.Length

                zipStream.PutNextEntry(newentry)

                ' Zip the file in buffered chunks
                ' the "using" will close the stream even if an exception occurs
                Dim buffer As Byte() = New Byte(4095) {}
                Using streamReader As FileStream = File.OpenRead(filename)
                    StreamUtils.Copy(streamReader, zipStream, buffer)
                End Using

                zipStream.CloseEntry()


                ' End If

            End If
        Next
        zipStream.IsStreamOwner = True  ' Makes the Close also Close the underlying stream
        zipStream.Close()
    End Sub

End Module



My daily "ToDo" list is usually open. It is a text document where I list my things pending to do, and it alwas has to be ready to print the first page, because I do house calls, and I have to carry with me the information there. This is written on Open Office Writer, and it happens that OO locks the file. If I saved and closed the program before compressing, it was ok, but I had to start OO again.

If I did a manual compression with another utility, things worked ok, and Windows and programs like ImgBurn coud read it and copy it , even been open om OO.
I needed to find something else. My thoughs told me that if there was copy on disk, and another in memory, why can't I read the one on disk?

Some programs can do that. And the "shell32.dll", part of the windows system, can do that. It doesn't compress hidden files (At least I haven't found how, yet), but compresses open files, and that is what I need. Anothed advantage is that it is part of the operating system , and may stay on it.
Anyway, it can compress in one shot, files, subdirectories, and files inside subdirectories:
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        'Compress Files Only
        TextBox1.Text = ""

        Try
            Dim emptyzip() As Byte = New Byte() {80, 75, 5, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}

            Dim fs As FileStream = File.Create("test.zip")
            fs.Write(emptyzip, 0, emptyzip.Length)
            fs.Flush()
            fs.Close()
            fs = Nothing
            Dim sc As New Shell32.Shell
            Dim SrcFlder As Shell32.Folder = sc.NameSpace("e:\repairs")
            Dim DestFlder As Shell32.Folder = sc.NameSpace("("e:\repairs\test.zip")
            Dim items As Shell32.FolderItems = SrcFlder.Items()
            counter = SrcFlder.Items.Count

            For Each Y As Object In SrcFlder.Items 'Shell32.FolderItem In SrcFlder.Items

                TextBox1.Text = TextBox1.Text & vbNewLine & Y.Name & " is " & Y.Type.ToString

                If UCase(Y.Type.ToString).Contains("FOLDER") Then
                    TextBox1.Text = TextBox1.Text & "------------> Skipped This One "
                    ' counter = counter - 1

                Else
                    DestFlder.CopyHere(Y, 16) '20)
                    System.Threading.Thread.Sleep(1500)

                End If
            Next


              Dim i As Integer = SrcFlder.Items.Count
             Console.WriteLine(i)
            'Check if all items have been copied to finish operation, check if some are skipped

             While (DestFlder.Items.Count < counter)
            System.Threading.Thread.Sleep(1000)
            End While
        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try
    End Sub


But to compress files only or a subset of files, you have to use some filters and compress one file at the time, and if you don't allow enough time to compress each file, it may give you access errors, because every time you call the copyhere function, you start a new process.
I have few gigabytes of data, from 12 years of running my business, every year's data is in a separate directory. I check things there some times, but the data doesn't change, this year's data is about 20 megabytes, and the files that have changed recently may be about 1 megabyte.
I only need to backut about 20 files, in total, about 300k compressed.
Still with this limited amount of data to compress, I can't guarantee that there won't be errors.
The "items" object in a folder can not be manipulated. You have ojects, inside it that represent files, folders and links, but the only way it changes is by changing the objects on it.

    Private Sub Button3_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button3.Click
        ' Skip Some Files And Folders
        TextBox1.Text = ""
        Try
            Dim emptyzip() As Byte = New Byte() {80, 75, 5, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
            Dim fs As FileStream = File.Create(Format(Now, "MMMddyy") & ".zip")
            fs.Write(emptyzip, 0, emptyzip.Length)
            fs.Flush()
            fs.Close()
            fs = Nothing
            Dim sc As New Shell32.Shell
            Dim SrcFlder As Shell32.Folder = sc.NameSpace(AppDomain.CurrentDomain.BaseDirectory) '("e:\repairs")
            ' Dim DestFlder As Shell32.Folder = sc.NameSpace(AppDomain.CurrentDomain.BaseDirectory & "test.zip")
            Dim DestFlder As Shell32.Folder = sc.NameSpace(AppDomain.CurrentDomain.BaseDirectory & Format(Now, "MMMddyy") & ".zip")
            Dim items As Shell32.FolderItems = SrcFlder.Items()

            counter = SrcFlder.Items.Count

            Me.Cursor = Cursors.WaitCursor
            For Each Y As Shell32.FolderItem In SrcFlder.Items
                If UCase(Y.Name).Contains(".REC") Or UCase(Y.Name).Contains(".REP") Or UCase(Y.Name).Contains(".DOC") _
                    Or UCase(Y.Name).Contains(".ODS") Or UCase(Y.Name).Contains(".ODT") Or UCase(Y.Name).Contains(".TXT") Then

                    DestFlder.CopyHere(Y, 16) '20)
                    If Y.Size > 500000 Then
                        System.Threading.Thread.Sleep(2500)
                    Else
                        System.Threading.Thread.Sleep(500)
                    End If
                Else

                End If

            Next

            '  DestFlder.CopyHere(items, 16) '20)
        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try
        Me.Cursor = Cursors.Default
    End Sub



Here I am allowing different times, depending on file size, but, How do I know how much is enough?
I don't know!
My solution is to create a temporary directory, copy there the files to compress and creating my zip file with the files there afterwards, in only one separate process.
    Private Sub ToolStripMenuItem11_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ToolStripMenuItem11.Click
        If Directory.Exists(AppDomain.CurrentDomain.BaseDirectory & "Temp") Then
            My.Computer.FileSystem.DeleteDirectory(AppDomain.CurrentDomain.BaseDirectory & "Temp", FileIO.DeleteDirectoryOption.DeleteAllContents)
        End If
        Directory.CreateDirectory(AppDomain.CurrentDomain.BaseDirectory & "Temp")

        Dim objShell As Shell
        Dim objFolder, objFolder1 As Folder

        objShell = New Shell
        objFolder = objShell.NameSpace(AppDomain.CurrentDomain.BaseDirectory & "Temp")
        objFolder1 = objShell.NameSpace(AppDomain.CurrentDomain.BaseDirectory)

        For Each Itm As ShellFolderItem In objFolder1.Items
            If UCase(Itm.Name).EndsWith(".REC") Or UCase(Itm.Name).EndsWith(".REP") Or UCase(Itm.Name).EndsWith(".DOC") _
                Or UCase(Itm.Name).EndsWith(".ODS") Or UCase(Itm.Name).EndsWith(".ODT") Or UCase(Itm.Name).EndsWith(".RTF") Or UCase(Itm.Name).EndsWith(".TXT") Then

                ' Replace Files OlderThan 3 Months
                ' System.IO.File.Delete(currentFile)

                Dim dateDiff As TimeSpan = DateTime.Now.Subtract(Itm.ModifyDate)

                If dateDiff.Days <= 90 Then

                    objFolder.CopyHere(AppDomain.CurrentDomain.BaseDirectory & Itm.Name, 16)

                End If
            End If
        Next

        objFolder = Nothing
        objShell = Nothing

        Try
            Dim emptyzip() As Byte = New Byte() {80, 75, 5, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}

            Dim fs As FileStream = File.Create(Format(Now, "MMMddyy") & ".zip")
            fs.Write(emptyzip, 0, emptyzip.Length)
            fs.Flush()
            fs.Close()
            fs = Nothing
            Dim sc As New Shell32.Shell
            Dim SrcFlder As Shell32.Folder = sc.NameSpace(AppDomain.CurrentDomain.BaseDirectory & "Temp")
            Dim DestFlder As Shell32.Folder = sc.NameSpace(AppDomain.CurrentDomain.BaseDirectory & Format(Now, "MMMddyy") & ".zip")
            Dim items As Shell32.FolderItems = SrcFlder.Items()

            For Each Y As Shell32.FolderItem In SrcFlder.Items

                TextBox1.Text = TextBox1.Text & vbNewLine & Y.Name & " is " & Y.Type.ToString
            Next

            DestFlder.CopyHere(items, 20)
            Dim i As Integer = SrcFlder.Items.Count
            'Check if all items have been copied to finish operation, check if some are skipped

            While (DestFlder.Items.Count < i)
                System.Threading.Thread.Sleep(1000)
            End While
            MessageBox.Show(String.Format("{0:MMMddyy}", DateTime.Now) & ".zip was created")

        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try

    End Sub


A side effect of the use of shell32 is that the shell keyword is used for a type, if you call a program from inside your program like this, gives you an error:
Shell("Explorer " & AppDomain.CurrentDomain.BaseDirectory, AppWinStyle.NormalFocus)


shell is a type and can not be used in an expression, so call it like this:
    Private Sub ToolStripMenuItem12_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles ToolStripMenuItem12.Click
        Microsoft.VisualBasic.Shell("Explorer " & AppDomain.CurrentDomain.BaseDirectory, AppWinStyle.NormalFocus)
    End Sub


Don't forger to add a reference to shell32.dll on windows\system32 folder and import shell32.

I hope this is helpfull to someone.
Thanks for reading.
ricardosms.

Is This A Good Question/Topic? 0
  • +

Page 1 of 1