3 Replies - 2012 Views - Last Post: 11 May 2012 - 11:44 AM Rate Topic: -----

#1 appzone  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 17
  • Joined: 07-March 12

[ASK] BinaryReader

Posted 04 May 2012 - 09:19 PM

hi all
i want to ask about read file and then convert to binary

i am using this code


    Function readd(ByVal path As String, Optional ByVal Separator As String = " ")
        ' Store the line in this String.
        Dim line As String

        ' Create new StreamReader instance with Using block.
        Using reader As StreamReader = New StreamReader(path)
            ' Read one line from file
            line = reader.ReadToEnd
        End Using

        ' Write the line we read from "file.txt"
        Dim oReturn As New StringBuilder

        ' Convert to ASCII and go through all the bytes
        For Each Character As Byte In ASCIIEncoding.ASCII.GetBytes(line)
            oReturn.Append(Convert.ToString(Character, 2).PadLeft(8, "0"))
            oReturn.Append(Separator)
        Next




        Return oReturn.ToString

    End Function




it works fine to read small file, example 2MB file, etc

but it get error "Exception of type 'System.OutOfMemoryException' was thrown." when try to read 100MB file

i think the error code because the file is too big

For Each Character As Byte In ASCIIEncoding.ASCII.GetBytes(line)
            oReturn.Append(Convert.ToString(Character, 2).PadLeft(8, "0"))
            oReturn.Append(Separator)
        Next




how can i read 100MB or bigger file??
or maybe is it another method / code to read a file and convert it into binary?

thanx :D

Is This A Good Question/Topic? 0
  • +

Replies To: [ASK] BinaryReader

#2 Ionut  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 386
  • View blog
  • Posts: 1,057
  • Joined: 17-July 10

Re: [ASK] BinaryReader

Posted 05 May 2012 - 12:12 AM

First of all, storing into a variable 100MB is a big waste of memory. I would recommand you to read line by line(or chunks of code(for example 1000/2000/4096 characters)), convert each line/chunk, and write it to another file(or do whatever you have to do with that information). Also, you should use in this situation StringBuilder class
Was This Post Helpful? 0
  • +
  • -

#3 Tethik  Icon User is offline

  • D.I.C Head

Reputation: 17
  • View blog
  • Posts: 63
  • Joined: 14-March 10

Re: [ASK] BinaryReader

Posted 05 May 2012 - 03:52 AM

Let's analyse where all the memory disappears to:
    Function readd(ByVal path As String, Optional ByVal Separator As String = " ")
        ' Store the line in this String.
        Dim line As String

        ' Create new StreamReader instance with Using block.
        Using reader As StreamReader = New StreamReader(path)
            ' Read one line from file
            line = reader.ReadToEnd
        End Using


After this the variable line should contain all the contents of the file given by path. So 100 mb.

        ' Convert to ASCII and go through all the bytes
        For Each Character As Byte In ASCIIEncoding.ASCII.GetBytes(line)
            oReturn.Append(Convert.ToString(Character, 2).PadLeft(8, "0"))
            oReturn.Append(Separator)
        Next


Out of every byte you're now converting to a string in base 2. So now for example:
"a" -> "01100001" // 8 times bigger than original byte.


You then add another zero and a separator, so at least 2 more chars per char to the new string that is being built.

I guestimate that in the end the program will be using something like 1,1 GB of memory to read your file 100 MB into a string.

To fix you may want to consider writing it to another file like Ionut said. You could also grab the bytes directly from the stream by reading them one at a time, instead of storing them all in the line variable. This should cut 100 MB of memory usage.

This post has been edited by Tethik: 05 May 2012 - 03:53 AM

Was This Post Helpful? 1
  • +
  • -

#4 appzone  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 17
  • Joined: 07-March 12

Re: [ASK] BinaryReader

Posted 11 May 2012 - 11:44 AM

View PostIonut, on 05 May 2012 - 12:12 AM, said:

First of all, storing into a variable 100MB is a big waste of memory. I would recommand you to read line by line(or chunks of code(for example 1000/2000/4096 characters)), convert each line/chunk, and write it to another file(or do whatever you have to do with that information). Also, you should use in this situation StringBuilder class



View PostTethik, on 05 May 2012 - 03:52 AM, said:

Let's analyse where all the memory disappears to:
    Function readd(ByVal path As String, Optional ByVal Separator As String = " ")
        ' Store the line in this String.
        Dim line As String

        ' Create new StreamReader instance with Using block.
        Using reader As StreamReader = New StreamReader(path)
            ' Read one line from file
            line = reader.ReadToEnd
        End Using


After this the variable line should contain all the contents of the file given by path. So 100 mb.

        ' Convert to ASCII and go through all the bytes
        For Each Character As Byte In ASCIIEncoding.ASCII.GetBytes(line)
            oReturn.Append(Convert.ToString(Character, 2).PadLeft(8, "0"))
            oReturn.Append(Separator)
        Next


Out of every byte you're now converting to a string in base 2. So now for example:
"a" -> "01100001" // 8 times bigger than original byte.


You then add another zero and a separator, so at least 2 more chars per char to the new string that is being built.

I guestimate that in the end the program will be using something like 1,1 GB of memory to read your file 100 MB into a string.

To fix you may want to consider writing it to another file like Ionut said. You could also grab the bytes directly from the stream by reading them one at a time, instead of storing them all in the line variable. This should cut 100 MB of memory usage.


yea thanx for the solution sir

i just noticed and help from your help
thanx you
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1