[tomadom]

I'm trying to understand something about compressed streams.
If I'm inflating a compressed stream contained in a file and am sending the inflated output to a second file, does the stream from the source file have to be all on one line? I'm not looking for code examples, just trying to understand this a little better.

Basically I have a small program which compresses text data (using Zlib) and decompresses the same data (also using Zlib). Works fine. But, if I take a stream from, say, a pdf file which has been compressed using Flate Decode and paste that stream into a text file, then why wouldn't it decompress? I'm thinking it has to do with the way I've pasted it into the text file. Maybe some characters aren't displayed properly in a text file? Would that be the reason?

I'm trying to work this out myself but am not sure where to start looking.

Thanks

[Skydiver]

Does your text editor have a maximum line length? Does your text editor automatically break up long line? Does your text editor replace some characters? Does your text editor drop some characters? If you answered yes to any of these, then you just corrupted the compressed data stream.

In other words, you shouldn't be using a text editor to put binary data into a binary file. If you want to put in binary data into a binary file, you should be using a binary file editor.

[tomadom]

I just saved the compressed stream data in Notepad (Windows XP) text file.

I tried saving it in 'Unicode', 'Unicode big endian', 'UTF-8' as well but still doesn't work.

I'm yet to try a binary file editor assuming there's a way to save a file from it.

As you can tell, I'm not very well versed in this area.

[Peter O]

The compressed data is binary data. It should not be handled as text. Some bytes might not correspond to a valid character with a graphical representation. It is possible that a text editor would just drop invalid characters and change or add line endings. Even the slightest change could make the binary data useless. It is very important that the data is saved exactly as it should be.

The best way is to let the program save the file with the compressed data, and make sure you open the file in binary mode when reading and writing to/from the file.

This post has been edited by Peter O: 04 November 2013 - 02:11 AM

[tomadom]

Thanks.. It looks like my cunning plan to test the decoding program on pdf streams was flawed from the start. That's almost certainly what is happening (the text editor is corrupting the compressed stream).

I'll figure something out.

[tomadom]

Peter O, on 04 November 2013 - 01:45 AM, said:

The compressed data is binary data. It should not be handled as text. Some bytes might not correspond to a valid character with a graphical representation. It is possible that a text editor would just drop invalid characters and change or add line endings. Even the slightest change could make the binary data useless. It is very important that the data is saved exactly as it should be.

The best way is to let the program save the file with the compressed data, and make sure you open the file in binary mode when reading and writing to/from the file.

I've taken your advice and learned a bit since then.
I'm still getting data errors though. I'm opening the source file in binary mode and even the destination file in binary mode and transferring the data between the two initially. I'm I'm then taking the output file and trying to decompress the data (all metadata removed by my code). Nothing was opened and saved so I'm not putting in any text viewing characteristics such as newlines etc. Exactly as you've specified above.

When transferring stream data from one file to another, should I include spaces or newlines for each chunk of data transferred? After all, the each stream chunk is only up until the nearest white space.

[jimblumberg]

You need to show the code that is causing the problem.

Normally when writing/reading binary streams you read or write a fixed sized block at a time, this block can have any character embedded within it, including whitespace.


Jim

[tomadom]

jimblumberg, on 05 November 2013 - 10:15 PM, said:

You need to show the code that is causing the problem.

Normally when writing/reading binary streams you read or write a fixed sized block at a time, this block can have any character embedded within it, including whitespace.


Jim

Very good point.. I'm learning allot from this exercise so I'll try that and go from there.