[killnine]

Howdy guys (or should I say humble advisors?),

Anyhow, I am writing a program that takes a bunch of output from a real-time system at 10ms loops and writes them to a comma-separated file. I can output the file just fine. Everything is working there.

However, as you can imagine, at 10ms loops the CSV blows up in size pretty quick.

I was hoping to ZIP up the ole' file before writing it to disk so that way it wouldn't take up as much space (our post-processing tools can unzip files so that's not a big deal).

Anyhow, do you think I should just write the .CSV as I have it now, zip it to a new .ZIP file, and then delete the temporary (raw) CSV or is there a way to do it off the bat?

Here is some code that I use now. Dont mind most of the loops, its really a PITA to get the data from the API of the real-time application. I just wanted to show you guys that I create a steam and write it to a csv.

if (saveFileDialog1.ShowDialog() == DialogResult.OK)
					{
						if ((myStream = saveFileDialog1.OpenFile()) != null)
						{

							StreamWriter wText = new StreamWriter(myStream);

							// write model name
							wText.WriteLine("Model Name: " + tg.GetAppName());
							// write model version
							int[] temp;
							temp = (int[])(Array)tg.GetSignalidsfromLabel("SIG_MODEL_VERSION");
							wText.WriteLine("Model Version: " + tg.GetSignal(temp[0]));
							// write system tunable parameters version
							temp = (int[])(Array)tg.GetSignalidsfromLabel("SIG_PARAMETERS_TUNABLE_VERSION");
							wText.WriteLine("Tunable Parameters Version: " + tg.GetSignal(temp[0]));
							// write system nontunable parameters version
							temp = (int[])(Array)tg.GetSignalidsfromLabel("SIG_PARAMETERS_NONTUNABLE_VERSION");
							wText.WriteLine("Nontunable Parameters Version: " + tg.GetSignal(temp[0]));

							//write time header
							wText.Write("Execution Time,");

							//write signal names to signal header
							for (int i = 0; i < sig_names.Count; i++)
							{
								System.Array sig_index_array = (Array)sig_indexes[i];
								int z = 0;
								for (int y = 0; y < sig_index_array.GetLength(0); y++)
								{
									// simulink arrays not zero index (start at 1)
									z = y + 1;
									wText.Write(sig_names[i] + z.ToString() + ',');
								}
							}

							wText.Write("\r\n");

							//write time stamps and vaules to file
							for (int i = 0; i < sig_values.Count; i = i + 3)
							{
								//write time stamp at start of get
								wText.Write(sig_values[i].ToString() + ", ");

								//write signal values
								double[] sig_text_array = (double[])(Array)sig_values[i + 1];
								for (int y = 0; y < sig_text_array.GetLength(0); y++)
									wText.Write(sig_text_array[y].ToString() + ", ");

								//write time stamp at end of get
								wText.Write(sig_values[i + 2].ToString() + "\r\n");
							}

							wText.Close();
							sig_values.Clear();

						} 

This post has been edited by killnine: 19 July 2007 - 08:27 AM

[Martyr2]

Hi,

Personally I would attempt to go straight to zip with it rather than write, zip and then delete because for one that would be a lot of work for your hard drives and may actually cause them to fragment faster (especially if each file you write is not the same length).

The real question I am not too sure about is how to go about zipping. The .NET framework doesn't currently include classes for handling zip files (that I know of) but as I took a look around I stumbled across a resource that mentions how J# does! Being that both are on the .NET framework and follow a common language specification that you could use a Java zip utility set of classes to do the zipping for you. The alternate choice would be using a utility like #ziplib.

The article I found from MSDN magazine talks about using J# in C# to give you the ability to do some on the fly zipping of files. The article has a downloadable example project or you can even browse the code online without the download.

It might be worth the look if you can't find a more suitable answer.

http://msdn.microsof...on/default.aspx

Hope this helps.

[killnine]

Martyr2, on 19 Jul, 2007 - 10:10 AM, said:

Hi,

Personally I would attempt to go straight to zip with it rather than write, zip and then delete because for one that would be a lot of work for your hard drives and may actually cause them to fragment faster (especially if each file you write is not the same length).

The real question I am not too sure about is how to go about zipping. The .NET framework doesn't currently include classes for handling zip files (that I know of) but as I took a look around I stumbled across a resource that mentions how J# does! Being that both are on the .NET framework and follow a common language specification that you could use a Java zip utility set of classes to do the zipping for you. The alternate choice would be using a utility like #ziplib.

The article I found from MSDN magazine talks about using J# in C# to give you the ability to do some on the fly zipping of files. The article has a downloadable example project or you can even browse the code online without the download.

It might be worth the look if you can't find a more suitable answer.

http://msdn.microsof...on/default.aspx

Hope this helps.

Thanks for the reply. I am currently in the process of creating a ZIP file straight from the stream. I have downloaded the #ziplib and currently have the .dll referenced in my project. It seems sort of straightforward, but I will definitely give any alternative a look. I would prefer to use libraries within the standard .net framework, to be completely honest.

I will likely create a short tutorial on outputting data directly to a .CSV and then zip up the file. I was thinking about what I said earlier, and I guess it doesn't make a whole lot of sense. I am always going to have a CSV, but the idea is to compress it before it goes out to the user. Thus, the only file that shows up after execution is the .zip file. However, the application actually makes both a CSV and a ZIP file.

Thanks again, I will post with updates as they occur.

[gogole]

c'mon why would you want to go extra mile when you can end there. ZIP right from outputStream.