8 Replies - 2729 Views - Last Post: 13 June 2008 - 03:49 AM Rate Topic: -----

#1 mloone01  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 7
  • Joined: 09-June 08

Write large amounts of data to file

Posted 11 June 2008 - 07:23 AM

I am writing to an excel file a very large amount of data using fprintf . On writing 65535 items of data to excel the file is full and therefore no more data is written. I would like to be able to write alot of data to a file so that i can graph this data after. Any ideas for C/C++ on how i can gather and plot this data?
Is This A Good Question/Topic? 0
  • +

Replies To: Write large amounts of data to file

#2 MikeRaines  Icon User is offline

  • D.I.C Head

Reputation: 1
  • View blog
  • Posts: 78
  • Joined: 30-October 07

Re: Write large amounts of data to file

Posted 11 June 2008 - 08:13 AM

What do you mean by "the file is full"? If my understanding is correct you should be able to write until you run out of disk space.

As for writing large amounts of data. Pre-Buffering all your data into one giant array and then initiating one big I/O write session would be the way I would do it.
Was This Post Helpful? 0
  • +
  • -

#3 mloone01  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 7
  • Joined: 09-June 08

Re: Write large amounts of data to file

Posted 11 June 2008 - 08:17 AM

View PostMikeRaines, on 11 Jun, 2008 - 08:13 AM, said:

What do you mean by "the file is full"? If my understanding is correct you should be able to write until you run out of disk space.

As for writing large amounts of data. Pre-Buffering all your data into one giant array and then initiating one big I/O write session would be the way I would do it.


When writing to Excel a limit of 65535 rows is only permitted and nothing greater than this. This cannot be changed and therefore only the first 65535 values are gathered.

How would you display this array as a graph in C/C++?
Was This Post Helpful? 0
  • +
  • -

#4 MikeRaines  Icon User is offline

  • D.I.C Head

Reputation: 1
  • View blog
  • Posts: 78
  • Joined: 30-October 07

Re: Write large amounts of data to file

Posted 11 June 2008 - 08:22 AM

Ah I see :)

Do you have access to MATLAB?

MATLAB has excellent graphing capabilities.

If not your going to have to write your own sort of imaging GUI.

Time consuming but not too difficult.
Was This Post Helpful? 0
  • +
  • -

#5 mloone01  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 7
  • Joined: 09-June 08

Re: Write large amounts of data to file

Posted 11 June 2008 - 08:30 AM

View PostMikeRaines, on 11 Jun, 2008 - 08:22 AM, said:

Ah I see :)

Do you have access to MATLAB?

MATLAB has excellent graphing capabilities.

If not your going to have to write your own sort of imaging GUI.

Time consuming but not too difficult.

I have access to MATLAB so I can perhaps transfer over my code, thanks
Was This Post Helpful? 0
  • +
  • -

#6 MikeRaines  Icon User is offline

  • D.I.C Head

Reputation: 1
  • View blog
  • Posts: 78
  • Joined: 30-October 07

Re: Write large amounts of data to file

Posted 11 June 2008 - 08:33 AM

Well I would suggest sticking to the path of C/C++ since you already are writing to a file. Just simply write your data to binary file.

Then you can gum up a simple script in MATLAB to grab the file you write to and plot it.

If you have any questions about MATLAB I/O or plotting just send me a PM or respond to this. MATLAB is my bread and butter.

Hope this was helpful,
Mike

This post has been edited by MikeRaines: 11 June 2008 - 08:34 AM

Was This Post Helpful? 0
  • +
  • -

#7 perfectly.insane  Icon User is offline

  • D.I.C Addict
  • member icon

Reputation: 70
  • View blog
  • Posts: 644
  • Joined: 22-March 08

Re: Write large amounts of data to file

Posted 12 June 2008 - 03:17 PM

View Postmloone01, on 11 Jun, 2008 - 08:17 AM, said:

View PostMikeRaines, on 11 Jun, 2008 - 08:13 AM, said:

What do you mean by "the file is full"? If my understanding is correct you should be able to write until you run out of disk space.

As for writing large amounts of data. Pre-Buffering all your data into one giant array and then initiating one big I/O write session would be the way I would do it.


When writing to Excel a limit of 65535 rows is only permitted and nothing greater than this. This cannot be changed and therefore only the first 65535 values are gathered.

How would you display this array as a graph in C/C++?


Excel 2007 does not have this limitation, but workbooks with more than 2**16 rows must be saved in an Excel 2007 format.

Scilab is a free program that has a programming language that is similar to MATLAB.
Was This Post Helpful? 0
  • +
  • -

#8 bizzehdee  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 5
  • View blog
  • Posts: 54
  • Joined: 06-April 08

Re: Write large amounts of data to file

Posted 12 June 2008 - 03:21 PM

if your writing the graphing in C++, why use excell at all?
write the data out in your own format, and possibly write an excell import/export function for it, this way your graphing and statistics is always ran how YOU want it to be ran
Was This Post Helpful? 0
  • +
  • -

#9 mloone01  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 7
  • Joined: 09-June 08

Re: Write large amounts of data to file

Posted 13 June 2008 - 03:49 AM

View PostMikeRaines, on 11 Jun, 2008 - 08:33 AM, said:

Well I would suggest sticking to the path of C/C++ since you already are writing to a file. Just simply write your data to binary file.

Then you can gum up a simple script in MATLAB to grab the file you write to and plot it.

If you have any questions about MATLAB I/O or plotting just send me a PM or respond to this. MATLAB is my bread and butter.

Hope this was helpful,
Mike


Thank you all for your replies. However, I have taken MikeRaines advice on continuing with C/C++ and using Matlab to grab my file. I tried to PM you Mike but received an error so am posting here. Is there a way of reading in from Matlab row by row. For instance, I have written out to a binary file 2 columns of data. I would like to read in each row of data and plot the data onto a file as I am going along. The reason I am doing this is that when I use
<load filename.bin> I am loading a v.large file and eventually an error file informs me that it is out of memory. Therefore I am hoping that by reading in the data row by row and plotting it out to the same figure I should get the graph of detail I am looking for.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1