7 Replies - 298 Views - Last Post: 14 April 2019 - 05:43 AM Rate Topic: -----

#1 Larry71   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 66
  • Joined: 05-June 12

Trying to write to a file with ByteBuffer

Posted 05 April 2019 - 11:03 PM

Hallo to everyone,

I am trying to write to a file but with no success till now.
What I have, is an ArrayList of strings and what I try to do is take 30.000 rows everytime, put them in buffer, write to file.
What I get is only the last 30.000 rows.

Here is my code:

 

		
		
		String str = "11111--";

		ArrayList<String> myRows = new ArrayList<String>();

		for(int i = 0; i< 90000; i++) {
			myRows.add(str + i + "\n");
		}
			
		byte[] buffer = "1111111111111".getBytes();

		int number_of_lines = 30000;

                 int times = myRows.size() / number_of_lines;
        
        

		FileChannel rwChannel = null;
		try {
			rwChannel = new RandomAccessFile("myFile.dat", "rw").getChannel();
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}

		ByteBuffer wrBuf;

		try {
			
			wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, buffer.length * number_of_lines );
			

		for(int j=0; j<times; j++) {


		    wrBuf.clear();

		for (int i = (j * number_of_lines); i < ((j+1) * number_of_lines); i++) {
			
			buffer = myRows.get(i).getBytes();
			
		        wrBuf.put(buffer);
		    
		}

		
		
		
		rwChannel.write(wrBuf);

		wrBuf.flip();
		
		

		}

		rwChannel.close();

		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
		System.out.println("File ready");





Thanks a lot.

This post has been edited by Larry71: 05 April 2019 - 11:04 PM


Is This A Good Question/Topic? 0
  • +

Replies To: Trying to write to a file with ByteBuffer

#2 g00se   User is online

  • D.I.C Lover
  • member icon

Reputation: 3623
  • View blog
  • Posts: 16,656
  • Joined: 20-September 08

Re: Trying to write to a file with ByteBuffer

Posted 06 April 2019 - 03:28 AM

Quote

rwChannel.write(wrBuf);

Whatever that's doing will only be done times (3) times as it's outside the inner loop.

I've got to say that my instincts tell me that code is more complex and opaque than it need be
Was This Post Helpful? 0
  • +
  • -

#3 Larry71   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 66
  • Joined: 05-June 12

Re: Trying to write to a file with ByteBuffer

Posted 06 April 2019 - 04:08 AM

View Postg00se, on 06 April 2019 - 03:28 AM, said:

Quote

rwChannel.write(wrBuf);

Whatever that's doing will only be done times (3) times as it's outside the inner loop.

I've got to say that my instincts tell me that code is more complex and opaque than it need be


Yes, you are right. It should be writing 3 times 1/3 of the arrayList.
First from 0 to 29999, second from 30000 to 59999 and third from 60000 to 89999.
But it doesn't.

Well this is a part of a bigger problem. I have to handle a cartesian product of arrayLists of Strings that gives about 15 millions combinations and make some (fast) calculations.
Of course it is impossible to store all of them in memory and decided to use the disc. ByteBuffer and RandomAccessFile was the fastest solution, as it wrote 2 Gb in 25 seconds.
If someone can suggest a solution to the code or the problem, I'd be glad to hear.
Was This Post Helpful? 0
  • +
  • -

#4 g00se   User is online

  • D.I.C Lover
  • member icon

Reputation: 3623
  • View blog
  • Posts: 16,656
  • Joined: 20-September 08

Re: Trying to write to a file with ByteBuffer

Posted 06 April 2019 - 05:31 AM

Using a memory-mapped file is probably a good idea, but not for the creation of the table. You might as well write it using standard IO and only read it using memory-mapping.
Also i'm wondering why, if the strings are numbers, you're not actually writing numbers to the file, since that will use way less memory when it's read
Was This Post Helpful? 0
  • +
  • -

#5 Larry71   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 66
  • Joined: 05-June 12

Re: Trying to write to a file with ByteBuffer

Posted 13 April 2019 - 09:16 AM

View Postg00se, on 06 April 2019 - 05:31 AM, said:

Using a memory-mapped file is probably a good idea, but not for the creation of the table. You might as well write it using standard IO and only read it using memory-mapping.
Also i'm wondering why, if the strings are numbers, you're not actually writing numbers to the file, since that will use way less memory when it's read


Yes, I've tried also standard IO writing and it was too slow. Maybe it was my hardware's fault and especially my hdd.

I don't know if it is the best choise but the solution was to use function force() for my buffer.

g00se, how much will I gain by using numbers? If it is worth, then I' ll have to redesign my classes, that are reading .json files from the internet and all of the fields are coming as Strings.

Thanks a lot.
Was This Post Helpful? 0
  • +
  • -

#6 g00se   User is online

  • D.I.C Lover
  • member icon

Reputation: 3623
  • View blog
  • Posts: 16,656
  • Joined: 20-September 08

Re: Trying to write to a file with ByteBuffer

Posted 13 April 2019 - 11:37 AM

It would really help to know more about what you're actually doing
Was This Post Helpful? 0
  • +
  • -

#7 Larry71   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 66
  • Joined: 05-June 12

Re: Trying to write to a file with ByteBuffer

Posted 14 April 2019 - 12:28 AM

What the programm does is following:

Downloads a json file from the internet. In this file, there are fields that represent Integers but they are stored as Strings, for example "1", "4", "10".
After that, the user may construct an ArrayList of ArrayLists of those Strings, of length 15 max, and the sum can reach about 15 millions. I calculated that storing all of them in memory would need about 3 gb.
Was This Post Helpful? 0
  • +
  • -

#8 g00se   User is online

  • D.I.C Lover
  • member icon

Reputation: 3623
  • View blog
  • Posts: 16,656
  • Joined: 20-September 08

Re: Trying to write to a file with ByteBuffer

Posted 14 April 2019 - 05:43 AM

Still not really enough info -
What is the range of these numbers? How many are there? Why do you think you need to store them in memory? Where is the json file?

This post has been edited by g00se: 14 April 2019 - 05:45 AM

Was This Post Helpful? 0
  • +
  • -

Page 1 of 1