12 Replies - 620 Views - Last Post: 03 March 2013 - 05:48 PM Rate Topic: -----

#1 chipicau  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 48
  • Joined: 15-May 12

Winsock length prefix

Posted 03 March 2013 - 11:31 AM

Hello.

I am building a chat client-server application and it works mostly.
But now I'm faced with the problem of transferring files, as they are binary data.
I was previously using a message delimiter to delimit the end of each message. Unfortunately it doesn't work the same for binary data, so I'm now using a length prefix type of data parsing.
It works, but when I try to read simultaneous sent data, they end up becoming together, and I can't split them because before I could split them by their delimiter, but now I have nothing to split them with.

The way I'm sending the data is:
"8/aTest"

8 is the data's length, "/a" is the character I use to know when to stop looking for the data's length (8), and Test is the actual data in Unicode.

This is the receiver part:

int bytes_to_read = 0;
string data = null;
while (true)
{
	byte[] bytes = new byte[1024];
	int read = tcpclient.GetStream().Read(bytes, 0, bytes.Length);
	if (read == 0)
	{
		break;
	}
	if (bytes_to_read == 0)
	{
		bytes_to_read = System.Convert.ToInt32(System.Text.Encoding.Unicode.GetString(bytes, 0, read).Substring(0, System.Text.Encoding.Unicode.GetString(bytes, 0, read).IndexOf('\a')));
	}
	data += System.Text.Encoding.Unicode.GetString(bytes, 0, read).Substring(System.Text.Encoding.Unicode.GetString(bytes, 0, read).IndexOf('\a') + 1);



Any hints?

Is This A Good Question/Topic? 0
  • +

Replies To: Winsock length prefix

#2 tlhIn`toq  Icon User is online

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5681
  • View blog
  • Posts: 12,224
  • Joined: 02-June 10

Re: Winsock length prefix

Posted 03 March 2013 - 11:50 AM

Well, you can either continue down this road of re-inventing the wheel, or you could follow one of our tutorials on how to build a chat program.

See FAQ # 11


tlhIn`toq's FAQ list

Learning to debug one's own code is an essential skill. Sadly, one that apparently few college courses teach. Silly if you ask me.

Placing breakpoints and walking through the code line by line allows you to actually WATCH it execute.

Visualizing what your code does will let you see why it behaves the way it does.

It would be well worth your time to do the tutorials on FAQ 2. A couple hours learning this skill will save you hundreds of hours of confusion in one project alone.


TOP most asked:
What does this error message mean?
FAQ 2: How do I debug
FAQ 3: How do I make Class1/Form1 talk to Class2/Form2


FAQ (Frequently Asked Questions - Updated Feb 2013
Spoiler



Was This Post Helpful? 1
  • +
  • -

#3 chipicau  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 48
  • Joined: 15-May 12

Re: Winsock length prefix

Posted 03 March 2013 - 12:12 PM

Hello, tlhIn`toq.

Thank you for the fast reply.
I've looked through the chat tutorial you've pointed me to, but it only covered the basics, like sending messages, which I already know.

My problem is with parsing data, if it gets sent simultaneously.

For example, if I send "4\aXY" and right away send "4\aOP", it gets received as "4\aXY4\aOP".

If I were using a delimiter to end the message, I could just do a for loop searching for the delimiter and splitting each message and everything would be fine, which actually was what I was using before. But now I've added a file transfer function into my chat program and instead of messages, I have to send bytes.
Was This Post Helpful? 0
  • +
  • -

#4 tlhIn`toq  Icon User is online

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5681
  • View blog
  • Posts: 12,224
  • Joined: 02-June 10

Re: Winsock length prefix

Posted 03 March 2013 - 12:32 PM

Had you considered adding an EOT character to the End Of your Transmission? (0x004)

Posted Image

Or end each line with a carriage return. That way you can just ReadLine() instead of Read() if you are only sending text.

Or end every transmission with a comma. Then make a point of .Spliting every incoming transmission on the comma and process the collection of data in a loop.

Or since the first value you have is the number of bytes to read, you take that number then read that many bytes.
If there is still more data then its the next transmission: Read the number, read that many bytes.
If there is still more data then its the next transmission: Read the number, read that many bytes.
If there is still more data then its the next transmission: Read the number, read that many bytes.
[...]

As anyone who has read my cookbook can attest: There is more than one way to skin a targ.
Was This Post Helpful? 1
  • +
  • -

#5 chipicau  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 48
  • Joined: 15-May 12

Re: Winsock length prefix

Posted 03 March 2013 - 12:56 PM

Hi again.

Thank you for your valuable input.
Unfortunately, I'm sending bytes, not plain text, so I'm unable to use a delimiter.
I'm looking into your last suggestion and it seems to fit my needs.

I've written some code, though it doesn't fully work, as in, some data doesn't get split from each other.

Note: Because I'm sending Unicode characters to the receiver, each character is 2 bytes, not 1.

int bytes_to_read = 0; //variable that holds the length prefix of the data to receive
string data = null; //variable that holds the actual data
while (true)
{
    byte[] bytes = new byte[1024]; //buffer to hold the received bytes
    int read = tcpclient.GetStream().Read(bytes, 0, bytes.Length); //read the bytes and store the bytes read into a variable
    if (read == 0) //if bytes read is 0, then it means the connection closed, so exit the program
    {
        break;
    }
    if (bytes_to_read == 0)
    {
        bytes_to_read = System.Convert.ToInt32(System.Text.Encoding.Unicode.GetString(bytes, 0, read).Substring(0, System.Text.Encoding.Unicode.GetString(bytes, 0, read).IndexOf('\a'))); //look for the \a character and store every byte into a variable until it reaches it. this variable is the length prefix
    }
    data += System.Text.Encoding.Unicode.GetString(bytes, 0, read).Substring(System.Text.Encoding.Unicode.GetString(bytes, 0, read).IndexOf('\a') + 1); //append the unicode message read into the data variable. the unicode message corresponds to every byte in the buffer array, with the exception from the length prefix bytes
    if (data.Length * 2 >= bytes_to_read) //if the length of the string * 2 (we have to multiply by 2 to get the unicode bytes, because remember that I said that 1 unicode char = 2 bytes) is equal or more than the length prefix, then we have a successful data reading
    {
        string next_data = null; //variable that will be used if there is still some data after the size of the length prefix
        next_data = data.Substring(bytes_to_read / 2); //this variable will be the string contained from the length prefix up until the end
        data = data.Substring(0, bytes_to_read / 2); //the variable data will be parsed so it only contains the string that has the size of the length prefix
        switch (data)
        {
            case "command1":
                //...
                break;
            case "command2":
                //...
                break;
        }
        bytes_to_read = 0;
        data = next_data; //set the variable data to have the data remaining after the length of the message, if any
    }
}

Was This Post Helpful? 0
  • +
  • -

#6 tlhIn`toq  Icon User is online

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5681
  • View blog
  • Posts: 12,224
  • Joined: 02-June 10

Re: Winsock length prefix

Posted 03 March 2013 - 01:08 PM

Quote

Unfortunately, I'm sending bytes, not plain text, so I'm unable to use a delimiter.

Why not? ITs been done for decades in communications. EOT is a byte of 4. What's the problem? Have you even tried before you dismiss the idea?

You said the tutorials wouldn't help you because you already know how to send info. I'd argue that based on your response. I think you should do more than just read that tutorial. I think you should actually stop on your own project and BUILD the project from the tutorial. I think you would find that it doesn't suffer the same shortcomings as your project. IE: It doesn't concatenate transmissions together. So instead of spending your time trying to patch your current problem, you should learn how to do this in a way that you don't create the problem in the first place.
Was This Post Helpful? 1
  • +
  • -

#7 chipicau  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 48
  • Joined: 15-May 12

Re: Winsock length prefix

Posted 03 March 2013 - 02:31 PM

Please just help me getting the code I posted to work.
It's best to use length prefixes when dealing with binary data.
Was This Post Helpful? 0
  • +
  • -

#8 Momerath  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 1012
  • View blog
  • Posts: 2,444
  • Joined: 04-October 09

Re: Winsock length prefix

Posted 03 March 2013 - 02:38 PM

Why would you send 4\aXY, isn't that invalid? I only see 2 bytes there, why are you indicating 4? And even if it was 4, what's wrong with 4\aXY4\aAB? You read the first part to determine the length, get the next 4 bytes, then you know the next part has to be another length indicator.

Quote

Please just help me getting the code I posted to work.


If you are doing it wrong, why would we want to help you continue on that path?
Was This Post Helpful? 1
  • +
  • -

#9 tlhIn`toq  Icon User is online

  • Please show what you have already tried when asking a question.
  • member icon

Reputation: 5681
  • View blog
  • Posts: 12,224
  • Joined: 02-June 10

Re: Winsock length prefix

Posted 03 March 2013 - 02:53 PM

View Postchipicau, on 03 March 2013 - 03:31 PM, said:

Please just help me getting the code I posted to work.
It's best to use length prefixes when dealing with binary data.


Sorry for how this sounds, but this is based a few years experience here on DIC.
I don't help patch crappy code. It inevitably leads to more problems, more work, more questions on how to make more patches - and becomes precident for how to do something. In the future someone will post a question saying "I saw you tell someone else this was how to do xyz and now I'm stuck."

Help you patch a bug that should never exist isn't helping you. You need to make your program not throw the bug to begin with. If you continue down this road your program becomes a patchwork quilt of bug and band-aid on top of bug and band-aid.

In other words we would rather fix your broken leg, than continuously dispense pain meds for the symptoms and send you home still broken.

View PostMomerath, on 03 March 2013 - 03:38 PM, said:

what's wrong with 4\aXY4\aAB? You read the first part to determine the length, get the next 4 bytes, then you know the next part has to be another length indicator.
[...]
If you are doing it wrong, why would we want to help you continue on that path?


Advice already given in post #4, but the OP either couldn't be bothered with trying to make that idea work - or is waiting for someone to just give them working code.

Either way this thread is rapidly deteriorating into the same advice being given repeatedly.
chipicau - you need to provide something to show you are at least trying the advice we are providing or this thread is going to get closed as unproductive/'gemme teh codez'
Was This Post Helpful? 1
  • +
  • -

#10 chipicau  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 48
  • Joined: 15-May 12

Re: Winsock length prefix

Posted 03 March 2013 - 03:51 PM

Hey.

I'd just like to point that I'm not asking you to give me the code like you said, I tried to come up with a solution based on one of your suggestions and it failed. I only asked for a fix.
But alright, if it's the wrong way, I understand.

Assuming I would add an EOT character, isn't it possible that some data exchanged by the chat applications to have an EOT character already?
That is my only fear.

@Momerath:
Because I'm sending the data as an unicode string, each character is 2 bytes, not 1.


Thank you.
Was This Post Helpful? 0
  • +
  • -

#11 Momerath  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 1012
  • View blog
  • Posts: 2,444
  • Joined: 04-October 09

Re: Winsock length prefix

Posted 03 March 2013 - 04:03 PM

I thought you were sending binary data, not text data. No need for lengths and crap for text data, just put an EOL on the end and split on the EOL.

Now if you really want to send binary data, then stop sending unicode.
Was This Post Helpful? 2
  • +
  • -

#12 chipicau  Icon User is offline

  • New D.I.C Head

Reputation: 1
  • View blog
  • Posts: 48
  • Joined: 15-May 12

Re: Winsock length prefix

Posted 03 March 2013 - 04:06 PM

Hello Momerath.

I'm only sending binary data because I have to support file transferring.

This post has been edited by chipicau: 03 March 2013 - 04:06 PM

Was This Post Helpful? 0
  • +
  • -

#13 Momerath  Icon User is offline

  • D.I.C Lover
  • member icon

Reputation: 1012
  • View blog
  • Posts: 2,444
  • Joined: 04-October 09

Re: Winsock length prefix

Posted 03 March 2013 - 05:48 PM

Ok, you want to send binary data. So create a 'frame' for the data. The first two bytes are the length of the data, the next x (based on the first two) are the data itself. This will allow you to send up to 65535 bytes at a time. If you need more, use more bytes at the front (4 bytes gives you 4 billion+ bytes).

If you want to break it up in chunks you can add some info (in case you use UDP instead of TCP) that indicates what chunk it is:

Bytes 0-1 : length of data
Bytes 2-3 : chunk number
Bytes 4-x : data

Again, this gives you 4 billion+ bytes per 'file'.

No need for doing any conversions to/from other data types, it's all bytes.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1