14 Replies - 720 Views - Last Post: 04 February 2013 - 08:48 PM Rate Topic: -----

#1 Cbeppe  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 31
  • View blog
  • Posts: 215
  • Joined: 16-September 09

Winsock Connecting via URL

Posted 02 February 2013 - 03:49 AM

Hey guys,

I'm trying to fetch a .txt file from a website and have set up a socket to do so. The socket works perfectly and I can connect to 99% of websites. However, the specific site I need to connect to is hosted on some kind of CDN (CloudFlare) that doesn't allow me to connect using an IP address. I need to use the URL. I know this because I:

1) Tried to connect with the IP in my browser and it doesn't work
2) Wrote a Perl script that uses the URL (in an LWP browser) and it does work.

So my question is then, how can I resolve this problem?

This is my socket application:

Connector.cpp
#include "Connector.h"
#include "stdio.h"
#include <string>

#define DEFAULT_BUFLEN 512
#define DEFAULT_PORT "80"

using namespace std;

int iResult;

void Connector::init()
{
	WSADATA wsaData;

	iResult = WSAStartup(MAKEWORD(2,2), &wsaData);

	if (iResult != 0) {
		printf("WSAStartup failed: %d\n", iResult);
		return;
	}

	return;
}

SOCKET Connector::connectSocket(string address)
{
	struct addrinfo *result = NULL,
					*ptr = NULL,
					hints;
	SOCKET ConnectSocket = INVALID_SOCKET;

	ZeroMemory(&hints, sizeof(hints));
	hints.ai_family = AF_UNSPEC;
	hints.ai_socktype = SOCK_STREAM;
	hints.ai_protocol = IPPROTO_TCP;

	// Resolve server address and port
	iResult = getaddrinfo(address.c_str(), DEFAULT_PORT, &hints, &result);
	if (iResult != 0) {
		printf("getaddrinfo failed with error: %d\n", iResult);
		WSACleanup();
		return INVALID_SOCKET;
	}

	// connect to address
	for (ptr=result; ptr != NULL; ptr=ptr->ai_next) {

		// Create socket for connecting to server
		ConnectSocket = socket(ptr->ai_family, ptr->ai_socktype, ptr->ai_protocol);
		if (ConnectSocket == INVALID_SOCKET) {
			printf("socket failed with error: %ld\n", WSAGetLastError());
            WSACleanup();
			return INVALID_SOCKET;
		}

		// Connect to Server
		iResult = connect(ConnectSocket, ptr->ai_addr, (int)ptr->ai_addrlen);
		if (iResult == SOCKET_ERROR) {
            closesocket(ConnectSocket);
            ConnectSocket = INVALID_SOCKET;
            continue;
        }
        break;
	}

	freeaddrinfo(result);

	if (ConnectSocket == INVALID_SOCKET) {
		printf("Unable to connect to server!\n");;
		WSACleanup();
		return INVALID_SOCKET;
	}

	return ConnectSocket;
}

string Connector::sendRequest(string address, string request)
{
	SOCKET sock = connectSocket(address);
	string response = "";
	int recvbuflen = DEFAULT_BUFLEN;
	char recvbuf[DEFAULT_BUFLEN];

	// Exit the function if the socket failed to connect
	if (sock == INVALID_SOCKET)
		return "ERROR: COULD NOT CONNECT SOCKET";

	// Send a request to the server; Print error if relevant.
	iResult = send(sock, request.c_str(), request.length(), 0);
	if (iResult == SOCKET_ERROR) {
		printf("send failed: %d\n", WSAGetLastError());
		closesocket(sock);
		WSACleanup();
		return "ERROR: COULD NOT SEND REQUEST";
	}

	// Shut down the send element of the socket. We can still receive.
	iResult = shutdown(sock, SD_SEND);
	if (iResult == SOCKET_ERROR) {
		printf("shutdown failed: %d\n", WSAGetLastError());
		closesocket(sock);
		WSACleanup();
		return "ERROR: COULD NOT SHUT DOWN SOCKET";
	}

	// Receive response from server
	do {
		iResult = recv(sock, recvbuf, recvbuflen, 0);
		if (iResult > 0)
			response+= string(recvbuf).substr(0,recvbuflen);
		else if (iResult == 0)
			printf("Connection closed\n");
		else
			printf("recv failed: %d\n", WSAGetLastError());
	} while (iResult > 0);

	// Cleanup
    closesocket(sock);
    WSACleanup();

	return response;
}




Main.cpp
#include "stdio.h"
#include "Connector.h"



using namespace std;

int main()
{
	Connector conn;
	string request;
	string response;

	request += "GET / HTTP/1.0\r\n";
	request += "Host: google.co.uk\r\n";
	request += "\r\n";

	conn.init();
	response = conn.sendRequest("173.194.75.94",request);
	printf(response.c_str());
	system("PAUSE");
}



This is my first project involving Winsock, so any help is greatly appreciated!

Thanks,

Cbeppe.

Is This A Good Question/Topic? 0
  • +

Replies To: Winsock Connecting via URL

#2 undefined behaviour  Icon User is offline

  • New D.I.C Head

Reputation: 7
  • View blog
  • Posts: 36
  • Joined: 17-January 13

Re: Winsock Connecting via URL

Posted 03 February 2013 - 07:08 AM

Which problem are you referring to?

stdio.h? You mean cstdio (without the .h), right?

Are you sure the "Host:" header is defined in HTTP/1.0? I thought that came about in 1.1.

This is dangerous:
printf(response.c_str());


I suggest this, instead:
std::cout << response;


system("PAUSE"); is probably better substituted for something like this pause():
void pause() { std::cin.putback(std::cin.get()); }


Lines 52 through to 54 in Connector.cpp seem silly. You have a list of addresses to try. Why is it that your code gives up immediately after the first failure? I suggest something more along the lines of:
if (ConnectSocket == SOCKET_ERROR) { continue; }


Line 111 in Connector.cpp also seems silly. If iResult is 1, indicating that only 1 byte was recieved, then why is your code copying recvbuflen bytes?

This post has been edited by undefined behaviour: 03 February 2013 - 07:11 AM

Was This Post Helpful? 1
  • +
  • -

#3 Cbeppe  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 31
  • View blog
  • Posts: 215
  • Joined: 16-September 09

Re: Winsock Connecting via URL

Posted 03 February 2013 - 08:47 AM

Hey, thanks for looking into this!

Quote

Which problem are you referring to?

I'm trying to use a socket to fetch a .txt file that is hosted on what appears to be a CloudFlare CDN. Unfortunately, I'm not allowed to connect to the site directly using an IP address. Apparently, this is a security precaution by the CDN. My problem is then: How would I connect to that site using the URL rather than the IP of the server using my Windows Socket?

As a side-note, the socket connection works perfectly and I'm able to connect to other websites.

Quote

This is dangerous: ...

Thanks! I atually thought that printf() was preferred. I'm still trying to get a grip on C++.

Quote

system("PAUSE"); is probably better substituted for something like this pause():

I know :) I threw it in quickly because I got excited that my socket worked.

Quote

Lines 52 through to 54 in Connector.cpp seem silly. You have a list of addresses to try. Why is it that your code gives up immediately after the first failure? I suggest something more along the lines of ...

Thanks for that. I wrote that class while looking at two different tutorials on Winsock, so the code is probably a bigger mess than I can imagine.

Quote

Line 111 in Connector.cpp also seems silly. If iResult is 1, indicating that only 1 byte was recieved, then why is your code copying recvbuflen bytes?

You're completely right. It should of course only copy the number of bytes received.

Again, thanks for looking into this. All help is greatly appreciated. If you know anything about the issue I mentioned above, I'd be grateful if you'd share.

Cbeppe.
Was This Post Helpful? 0
  • +
  • -

#4 snoopy11  Icon User is offline

  • Engineering ● Software
  • member icon

Reputation: 770
  • View blog
  • Posts: 2,249
  • Joined: 20-March 10

Re: Winsock Connecting via URL

Posted 03 February 2013 - 09:09 AM

Hi,



I would recommend using libcurl or something like it

it would make the task so much easier than trying to implement HTTPS communication yourself.


Snoopy.
Was This Post Helpful? 1
  • +
  • -

#5 Cbeppe  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 31
  • View blog
  • Posts: 215
  • Joined: 16-September 09

Re: Winsock Connecting via URL

Posted 03 February 2013 - 09:18 AM

Hey Snoopy,

I'm aware of that, but it's more of a learning project than anything else. I've been curious about sockets for some time now. I'll probably end up using libcurl.

However, there is absolutely no HTTPS involved. It's not an encrypted transfer, just plain HTTP, but apparently it has to go via the nameserver rather than the host IP address...

Thanks for the suggestion though :)
Was This Post Helpful? 0
  • +
  • -

#6 snoopy11  Icon User is offline

  • Engineering ● Software
  • member icon

Reputation: 770
  • View blog
  • Posts: 2,249
  • Joined: 20-March 10

Re: Winsock Connecting via URL

Posted 03 February 2013 - 09:40 AM

Well if its more a learning experience in HTTP

then

RFC2616

and

RFC 2617

are the relevant RFC's

Snoopy.
Was This Post Helpful? 1
  • +
  • -

#7 GunnerInc  Icon User is offline

  • "Hurry up and wait"
  • member icon




Reputation: 858
  • View blog
  • Posts: 2,279
  • Joined: 28-March 11

Re: Winsock Connecting via URL

Posted 03 February 2013 - 09:47 AM

You cannot just connect to an IP address, that is not how network communication works. When you type an IP address in a browser, the browser converts that to a network address.

Take the IP string and pass that to inet_addr to convert the string IP to an actual IP address, then pass the returned value to gethostbyaddr which will return a pointer to a hostent structure filled with the correct info for you to use with connect.

What I do is this:
inet_addr
if return != INADDR_NONE
call gethostbyaddr
else
call gethostbyname

Or for Async (Sorry Assembly)
    invoke  inet_addr, esi
    .if eax == INADDR_NONE
        invoke  WSAAsyncGetHostByName, hMain, WM_RESOLVE_SERVER, esi, offset lpBufHostent, MAXGETHOSTSTRUCT 
    .else
        mov     dwIP, eax
        invoke  WSAAsyncGetHostByAddr, hMain, WM_RESOLVE_SERVER, addr dwIP, 4, AF_INET, offset lpBufHostent, MAXGETHOSTSTRUCT
    .endif

Was This Post Helpful? 1
  • +
  • -

#8 Cbeppe  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 31
  • View blog
  • Posts: 215
  • Joined: 16-September 09

Re: Winsock Connecting via URL

Posted 03 February 2013 - 10:39 AM

Hi GunnerInc,

Thanks for the advice, I'm trying your method right now and have the following code in my connectSocket() method:

SOCKET Connector::connectSocket(string address)
{
	SOCKET ConnectSocket = INVALID_SOCKET;
	struct hostent *remoteHost;
	struct in_addr addr = { 0 };
	
	// Resolve server address and port
	addr.s_addr = inet_addr(address.c_str());
        if (addr.s_addr == INADDR_NONE) {
            printf("The IPv4 address entered must be a legal address\n");
            return 1;
        } else
            remoteHost = gethostbyaddr((char *) &addr, 4, AF_INET);

	// Create socket for connecting to server
	ConnectSocket = socket(AF_UNSPEC, SOCK_STREAM, IPPROTO_TCP);
	if (ConnectSocket == INVALID_SOCKET) {
		printf("socket failed with error: %ld\n", WSAGetLastError());
        WSACleanup();
		return INVALID_SOCKET;
	}

	// Connect to Server
        // THIS IS WHERE I GET AN ERROR
	iResult = connect(ConnectSocket, (SOCKADDR *)remoteHost->h_addr_list[0], (int)remoteHost->h_length);
	if (iResult == SOCKET_ERROR) {
		cout << remoteHost->h_addr_list[0] << endl;
		cout << "Error Code:" << WSAGetLastError() << endl;
        closesocket(ConnectSocket);
        ConnectSocket = INVALID_SOCKET;
        return INVALID_SOCKET;
    }

	if (ConnectSocket == INVALID_SOCKET) {
		printf("Unable to connect to server!\n");;
		WSACleanup();
		return INVALID_SOCKET;
	}

	return ConnectSocket;
}



When I call connect() (line 25), I get the error code 10014 which means:

Quote

Bad address.

The system detected an invalid pointer address in attempting to use a pointer argument of a call. This error occurs if an application passes an invalid pointer value, or if the length of the buffer is too small. For instance, if the length of an argument, which is a sockaddr structure, is smaller than the sizeof(sockaddr).


Can anyone see where I'm going wrong?

Thanks!
Was This Post Helpful? 0
  • +
  • -

#9 snoopy11  Icon User is offline

  • Engineering ● Software
  • member icon

Reputation: 770
  • View blog
  • Posts: 2,249
  • Joined: 20-March 10

Re: Winsock Connecting via URL

Posted 03 February 2013 - 01:49 PM

hi,

10014 means the address is not valid, probably you have spelt it incorrectly.

it should be passed in as www.yoursite.com

Snoopy.
Was This Post Helpful? 0
  • +
  • -

#10 JackOfAllTrades  Icon User is offline

  • Saucy!
  • member icon

Reputation: 6058
  • View blog
  • Posts: 23,496
  • Joined: 23-August 08

Re: Winsock Connecting via URL

Posted 03 February 2013 - 02:21 PM

Actually, it's that one of the arguments passed to the function is invalid (NULL?). Where there is no check of the returned value from the gethostbyaddr function, and it's being passed, that's the only pointer (address) value being passed to the function, I think there is where you should start looking.
Was This Post Helpful? 0
  • +
  • -

#11 Cbeppe  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 31
  • View blog
  • Posts: 215
  • Joined: 16-September 09

Re: Winsock Connecting via URL

Posted 03 February 2013 - 02:32 PM

Hey Snoopy,

I'm sorry, but unless I'm missing something, I don't think you're right about that. The only address variable that I pass is the argument in the SOCKET Connector::connectSocket(string address).

This variable is then used as an argument in inet_addr(address.c_str()) and inet_addr() requires its argument to be a valid IPv4 in the format of "255.255.255.255". Passing "www.example.com" gives an error.

The result of the call to inet_addr() is then used as the first argument in the call to gethostbyaddr() in line 13 (remoteHost = gethostbyaddr((char *) &addr, 4, AF_INET);).

This then gives me a HOSTENT object that I store in remoteHost.

I then pass the information combined in that object to the Winsock connect() method in line 25 (shown below). This is where I get the error saying that the address is badly formed. I am at a loss, because that indicates that the bad address results from the call to gethostbyaddr().
iResult = connect(ConnectSocket, (SOCKADDR *)remoteHost->h_addr_list[0], (int)remoteHost->h_length);

Was This Post Helpful? 0
  • +
  • -

#12 Cbeppe  Icon User is offline

  • D.I.C Head
  • member icon

Reputation: 31
  • View blog
  • Posts: 215
  • Joined: 16-September 09

Re: Winsock Connecting via URL

Posted 03 February 2013 - 02:41 PM

@JackOfAllTrades:

I tried printing the variables passed to the connect() function like so:
iResult = connect(ConnectSocket, (SOCKADDR *)remoteHost->h_addr_list[0], (int)remoteHost->h_length);
if (iResult == SOCKET_ERROR) {
	cout << remoteHost->h_addr_list[0] << endl;
	cout << "SOCKADDR: " << (SOCKADDR*)remoteHost->h_addr_list[0] << endl;
	cout << "Length: " << (int)remoteHost->h_length << endl;
	cout << "Error Code: " << WSAGetLastError() << endl;
        closesocket(ConnectSocket);
        ConnectSocket = INVALID_SOCKET;
        return INVALID_SOCKET;
}



and I get the following result:

Quote

<ecf-173-xxx-xx-xxx.cloudflare.com
SOCKADDR: 00305704
Length: 4
Error Code: 10014


That doesn't seem right to me. The first line is what is stored in the HOSTENT object and seems fine, since it's an IPv4 address, but the other lines seem wrong. The SOCKADDR is what is actually passed to the connect() function and the Length of either of those is most definitely not 4 bytes.

Can you see what is going on?

Thanks :)
Was This Post Helpful? 0
  • +
  • -

#13 JackOfAllTrades  Icon User is offline

  • Saucy!
  • member icon

Reputation: 6058
  • View blog
  • Posts: 23,496
  • Joined: 23-August 08

Re: Winsock Connecting via URL

Posted 03 February 2013 - 04:15 PM

Perhaps reading Beej might help?
Was This Post Helpful? 0
  • +
  • -

#14 undefined behaviour  Icon User is offline

  • New D.I.C Head

Reputation: 7
  • View blog
  • Posts: 36
  • Joined: 17-January 13

Re: Winsock Connecting via URL

Posted 04 February 2013 - 05:05 PM

View PostGunnerInc, on 03 February 2013 - 09:47 AM, said:

Take the IP string and pass that to inet_addr to convert the string IP to an actual IP address, then pass the returned value to gethostbyaddr which will return a pointer to a hostent structure filled with the correct info for you to use with connect.


The getaddrinfo method used in the original post is sufficient to handle parsing of IPV4, IPV6 and DNS resolution...

Quote

Quote

Lines 52 through to 54 in Connector.cpp seem silly. You have a list of addresses to try. Why is it that your code gives up immediately after the first failure? I suggest something more along the lines of ...

Thanks for that. I wrote that class while looking at two different tutorials on Winsock, so the code is probably a bigger mess than I can imagine.


This is an issue that could result in your code failing to connect to otherwise reachable servers. It could be relevant to your problem, and if it isn't then it should be fixed anyway.

I spent a lot of time auditing your code because you asked for help. Have you fixed ANY of the issues that I raised? Some of them would allow a malicious website to compromise your system! Are you still having problems? Please describe. If you won't help me to help you, then I won't bother helping you.
Was This Post Helpful? 0
  • +
  • -

#15 snoopy11  Icon User is offline

  • Engineering ● Software
  • member icon

Reputation: 770
  • View blog
  • Posts: 2,249
  • Joined: 20-March 10

Re: Winsock Connecting via URL

Posted 04 February 2013 - 08:48 PM

hmm,

Why not look at this msdn code...

#undef UNICODE

#include <winsock2.h>
#include <ws2tcpip.h>
#include <stdio.h>
#include <conio.h>
// link with Ws2_32.lib
#pragma comment (lib, "Ws2_32.lib")

int  main()
{

    //-----------------------------------------
    // Declare and initialize variables
    WSADATA wsaData;
    int iResult;
    INT iRetval;

    DWORD dwRetval;

    int i = 1;
    
    struct addrinfo *result = NULL;
    struct addrinfo *ptr = NULL;
    struct addrinfo hints;

    struct sockaddr_in  *sockaddr_ipv4;
//    struct sockaddr_in6 *sockaddr_ipv6;
    LPSOCKADDR sockaddr_ip;

    char ipstringbuffer[46];
    DWORD ipbufferlength = 46;

    char* location = "www.google.com";
    char* port ="0";

    // Initialize Winsock
    iResult = WSAStartup(MAKEWORD(2, 2), &wsaData);
    if (iResult != 0) {
        printf("WSAStartup failed: %d\n", iResult);
        return 1;
    }

    //--------------------------------
    // Setup the hints address info structure
    // which is passed to the getaddrinfo() function
    ZeroMemory( &hints, sizeof(hints) );
    hints.ai_family = AF_UNSPEC;
    hints.ai_socktype = SOCK_STREAM;
    hints.ai_protocol = IPPROTO_TCP;

    printf("Calling getaddrinfo with following parameters:\n");
    printf("\tnodename = %s\n", location);
    printf("\tservname (or port) = %s\n\n", port);
    
//--------------------------------
// Call getaddrinfo(). If the call succeeds,
// the result variable will hold a linked list
// of addrinfo structures containing response
// information
    dwRetval = getaddrinfo(location, port, &hints, &result);
    if ( dwRetval != 0 ) {
        printf("getaddrinfo failed with error: %d\n", dwRetval);
        WSACleanup();
        return 1;
    }

    printf("getaddrinfo returned success\n");
    
    // Retrieve each address and print out the hex bytes
    for(ptr=result; ptr != NULL ;ptr=ptr->ai_next) {

        printf("getaddrinfo response %d\n", i++);
        printf("\tFlags: 0x%x\n", ptr->ai_flags);
        printf("\tFamily: ");
        switch (ptr->ai_family) {
            case AF_UNSPEC:
                printf("Unspecified\n");
                break;
            case AF_INET:
                printf("AF_INET (IPv4)\n");
                sockaddr_ipv4 = (struct sockaddr_in *) ptr->ai_addr;
                printf("\tIPv4 address %s\n",
                    inet_ntoa(sockaddr_ipv4->sin_addr) );
                break;
            case AF_INET6:
                printf("AF_INET6 (IPv6)\n");
                // the InetNtop function is available on Windows Vista and later
                // sockaddr_ipv6 = (struct sockaddr_in6 *) ptr->ai_addr;
                // printf("\tIPv6 address %s\n",
                //    InetNtop(AF_INET6, &sockaddr_ipv6->sin6_addr, ipstringbuffer, 46) );
                
                // We use WSAAddressToString since it is supported on Windows XP and later
                sockaddr_ip = (LPSOCKADDR) ptr->ai_addr;
                // The buffer length is changed by each call to WSAAddresstoString
                // So we need to set it for each iteration through the loop for safety
                ipbufferlength = 46;
                iRetval = WSAAddressToString(sockaddr_ip, (DWORD) ptr->ai_addrlen, NULL, 
                    ipstringbuffer, &ipbufferlength );
                if (iRetval)
                    printf("WSAAddressToString failed with %u\n", WSAGetLastError() );
                else    
                    printf("\tIPv6 address %s\n", ipstringbuffer);
                break;
            case AF_NETBIOS:
                printf("AF_NETBIOS (NetBIOS)\n");
                break;
            default:
                printf("Other %ld\n", ptr->ai_family);
                break;
        }
        printf("\tSocket type: ");
        switch (ptr->ai_socktype) {
            case 0:
                printf("Unspecified\n");
                break;
            case SOCK_STREAM:
                printf("SOCK_STREAM (stream)\n");
                break;
            case SOCK_DGRAM:
                printf("SOCK_DGRAM (datagram) \n");
                break;
            case SOCK_RAW:
                printf("SOCK_RAW (raw) \n");
                break;
            case SOCK_RDM:
                printf("SOCK_RDM (reliable message datagram)\n");
                break;
            case SOCK_SEQPACKET:
                printf("SOCK_SEQPACKET (pseudo-stream packet)\n");
                break;
            default:
                printf("Other %ld\n", ptr->ai_socktype);
                break;
        }
        printf("\tProtocol: ");
        switch (ptr->ai_protocol) {
            case 0:
                printf("Unspecified\n");
                break;
            case IPPROTO_TCP:
                printf("IPPROTO_TCP (TCP)\n");
                break;
            case IPPROTO_UDP:
                printf("IPPROTO_UDP (UDP) \n");
                break;
            default:
                printf("Other %ld\n", ptr->ai_protocol);
                break;
        }
        printf("\tLength of this sockaddr: %d\n", ptr->ai_addrlen);
        printf("\tCanonical name: %s\n", ptr->ai_canonname);
    }

    freeaddrinfo(result);
    WSACleanup();
	_getch();
    return 0;
}



It acutually does connect via the URL.

Snoopy.
Was This Post Helpful? 1
  • +
  • -

Page 1 of 1