[kevin_911]

Hi Guys need a bit of assistance. Basically I am in the middle of designing a GUi for a console application
which is a forum crawler written in C++ by using various libraries like libcurl and libxml2. Now this application
was designed by a different person who has left our team. I have carried out all the neccessary adjustments for the
application to run smoothly but to no avail

The application runs by connecting to SQL server db whereby reading the forum and thread url request by the user.
It then runs the crawl through the forum and eventually sends the information back to the db.

The error I am getting is a 'Debug Assertion Failed' which pops up during execution as shown in the attached image. According to the author the system works 100%

Please advice!

Edit: Also the table where it takes the information from for the location of thread includes, forum name, id, url and threadregex and forumregex. threadregex and forum regex has regular expressions for identification of the thread and forum location.

Thanks

Attached image(s)

• 

This post has been edited by kevin_911: 04 July 2011 - 11:12 AM

[gabehabe]

Without seeing the code it's hard to judge. I can, however, offer some insight:

Your error is "list iterator is not dereferencable." There are a number of examples that demonstrate that the issue is caused by way of attempting to access a part of a list via an iterator, but technically there is nothing to iterate.

Couple this with the fact that I notice your program is reporting numerous parse errors, I suspect that the application isn't populating the list correctly, then attempting to iterate through the empty list.

[jimblumberg]

Without seeing the code that is causing the error it is hard to see what is going on.
Have you run the program through your debugger it should show where the error is detected.

Jim

[kevin_911]

gabehabe, on 04 July 2011 - 12:22 PM, said:

Without seeing the code it's hard to judge. I can, however, offer some insight:

Your error is "list iterator is not dereferencable." There are a number of examples that demonstrate that the issue is caused by way of attempting to access a part of a list via an iterator, but technically there is nothing to iterate.

Couple this with the fact that I notice your program is reporting numerous parse errors, I suspect that the application isn't populating the list correctly, then attempting to iterate through the empty list.

jimblumberg, on 04 July 2011 - 12:25 PM, said:

Without seeing the code that is causing the error it is hard to see what is going on.
Have you run the program through your debugger it should show where the error is detected.

Jim

Thanks guys for your instant reply. The program has about 7 header and source files and I am not exactly sure where the error is, so dont know which source file to attach?!?

In the above image if I click 'Retry' it opens a new window where it says 'autocrawl.exe has triggered a breakpoint' and opens stdthrow.cpp.

Ok here is the main file which I think is executed first.

The main table in the db is 'forums' which in my previous post mentioned contains forum name, id, url, threadregex and forum regex:

This is the info I feed into the table for the exe to pick up.

Quote

Forum name: boards.ie
id: 1
url: http://www.boards.ie/vbulletin/
threadRegex: showthread\.php\?p=([0-9]+)
forumRegex: forumdisplay\.php\?f=([0-9]+)

// autocrawler.cpp : Defines the entry point for the console application.
//
#define OTL_ODBC_MSSQL_2008

#include "stdafx.h"
#include <curl/curl.h>
#include <string>
#include <iostream>
#include "html_downloader.h"
#include <libxml/HTMLparser.h>
#include <libxml/tree.h>
#include <list>
#include <regex>
#include <sstream>
#include "pathcount.h"
#include "crawler.h"
#include "otlv4.h"
#include <stdio.h>
#include "helper.h"

#ifdef LIBXML_TREE_ENABLED

using namespace std;


otl_connect db; // connect object

void insertChs(list<int> chs, int id)
{
	list<int> temp;
	temp.assign(chs.begin(), chs.end());
	int index = 1;

	otl_stream i(200, // buffer size
		"INSERT INTO forumpaths VALUES( "
		":f<int>, :f1<int>, :f2<int>)",
		// SELECT statement
		db // connect object
		); 
	// create select stream
	int tempint;
	while(temp.size() > 0)
	{
		tempint = temp.front();
		temp.pop_front();
		i<<index<<tempint<<id;
		index++;
	}
}


list<int> performAnalysis(int id)
{
	list<int> retVal;

	otl_stream i(200, // buffer size
		"select threadUrl from trainingthreads "
		"where fk_forumId>=:f<int> ",
		// SELECT statement
		db // connect object
		); 
	// create select stream
	i<<id;
	
	char tmpurl[200];
	list<list<int>> tempchs;
	list<int> tempint;
	crawler crawl(0, "", "", "", retVal);

	while(!i.eof()){ // while not end-of-data
		i >> tmpurl;
		tempint = crawl.analyse(tmpurl);
		if(tempint.size() > 1)
			tempchs.push_back(tempint);
		//retVal.push_back(tmp);
	}

	retVal = helper::subList(tempchs);

	return retVal;
}

list<int> getCh(int id)
{
	list<int> retVal;

	otl_stream i(200, // buffer size
		"select nodeindex from forumpaths "
		"where fk_forumId>=:f<int> "
		"ORDER BY [order] ASC",
		// SELECT statement
		db // connect object
		); 
	// create select stream
	i<<id;

	int tmp;
	while(!i.eof()){ // while not end-of-data
		i >> tmp;
		retVal.push_back(tmp);
	}

	if(retVal.size() == 0)
	{
		retVal = performAnalysis(id);
		insertChs(retVal,id);
	}

	return retVal;
}


void select()
{ 
	otl_stream i(200, // buffer size
		"select * from forums",
		db // connect object
		); 
	char f1[200];
	int f2;
	char f3[200];
	char forumRegex[200];
	char threadRegex[200];
	list<int> ch;

	while(!i.eof()){ // while not end-of-data
		i>>f1>>f2>>f3>>threadRegex>>forumRegex;
		cout<<"f1="<<f1<<", f2="<<f2<<", f3="<<f3<<endl;
		
		ch = getCh(f2);
		
		crawler c(f2, f3, threadRegex, forumRegex, ch);
		c.crawl();
	}
}

int _tmain(int argc, _TCHAR* argv[])
{
	otl_connect::otl_initialize(); // initialize ODBC environment
	try{
		db.rlogon(helper::getDbConnString().c_str());
		select();
	}
	catch(otl_exception& p){ // intercept OTL exceptions
		cerr<<p.msg<<endl; // print out error message
		cerr<<p.code<<endl; // print out error code
		cerr<<p.var_info<<endl; // print out the variable that caused the error
		cerr<<p.sqlstate<<endl; // print out SQLSTATE message
		cerr<<p.stm_text<<endl; // print out SQL that caused the error
	}
	db.logoff(); // disconnect from the data source

	string result;
	cin >> result;
				
	return 0;
}
#else
int main(void) {
	fprintf(stderr, "Tree support not compiled in\n");
	exit(1);
}
#endif

This is the second source file which fetches the data from the forum:

#include "htmlParser.h"
#include "helper.h"
#include <iostream>
#define OTL_ODBC_MSSQL_2008

#include "stdafx.h"
#include <curl/curl.h>
#include <string>
#include <iostream>
#include "html_downloader.h"
#include <libxml/HTMLparser.h>
#include <libxml/tree.h>
#include <list>
#include <regex>
#include <sstream>
#include "pathcount.h"
#include "crawler.h"
#include "otlv4.h"
#include <stdio.h>

int crawler::sel(string url)
{ 
	otl_stream o(1, // buffer size should be equal to 1 in case of stored procedure call
		"{call GetThreadId("
		" :forumId<int,in>, "
		" :B<char[200],in>, "
		" :C<int,inout> "
		")}",
		// stored procedure call
		db // connect object
		);
	
	o<<id<<url.c_str()<<0;

	int a;

	o>>a;
	cout<<"A="<<a<<endl;
	return a;
}

crawler::crawler(int i, std::string url, std::string threadRx, std::string forumRx, list<int> ch)
{
	//add url to list, depending on regex
	forumurls.push_back(url);
	threadRegex = threadRx;
	forumRegex = forumRx;
	children = ch;
	urlExp = "[A-Za-z]+://>/[A-Za-z.]+";
	id = i;	
}

crawler::~crawler(void)
{
}

list<int> crawler::min_analyse(string url, list<int> ch)
{
	html_downloader h(url);
	string result = h.getHtml();

	htmlParser hp(result, ch, true);

	return hp.min_analyse();
}

list<int> crawler::analyse(string url)
{
	html_downloader h(url);
	string result = h.getHtml();

	list<int> ch;

	htmlParser hp(result, ch, true);

	ch = hp.analyse();

	return ch;
}

void crawler::fetch(string url, bool thread)
{
	html_downloader h(url);
	string result = h.getHtml();

	htmlParser hp(result, children, thread);

	int threadId = 0;

	if(thread)
	{
		otl_connect::otl_initialize(); // initialize ODBC environment
		try{
			db.rlogon(helper::getDbConnString().c_str());
			//stored_proc(); // invoking stored procedure
			threadId = sel(url);
		}
		catch(otl_exception& p){ // intercept OTL exceptions
			cerr<<p.msg<<endl; // print out error message
			cerr<<p.code<<endl; // print out error code
			cerr<<p.var_info<<endl; // print out the variable that caused the error
			cerr<<p.sqlstate<<endl; // print out SQLSTATE message
			cerr<<p.stm_text<<endl; // print out SQL that caused the error
		}
		db.logoff(); // disconnect from the data source
	}


	list<xmlChar*> urls = hp.parseHtml(threadId);

	list<xmlChar*>::iterator i;

	list<string> absurls;
	string domain = helper::firstMatch(url, urlExp);

	int qz = 0;

	for(i=urls.begin(); i != urls.end(); ++i) 
	{
		string temp = (string)(char*)*i;

		if(!helper::contains(temp, urlExp))
		{
			if(temp.size() == 0)
			{
				temp = domain;
			}
			else
			{
				if(temp[0]=='/')
					temp = domain + temp;
				else
					temp = domain + '/' + temp;
			}
		}
		//disabling crawling
		absurls.push_back(temp);
	}

	list<string>::iterator s;
	list<string> localurls;

	for(s=absurls.begin(); s != absurls.end(); ++s)
		if(domain.compare(((string)*s).substr(0,domain.length()))==0)
			localurls.push_back(*s);

	for(s=localurls.begin(); s != localurls.end(); ++s)
	{	
		if(helper::contains(*s, threadRegex))
		{
			if(!helper::inStrList(threadurls, *s))
			{
				if(!helper::inStrList(visited, *s))
				{
					threadurls.push_back(*s);
				}
			}
		}
		else if(helper::contains(*s, forumRegex))
		{
			if(!helper::inStrList(forumurls, *s))
			{
				if(!helper::inStrList(visited, *s))
				{
					forumurls.push_back(*s);
				}
			}
		}
	}

	visited.push_back(url);
}

void crawler::crawl(void)
{
	while((forumurls.size() > 0 || threadurls.size() > 0) && visited.size() < 10000)
	{
		if(forumurls.size() > 0)
		{	
			string t = forumurls.front();
			forumurls.pop_front();

			crawler::fetch(t, false);
		}
		if(threadurls.size() > 0)
		{	
			string t = threadurls.front();
			threadurls.pop_front();

			crawler::fetch(t, true);
		}
	}
}

This post has been edited by kevin_911: 04 July 2011 - 12:27 PM

[Oler1s]

> In the above image if I click 'Retry' it opens a new window where it says 'autocrawl.exe has triggered a breakpoint' and opens stdthrow.cpp.

Walk up the callstack please. It should be on the bottom right hand side of the screen or somewhere in the bottom where you can click up the callstack. Go up until you find your code.

[kevin_911]

Oler1s, on 04 July 2011 - 01:51 PM, said:

> In the above image if I click 'Retry' it opens a new window where it says 'autocrawl.exe has triggered a breakpoint' and opens stdthrow.cpp.

Walk up the callstack please. It should be on the bottom right hand side of the screen or somewhere in the bottom where you can click up the callstack. Go up until you find your code.

Hi 'Oler1s' image attached below shows the result of my call stack.

Attached image(s)

• 

[Oler1s]

Ok, so you have an image there, but so what?

You need to actually go up the callstack until you find your code. Looking at that list, it's helper::subList, right? So click on that item in the list to go to your code, and you can see the lines of code that are problematic.

Look, your problem is that your program is failing, and you have no idea where in the code to start looking. The debugger is telling you exactly where the error came from, so that's a starting point. That's what I'm telling you to do.

[kevin_911]

Oler1s, on 04 July 2011 - 03:19 PM, said:

Ok, so you have an image there, but so what?

You need to actually go up the callstack until you find your code. Looking at that list, it's helper::subList, right? So click on that item in the list to go to your code, and you can see the lines of code that are problematic.

Look, your problem is that your program is failing, and you have no idea where in the code to start looking. The debugger is telling you exactly where the error came from, so that's a starting point. That's what I'm telling you to do.

Sorry Oler1s for being a noob The author has told me the code works perfectly but on my PC it is failing so dunno who is wrong, if you know what I mean

I have been trying to understand the code but not able to pinpoint the problem. Dunno if it is my input into the db which wrong or the code?

But anyhow, yes helper.cpp is one of the function while used by the system. If clicked on it takes me to helper.cpp line number 29 as shown below.

The helper function, basically carries out the following:

1. GetLinkHref: This function aids in the extraction of a link from an anchor element in the HTML

2. SubList: Takes a list of lists of integers, and compares them to find the largest common subset of valuues that bein the list.

3. FirstMatch: Used to simplify the use of a regular expression by excepting a string of input, and a string defining the expression.

4. Contains: This is another function to simplify the use of regular expression.

5. InStringList: This function accepts a list of strings, and a string and will return true if the string is contained in the list and false if it isn't.

6. StringList: This function accepts a node, and for each child element in that node it concatenates the contents of that node and all its childern into a string, and adds the string to a list then returns the list.

7. StringRex: This function recursively converts the content of a node and all its childern into a string and return the string.

#include "helper.h"
#include <stdio.h>
#include <locale>
#include <regex>
#include <sstream>


helper::helper(void)
{
}


helper::~helper(void)
{
}
xmlChar* helper::getLinkHref(xmlNode *a_node)
{
	xmlAttr *cur_node = NULL;

	for(cur_node = a_node->properties; cur_node; cur_node = cur_node->next)
	{
		if(xmlStrEqual(cur_node->name, xmlCharStrdup("href"))==1 && cur_node->children != NULL)
			{
				return (cur_node->children->content);
			}
	}
}

list<int> helper::subList(list<list<int>> nums)
{
	list<int> temp1 = nums.front();
	nums.pop_front();
	list<int> retVal;
	retVal.assign(temp1.begin(), temp1.end());
	list<int> temp2;
	list<int> temp3;

	while(nums.size() > 0)
	{
		temp1 = nums.front();
		nums.pop_front();

		temp2.assign(retVal.begin(), retVal.end());

		temp3.clear();

		while(temp1.size()>0 && temp2.size() >0)
		{
			int a = temp1.front();
			temp1.pop_front();

			int b = temp2.front();
			temp2.pop_front();

			if(a==B)/>
			{
				temp3.push_back(a);
			}
			else
			{
				break;
			}
		}

		retVal.assign(temp3.begin(), temp3.end());
	}
	return retVal;
}

string helper::firstMatch(string input, string exp)
{
	std::tr1::cmatch res;
    std::tr1::regex rx(exp);
    std::tr1::regex_search(input.c_str(), res, rx);
	return res[0].str();
}

bool helper::contains(string input, string exp)
{
	std::tr1::regex rx(exp);
	return regex_search(input.begin(), input.end(), rx);
}

string helper::intToString(int a)
{
	std::stringstream ss;
	std::string s;
	ss<<a;
	s = ss.str();
	return s;
}

bool helper::inStrList(list<string> l, string s)
{
	list<string>::iterator i;

	for(i=l.begin(); i != l.end(); ++i)
		if(s.compare(*i)==0)
			return true;
	return false;
}

list<string> helper::stringList(xmlNode * a_node)
{
	xmlNode *cur_node = NULL;
	list<string> retVal;

	for (cur_node = a_node; cur_node; cur_node = cur_node->next) {
        if (cur_node->type == XML_ELEMENT_NODE) {
			if(cur_node->children != NULL)
			{
				string s = getStringRec(cur_node->children);

				retVal.push_back(s);
			}
        }
    }

	return retVal;
}

string helper::getStringRec(xmlNode * a_node)
{
	xmlNode *cur_node = NULL;

	string temp = "";

	for (cur_node = a_node; cur_node; cur_node = cur_node->next) {
        if (cur_node->type == XML_ELEMENT_NODE) {
			if(cur_node->children != NULL && cur_node->children->content != NULL)
			{
				std::stringstream ss;
				std::string s;
				ss<<cur_node->children->content;
				s = ss.str();
				char c;
				locale loc;

				bool nonws = false;
				int i;
				for(i=0;i<s.length();i++)
				{
					c=s[i];
					if (!isspace(c,loc))
					{
						nonws = true;
					}
				}

				if(nonws)
				{
					temp.append(s);
				}
			}			
			if(cur_node->children != NULL)
			{
				temp.append(getStringRec(cur_node->children));
			}
        }
    }

	return temp;
}

string helper::getDbConnString()
{
	return "uid=tester;pwd=******;dsn=crawldb";
}

This post has been edited by kevin_911: 04 July 2011 - 02:44 PM

[Karel-Lodewijk]

pop_front() would segfault if the list was empty, is it ?

[kevin_911]

Karel-Lodewijk, on 04 July 2011 - 04:03 PM, said:

pop_front() would segfault if the list was empty, is it ?

I am sorry but how I can check this list?

[Oler1s]

Quote

The author has told me the code works perfectly but on my PC it is failing so dunno who is wrong

The author is. If he's still responsible for making sure the program works, you have every right to demand it of him. "It worked on my computer" or "It worked before" are not acceptable excuses.

The function in question is subList, so you should look at that function for now, as opposed to other functions like contains, right?

If you look at the callstack, you can see that the error originates on a call to front(). In the subList function, I see the first line of code calling nums.front. This requires that the list be populated, as you can't get the first item of an empty list. Since there's no error checking here, it has be done elsewhere up stack at some point.

Looking at the callstack, sublist is called by performAnalysis, and you posted code for that. I don't see any code that checks the size of the list. Since that list is being created in performAnalysis, and neither performAnalysis, nor subList ensures that the list has any data. So like gabehabe guessed before, it's quite possible that the error is because the code is operating on an empty list, but assumes there is actually something.

You can verify this if you actually put up a debug point (I would put it in perform analysis). Step through with a debugger and see the behavior for yourself.

EDIT: You can inspect variables as you step through code. While stepping through, if you hover over a variable, or click it, or right click or something, you should be able to either get a popup with the value or add it to a watch list. I believe the list is by default on the bottom left of the screen.

This post has been edited by Oler1s: 04 July 2011 - 03:18 PM

[kevin_911]

Thanks guys for your help. I am going to forward this to the author somehow and see what
has to be said about it. Likewise I will update this thread accordingly.