CF8 & IIS Logs

Large text files make CF sad.

Page 1 of 1

3 Replies - 1056 Views - Last Post: 05 January 2011 - 01:48 PM Rate Topic: -----

#1 cyb1n  Icon User is offline

  • New D.I.C Head
  • member icon

Reputation: 3
  • View blog
  • Posts: 27
  • Joined: 08-May 09

CF8 & IIS Logs

Posted 05 January 2011 - 09:37 AM

Some time ago our department lost the use of our analytics engine. Since then we've been using Google Analytics, which is fine if you don't want to count the people who block the script from running. I've been trying to develop a solution using ColdFusion that will read and parse the server's IIS log files. Unfortunately, our IIS log files are anywhere between 400MB and 1GB for a month's worth of web traffic and ColdFusion runs out of memory before it can process enough. If anyone has any code tricks or server tweaks that might help me accomplish this task I'd really appreciate it.

Is This A Good Question/Topic? 0
  • +

Replies To: CF8 & IIS Logs

#2 Craig328  Icon User is offline

  • I make this look good
  • member icon

Reputation: 1866
  • View blog
  • Posts: 3,391
  • Joined: 13-January 08

Re: CF8 & IIS Logs

Posted 05 January 2011 - 10:01 AM

Well, first things first, IIS generates a log file (through it's default settings) once per day. If you're processing a month's worth of log files I'm guessing you're reading each individual log file and then looping over each line therein, right? You could build the CF process where it grabs just one log file at a time and does whatever it is you're wanting to do with it. Are you getting the out of memory error with just the one logfile?

Also, can you post the code you're using now and maybe we can come up with some suggestions? Finally, any chance you're using MySQL as a database backend? If so, the INFILE command for MySQL might be helpful.
Was This Post Helpful? 0
  • +
  • -

#3 cyb1n  Icon User is offline

  • New D.I.C Head
  • member icon

Reputation: 3
  • View blog
  • Posts: 27
  • Joined: 08-May 09

Re: CF8 & IIS Logs

Posted 05 January 2011 - 11:55 AM

My mistake, the log files are 400MB - 1GB for each day. I'm not sure what I was thinking when I typed that. I'm receiving the out of memory error when trying to read only a single log file. I'm actually using Oracle for the database backend so I'm not sure if there is an equivalent to INFILE or not.

As for the code I'm using, it's pretty basic. Just reads the file and loops through each line and placing it in an array:

<cfset logFile = expandPath('./ex101201.log') />
<cfset aLogFile = arrayNew(1) />

<cfloop file="#logFile#" index="idx">
	<cfif left(idx,1) NEQ '##'>
		<cfset arrayAppend(aLogFile,idx) />
	</cfif>
</cfloop>


Was This Post Helpful? 0
  • +
  • -

#4 Craig328  Icon User is offline

  • I make this look good
  • member icon

Reputation: 1866
  • View blog
  • Posts: 3,391
  • Joined: 13-January 08

Re: CF8 & IIS Logs

Posted 05 January 2011 - 01:48 PM

Egad! Those are huge files. Yeah, I can see how they'd choke the CF server when you try and read them into memory.

While I was typing this I IM'd a buddy of mine who's a web analytics guru and he suggested possibly using AWStats to crunch the backlog IIS files. AWStats can produce a flat file from that crunching which you should then be able to import into your database.

However, not everyone has AWStats (and configuring it can be a pain in the ass). Instead, I did a little digging around and found this: New File I/O in CF8. Give that a try and see if that helps any.
Was This Post Helpful? 1
  • +
  • -

Page 1 of 1