6 Replies - 2258 Views - Last Post: 28 February 2013 - 08:53 AM Rate Topic: -----

#1 jjak  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 27-February 13

excessive recrawl of coldfusion dynamic pages

Posted 27 February 2013 - 01:43 PM

I am a google search appliance (gsa) admin, not a coldfusion programmer so please bear with me.
the gsa I admin has conent from multiple state agencies on it who use various methods to publish their content.
The folks who use coldfusion and serversideincludes are having issues with excessive recrawls on dynamic pages because there is no datelastmodfied set, which causes excessive server traffic. You can laugh if you want, but when I tell them the solution is setting a last modified date on the pages I get a universal huh? how do you do that? I opened a case with google originally and was told that yep, it's a page date problem. I have done a lot of research to try and find how to code this in the header and most of what I found talked about pulling a date from a page.
I did determine that it probably could be done using the CFHEADER tag. I'm just not sure about implementing.
Can I tell them that adding something like
<cfheader NAME="datelastmodified="Mon, 01 Feb 2013 08:00:00 GMT">

will suffice? Not sure about the date format, if the day name is required.
Have I tried just asking one of the webmasters to try this? No I haven't. I would like to know that I am at least on the right track before taking up too much of their time. And so far none of them have come up with a solution on their own other than useing robots.txt to block the crawl or things along those lines.
Any suggestions or thoughts would be appreciated.

jjak

Is This A Good Question/Topic? 0
  • +

Replies To: excessive recrawl of coldfusion dynamic pages

#2 jjak  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 27-February 13

Re: excessive recrawl of coldfusion dynamic pages

Posted 27 February 2013 - 01:57 PM

hmm, looks like I managed to mangle that simple code.
Should be
<cfheader NAME="datelastmodified" value="Mon, 01 Feb 2013 08:00:00 GMT"> 

Was This Post Helpful? 0
  • +
  • -

#3 Craig328  Icon User is offline

  • I make this look good
  • member icon

Reputation: 1926
  • View blog
  • Posts: 3,471
  • Joined: 13-January 08

Re: excessive recrawl of coldfusion dynamic pages

Posted 27 February 2013 - 02:44 PM

Welcome to DIC jjak!

Yes, you're on the right track with what you're doing. Google's crawlers do tend to respect the meta tag details and HTTP response values for pages they encounter and the way to set such in CF is indeed with the CFHEADER tag. You'll want to craft it to look something like this:

<CFHEADER NAME="Last-Modified" VALUE="#DateFormat(now (), 'ddd, dd mmm yyyy')# #TimeFormat(now(), 'HH:mm:ss')# GMT#gmt#">
<CFHEADER NAME="Expires" VALUE="Mon, 10 Mar 2013 05:00:00 GMT">



You will likely want a CF dev to do that work as I'm showing you two examples for the datetime value there. The first one dynamically sets it to right now (using the DateFormat() and Now() functions) and the second example sets the Expires header value with a hard coded date.

You'll probably want to include both the last-modified and expires tags and decide whether you want the dates applied to each to be either dynamic or hard coded.

Good luck!

This post has been edited by Craig328: 27 February 2013 - 02:45 PM
Reason for edit:: Fixed code example

Was This Post Helpful? 2
  • +
  • -

#4 jjak  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 27-February 13

Re: excessive recrawl of coldfusion dynamic pages

Posted 27 February 2013 - 03:39 PM

Thanks Craig328, I will give these a try. I want to make sure I understand the references here. If the last-modified tag was set to dynamic, wouldn't it always show the current time & date to the GSA, thus not resolving the issue? I guess maybe the Expires value could be dynamic so that when the link displayed in the GSA was clicked on it would use the current version of the page? I realize how difficult it must be not seeing examples of pages and code for the existing pages and I really appreciate your time and effort on this.
Was This Post Helpful? 0
  • +
  • -

#5 Craig328  Icon User is offline

  • I make this look good
  • member icon

Reputation: 1926
  • View blog
  • Posts: 3,471
  • Joined: 13-January 08

Re: excessive recrawl of coldfusion dynamic pages

Posted 27 February 2013 - 03:44 PM

View Postjjak, on 27 February 2013 - 05:39 PM, said:

If the last-modified tag was set to dynamic, wouldn't it always show the current time & date to the GSA, thus not resolving the issue?


Not necessarily. Notice we used the value that comes out of the Now() function. You can also pass in an explicit date value (like was done in the Expires example) or a mix of the two. That was why I said you'll probably want a CF dev to handle this. The DateFormat and TimeFormat functions will format whatever base info you pass in. In the Last-Modified example, we pass in whatever it is that Now() produces...but it can also be an explicit date...or a modification of the Now() product where, for instance, you can add 12 months to whatever the present date is (there are all kinds of CF date manipulation functions to do this with).

It can get kind of complicated looking so rather than guess at what it could look like, what would you want to set the Last-Modified value to be?
Was This Post Helpful? 2
  • +
  • -

#6 jjak  Icon User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 4
  • Joined: 27-February 13

Re: excessive recrawl of coldfusion dynamic pages

Posted 27 February 2013 - 04:03 PM

Thanks again. I have passed your examples and comments onto one of the coldfusion webmasters (I hope that's OK) to look at. It may take several days to get any type of verification done but I will post back at some point.
Was This Post Helpful? 0
  • +
  • -

#7 Craig328  Icon User is offline

  • I make this look good
  • member icon

Reputation: 1926
  • View blog
  • Posts: 3,471
  • Joined: 13-January 08

Re: excessive recrawl of coldfusion dynamic pages

Posted 28 February 2013 - 08:53 AM

Sounds good. Feel free to post back here if they can't get you set up.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1