homepage Welcome to WebmasterWorld Guest from 54.226.43.155
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
HEAD checks
wilderness




msg:4443213
 1:13 am on Apr 20, 2012 (gmt 0)

Absurd as this may sound!

If an IP and/or entity does a HEAD check, wouldn't it be a safe assumption that they have a cache file on hand for comparison?

205.188.116.79 - - [20/Apr/2012:01:06:43 +0100] "HEAD

I know it's long been a practice of AOL, however their membership taint what it used to be.

 

MxAngel




msg:4443249
 3:38 am on Apr 20, 2012 (gmt 0)

Not sure, I've seen many HEAD requests from the usual Twitter swarm. For example the twitter or bit.ly bot uses HEAD to check whether the URL is valid or not.

HEAD does return the full headers without the body / content so it could be used indeed to check if a ressource has been modified or not based upon LastModified/ContentLength.

wilderness




msg:4443252
 3:56 am on Apr 20, 2012 (gmt 0)

But, but. . .why on earth would they desire to check if the page changed unless they held a cache copy for comparison?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved