homepage Welcome to WebmasterWorld Guest from 54.204.94.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
MSIECrawler
zerillos

5+ Year Member



 
Msg#: 3518652 posted 1:11 pm on Dec 2, 2007 (gmt 0)

I got a HUGE amount of hits from this in the last 24 hours

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MSIECrawler)

Can anyone please tell me what it is?

 

encyclo

WebmasterWorld Senior Member encyclo us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3518652 posted 1:20 pm on Dec 2, 2007 (gmt 0)

MSIECrawler is usually (unless someone's spoofing it) the user agent for Internet Explorer when downloading your site for offline browsing. See:

[msdn2.microsoft.com...]

Also: [google.ca...]

It should respect robots.txt:

User-Agent:MSIECrawler
Disallow:/

zerillos

5+ Year Member



 
Msg#: 3518652 posted 1:59 pm on Dec 2, 2007 (gmt 0)

Thank you for your reply!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved