homepage Welcome to WebmasterWorld Guest from 54.198.148.191
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
MSIECrawler
zerillos




msg:3518651
 1:11 pm on Dec 2, 2007 (gmt 0)

I got a HUGE amount of hits from this in the last 24 hours

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MSIECrawler)

Can anyone please tell me what it is?

 

encyclo




msg:3518656
 1:20 pm on Dec 2, 2007 (gmt 0)

MSIECrawler is usually (unless someone's spoofing it) the user agent for Internet Explorer when downloading your site for offline browsing. See:

[msdn2.microsoft.com...]

Also: [google.ca...]

It should respect robots.txt:

User-Agent:MSIECrawler
Disallow:/

zerillos




msg:3518664
 1:59 pm on Dec 2, 2007 (gmt 0)

Thank you for your reply!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved