homepage Welcome to WebmasterWorld Guest from 54.211.97.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
ShrinkTheWeb.com Crawler
Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3943330 posted 4:29 pm on Jun 30, 2009 (gmt 0)

mail.shrinktheweb.com
ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com)

robots.txt? NO

"[W]e will eventually have millions of thumbnails in the database. ..." FAQ [community.shrinktheweb.com]

(Wonder how many of those will be shots of custom 403s?;)

 

Lord Majestic

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3943330 posted 8:44 pm on Jun 30, 2009 (gmt 0)

(Wonder how many of those will be shots of custom 403s?;)

Probably zero if:
1) you return correct 403 HTTP code as you should (custom txt page is irrelevant)
2) they (and that's very likely) only do screenshots on successfully retrieved pages

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved