Welcome to WebmasterWorld Guest from 54.159.190.106

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

ShrinkTheWeb.com Crawler

   
4:29 pm on Jun 30, 2009 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



mail.shrinktheweb.com
ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com)

robots.txt? NO

"[W]e will eventually have millions of thumbnails in the database. ..." FAQ [community.shrinktheweb.com]

(Wonder how many of those will be shots of custom 403s?;)

8:44 pm on Jun 30, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



(Wonder how many of those will be shots of custom 403s?;)

Probably zero if:
1) you return correct 403 HTTP code as you should (custom txt page is irrelevant)
2) they (and that's very likely) only do screenshots on successfully retrieved pages

 

Featured Threads

Hot Threads This Week

Hot Threads This Month