Welcome to WebmasterWorld Guest from 54.144.107.83

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

ShrinkTheWeb.com Crawler

     
4:29 pm on Jun 30, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


mail.shrinktheweb.com
ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com)

robots.txt? NO

"[W]e will eventually have millions of thumbnails in the database. ..." FAQ [community.shrinktheweb.com]

(Wonder how many of those will be shots of custom 403s?;)

8:44 pm on June 30, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 8, 2004
posts:1679
votes: 0


(Wonder how many of those will be shots of custom 403s?;)

Probably zero if:
1) you return correct 403 HTTP code as you should (custom txt page is irrelevant)
2) they (and that's very likely) only do screenshots on successfully retrieved pages