Forum Moderators: open

Message Too Old, No Replies

ShrinkTheWeb.com Crawler

         

Pfui

4:29 pm on Jun 30, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



mail.shrinktheweb.com
ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com)

robots.txt? NO

"[W]e will eventually have millions of thumbnails in the database. ..." FAQ [community.shrinktheweb.com]

(Wonder how many of those will be shots of custom 403s?;)

Lord Majestic

8:44 pm on Jun 30, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



(Wonder how many of those will be shots of custom 403s?;)

Probably zero if:
1) you return correct 403 HTTP code as you should (custom txt page is irrelevant)
2) they (and that's very likely) only do screenshots on successfully retrieved pages