Welcome to WebmasterWorld Guest from 54.226.23.160

Forum Moderators: Ocean10000

Message Too Old, No Replies

ShrinkTheWeb.com Crawler

     
4:29 pm on Jun 30, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2040
votes: 1


mail.shrinktheweb.com
ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com)

robots.txt? NO

"[W]e will eventually have millions of thumbnails in the database. ..." FAQ [community.shrinktheweb.com]

(Wonder how many of those will be shots of custom 403s?;)

8:44 pm on June 30, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 8, 2004
posts:1679
votes: 0


(Wonder how many of those will be shots of custom 403s?;)

Probably zero if:
1) you return correct 403 HTTP code as you should (custom txt page is irrelevant)
2) they (and that's very likely) only do screenshots on successfully retrieved pages

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members