| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|"Fast PartnerSite Crawler" Keeps Looking for missing Robots.txt|
Every day for the past five days "Fast PartnerSite Crawler" on 18.104.22.168 has come looking for my robots.txt, which I didn't have until today. It has gone away each time though.
I am wondering:
1. Am I sending the crawler away by not having a robots.txt that invites it in, and,
2. Is this the real fast crawler. should this mean my site gets into AllTheWeb.
<edit> Actually a look at the July logs shows it came every day in July aswell, sometimes using the agent "FAST-WebCrawler/3.8" </edit>
Hi decstar, welcome to the ATW Forum. Hope you'll like it around here ;)
>1. Am I sending the crawler away by not having a robots.txt that invites it in
Certainly not, but having one reduces the ammount of 404 errors in you logs.
>Is this the real fast crawler.
Yup. The real deal.
>should this mean my site gets into AllTheWeb.
|Am I sending the crawler away by not having a robots.txt that invites it in |
Yes, that should get you into ATW eventually.
echo on[/edit] :)
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved