homepage Welcome to WebmasterWorld Guest from 54.166.123.2
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
"Fast PartnerSite Crawler" Keeps Looking for missing Robots.txt
decstar




msg:220046
 8:48 pm on Aug 5, 2003 (gmt 0)

Every day for the past five days "Fast PartnerSite Crawler" on 66.77.73.109 has come looking for my robots.txt, which I didn't have until today. It has gone away each time though.

I am wondering:

1. Am I sending the crawler away by not having a robots.txt that invites it in, and,

2. Is this the real fast crawler. should this mean my site gets into AllTheWeb.

Decstar

<edit> Actually a look at the July logs shows it came every day in July aswell, sometimes using the agent "FAST-WebCrawler/3.8" </edit>

 

Rumbas




msg:220047
 9:02 pm on Aug 5, 2003 (gmt 0)

Hi decstar, welcome to the ATW Forum. Hope you'll like it around here ;)

>1. Am I sending the crawler away by not having a robots.txt that invites it in

Certainly not, but having one reduces the ammount of 404 errors in you logs.

>Is this the real fast crawler.

Yup. The real deal.

>should this mean my site gets into AllTheWeb.

Yup :)

skipfactor




msg:220048
 9:04 pm on Aug 5, 2003 (gmt 0)

Am I sending the crawler away by not having a robots.txt that invites it in

No

FAST-WebCrawler/3.8

Yes, that should get you into ATW eventually.

[edit]echo on[/edit] :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved