"Fast PartnerSite Crawler" Keeps Looking for missing Robots.txt

[decstar]

Every day for the past five days "Fast PartnerSite Crawler" on 66.77.73.109 has come looking for my robots.txt, which I didn't have until today. It has gone away each time though.

I am wondering:

1. Am I sending the crawler away by not having a robots.txt that invites it in, and,

2. Is this the real fast crawler. should this mean my site gets into AllTheWeb.

Decstar

<edit> Actually a look at the July logs shows it came every day in July aswell, sometimes using the agent "FAST-WebCrawler/3.8" </edit>

[Rumbas]

Hi decstar, welcome to the ATW Forum. Hope you'll like it around here ;)

>1. Am I sending the crawler away by not having a robots.txt that invites it in

Certainly not, but having one reduces the ammount of 404 errors in you logs.

>Is this the real fast crawler.

Yup. The real deal.

>should this mean my site gets into AllTheWeb.

Yup :)

[skipfactor]

Am I sending the crawler away by not having a robots.txt that invites it in

No

FAST-WebCrawler/3.8

Yes, that should get you into ATW eventually.

[edit]echo on[/edit] :)