Welcome to WebmasterWorld Guest from 54.163.40.152

Forum Moderators: open

Message Too Old, No Replies

"Fast PartnerSite Crawler" Keeps Looking for missing Robots.txt

     

decstar

8:48 pm on Aug 5, 2003 (gmt 0)

10+ Year Member



Every day for the past five days "Fast PartnerSite Crawler" on 66.77.73.109 has come looking for my robots.txt, which I didn't have until today. It has gone away each time though.

I am wondering:

1. Am I sending the crawler away by not having a robots.txt that invites it in, and,

2. Is this the real fast crawler. should this mean my site gets into AllTheWeb.

Decstar

<edit> Actually a look at the July logs shows it came every day in July aswell, sometimes using the agent "FAST-WebCrawler/3.8" </edit>

Rumbas

9:02 pm on Aug 5, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



Hi decstar, welcome to the ATW Forum. Hope you'll like it around here ;)

>1. Am I sending the crawler away by not having a robots.txt that invites it in

Certainly not, but having one reduces the ammount of 404 errors in you logs.

>Is this the real fast crawler.

Yup. The real deal.

>should this mean my site gets into AllTheWeb.

Yup :)

skipfactor

9:04 pm on Aug 5, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Am I sending the crawler away by not having a robots.txt that invites it in

No

FAST-WebCrawler/3.8

Yes, that should get you into ATW eventually.

[edit]echo on[/edit] :)

 

Featured Threads

Hot Threads This Week

Hot Threads This Month