Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

fast & robots.txt

did i get it wrong...



4:47 pm on Dec 1, 2002 (gmt 0)

10+ Year Member


FAST-WebCrawler has visited me twice, but only retrieved one file each time (which I assume was robots.txt).

Checking the correct syntax, I realise I made a small error and I wonder if this is the reason it hasn't done a full crawl. (google and altavista have crawled OK).

I had

User-agent: *

Disallow: /~blah

but i should have had

User-agent: *

Disallow: /~blah/ #don't forget the final slash

Did fast interpret that as Disallow: / (ie disallow the whole site?

Is my interpretation correct? Should I let fast know that I made a mistake or will they come back and look without me notifying them?




4:58 pm on Dec 1, 2002 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


The only result of your typo is that files in your top-level directory that start with "~blah" (e.g. "/~blah2.html" would be disallowed, as well as the subdirectory "/~blah/" that you intended to disallow. If no such files exist, then the typo will have no practical effect.

Check out the robots.txt validator page [searchengineworld.com] and the info linked on that page for more info on why you might have a problem. If you don't find any problems, it may just be that fast has found a link to your site while working on another site, and came over to check robots.txt to see if it would be allowed to spider your site later.



7:29 pm on Dec 1, 2002 (gmt 0)

10+ Year Member

Hi msampson

I also had some problems with Fast crawling my site. They visited regularly over a period of months but never requested anything other than robots.txt. I would recommend you contact them as I did as they are very helpful and my site is now being deep crawled and indexed...


10:22 am on Dec 2, 2002 (gmt 0)

10+ Year Member

thanks for very much for both your answers.

i'm always very impressed by the level of knowledge and helpfulness on these boards.


Featured Threads

Hot Threads This Week

Hot Threads This Month