Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

fast & robots.txt

did i get it wrong...

4:47 pm on Dec 1, 2002 (gmt 0)

New User

10+ Year Member

joined:Feb 24, 2002
votes: 0


FAST-WebCrawler has visited me twice, but only retrieved one file each time (which I assume was robots.txt).

Checking the correct syntax, I realise I made a small error and I wonder if this is the reason it hasn't done a full crawl. (google and altavista have crawled OK).

I had

User-agent: *

Disallow: /~blah

but i should have had

User-agent: *

Disallow: /~blah/ #don't forget the final slash

Did fast interpret that as Disallow: / (ie disallow the whole site?

Is my interpretation correct? Should I let fast know that I made a mistake or will they come back and look without me notifying them?



4:58 pm on Dec 1, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0


The only result of your typo is that files in your top-level directory that start with "~blah" (e.g. "/~blah2.html" would be disallowed, as well as the subdirectory "/~blah/" that you intended to disallow. If no such files exist, then the typo will have no practical effect.

Check out the robots.txt validator page [searchengineworld.com] and the info linked on that page for more info on why you might have a problem. If you don't find any problems, it may just be that fast has found a link to your site while working on another site, and came over to check robots.txt to see if it would be allowed to spider your site later.


7:29 pm on Dec 1, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 22, 2002
votes: 0

Hi msampson

I also had some problems with Fast crawling my site. They visited regularly over a period of months but never requested anything other than robots.txt. I would recommend you contact them as I did as they are very helpful and my site is now being deep crawled and indexed...

10:22 am on Dec 2, 2002 (gmt 0)

New User

10+ Year Member

joined:Feb 24, 2002
votes: 0

thanks for very much for both your answers.

i'm always very impressed by the level of knowledge and helpfulness on these boards.