homepage Welcome to WebmasterWorld Guest from 54.166.39.179
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
fast & robots.txt
did i get it wrong...
msampson

10+ Year Member



 
Msg#: 721 posted 4:47 pm on Dec 1, 2002 (gmt 0)

Hi

FAST-WebCrawler has visited me twice, but only retrieved one file each time (which I assume was robots.txt).

Checking the correct syntax, I realise I made a small error and I wonder if this is the reason it hasn't done a full crawl. (google and altavista have crawled OK).

I had

User-agent: *

Disallow: /~blah

but i should have had

User-agent: *

Disallow: /~blah/ #don't forget the final slash

Did fast interpret that as Disallow: / (ie disallow the whole site?

Is my interpretation correct? Should I let fast know that I made a mistake or will they come back and look without me notifying them?

Thanks

Miles

 

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 721 posted 4:58 pm on Dec 1, 2002 (gmt 0)

msampson,

The only result of your typo is that files in your top-level directory that start with "~blah" (e.g. "/~blah2.html" would be disallowed, as well as the subdirectory "/~blah/" that you intended to disallow. If no such files exist, then the typo will have no practical effect.

Check out the robots.txt validator page [searchengineworld.com] and the info linked on that page for more info on why you might have a problem. If you don't find any problems, it may just be that fast has found a link to your site while working on another site, and came over to check robots.txt to see if it would be allowed to spider your site later.

HTH,
Jim

CuriousWeb

10+ Year Member



 
Msg#: 721 posted 7:29 pm on Dec 1, 2002 (gmt 0)

Hi msampson

I also had some problems with Fast crawling my site. They visited regularly over a period of months but never requested anything other than robots.txt. I would recommend you contact them as I did as they are very helpful and my site is now being deep crawled and indexed...

msampson

10+ Year Member



 
Msg#: 721 posted 10:22 am on Dec 2, 2002 (gmt 0)

thanks for very much for both your answers.

i'm always very impressed by the level of knowledge and helpfulness on these boards.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved