Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
ERROR_DISALLOW: Crawling Disallowed By robots.txt
The thing is I have double checked the Bot name - Slurp/2.0 and the robots.txt has that exact bot and it is allowed anywhere on the site so there should be no probs.
Can someone shed some light on this for me?
[edited by: Marcia at 7:22 am (utc) on Dec. 28, 2002]
[edit reason] no specifics, please [/edit]
Welcome to WebmasterWorld [webmasterworld.com]!
(Please read this and the Terms of Service below - we don't post URL's except in profiles)
However... Your robots.txt has got several problems - problems which are likely "fatal errors" and may lead most 'bots to give up and take the default User-agent * record - which tells them to stay out.
I recommend you rename that robots.txt file to robots.tst immediately to put it safely out of the robots' sight. Then use the robots.txt Validator [searchengineworld.com] at WebmasterWorld's sister site, Search Engine World. Note that you can tell it to validate robots.tst, so that you can work on the file without having a real 'bot choke on it. Get it fixed, then name it back to robots.txt.
The major problem with your robots.txt is that you have "extra" newlines in it. User-agent and Disallow directives may not span multiple lines, and a blank line is required between records (before each new User-agent, that is).
I'd also take jdMorgan's advice and validate your file quickly. That last line in your robots.txt file is disallowing all spiders from indexing your site. You might as well get rid of the rest of them if you keep that last one. You could save on file size! ;)
P.S. You want to be very careful when handling the robots.txt file. If you are copying someone else's file, they may have entries in there that won't apply to your site. Where did you pick up that last one...
User agent: *
If it isn't right perhaps I need to alert the people at Search Engine World too.
Thanks for your help and will try to fix now.
This is a general wildcard directive for all bots (an exclusion directive in this example):
User agent: *
But a specific directive can override the general one, just for the specified bot, while all non-specified bots remain subject to the general directive. So this record would override the above general exclusion and specifically allow (ie not exclude) scooter while all other bots remain excluded:
Both records would have to appear in the file for this explanation to hold.
First thing I'd do, is take care of the errors, which you've done. Check your Ink listings and see if the error is still there.
If so, then I would drop your Ink provider an email and alert them to the issues. That last Disallow: line is fine, I did not see that you had an allow for Slurp/2.0. I need to make sure I am fully alert when hanging out around here!