| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
I kept getting an error validating my site blocking certain directories:
However when I put in a full address, it approved it:
(left out : so that it doesn't create a true link here, but in my robots.txt it exists)
Anyone else running into this issue?
Didn't know if it really mattered which way is better or should not be used.
The robots.txt file need not exist but if it does it must be called "robots.txt" and must be written and uploaded in ascii mode
It must be in the root directory of the web site as spiders will not look for it anywhere else
To exclude all robots from parts of the server
Exclude a specific spider from parts of the server
This indicates that nothing is disallowed and the spider can follow all links
To allow a single robot complete access and exclude all others
This would prevent your entire web site from being indexed
Hope this helps
No it didn't help because I already knew that information but I 'tested' my robots file to verify it was 'clean'.
The site gave me errors when I only tried the directory
but when I use a full link, it comes back as 'okay':
I have found that most bots don't listen anyways including Yahoo...they keep going after disallowed.
I'm so disgruntled with the net that I'm going to make my site go stealth.
What the hell is the point for search engines posting how to deal with their bots/spiders if they can't even control them properly?
And for all those worthless sites like DMOZ who fail to update on a timely basis, I have no respect for & could care less about dealing with them...
>>No it didn't help because I already knew that information
Well if you already knew the information above you would also of known that you have missed the ":" off all of your disallows
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved