Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
However when I put in a full address, it approved it:
(left out : so that it doesn't create a true link here, but in my robots.txt it exists)
Anyone else running into this issue?
Didn't know if it really mattered which way is better or should not be used.
It must be in the root directory of the web site as spiders will not look for it anywhere else
To exclude all robots from parts of the server
Exclude a specific spider from parts of the server
This indicates that nothing is disallowed and the spider can follow all links
To allow a single robot complete access and exclude all others
This would prevent your entire web site from being indexed
Hope this helps
The site gave me errors when I only tried the directory
but when I use a full link, it comes back as 'okay':
I have found that most bots don't listen anyways including Yahoo...they keep going after disallowed.
I'm so disgruntled with the net that I'm going to make my site go stealth.
What the hell is the point for search engines posting how to deal with their bots/spiders if they can't even control them properly?
And for all those worthless sites like DMOZ who fail to update on a timely basis, I have no respect for & could care less about dealing with them...