Welcome to WebmasterWorld Guest from 220.127.116.11 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Cannot find error Robots.txt syntax error webapache
I am getting errors in my robots.txt file.
Please help me to correct these errors if some one can .... I created this by using a tool.
Disallow: /cgi-bin/ Disallow: /pic/ Disallow: /images/ Disallow: /netforward/ Disallow: /reciprocal/ Disallow: /autorank/ Disallow: /members/ [ edited by: oilman at 11:55 pm (utc) on Mar. 13, 2004] [edit reason] no urls please [/edit]
Doesn't look like there's anything wrong. What did you use to check it for errors, and what errors did you get? kevinpate
First, leave off the self promoting url's (a mod will strip it out if you don't do it first)
Second, what you've posted doesn't match the actual robots file.
Instead of User-Agent: *
Delete this line:
I may be wrong, but I think you need to dump these three lines also:
Disallow: *.gif Disallow: *.jpg Disallow: *.bmp webapache
Thanks for your help ...... the error was not with self promoting urls. As you said agent should be spelled with all lower case and second the / at the end of each directory is not allowed as it is considered as second entry ..... Now I have no errors .....
Question now rises .... is what is Mozilla? is this a robot or what?
What kevinpate means by the self promoting urls is that you're not supposed to post links to sites that you own because it's a violation of this site's TOS.
The capital "A" in
User-Agent was definitely a problem. I guess I missed that.
AFAIK, the use of wildcards in
Disallows are only allowed as extensions to the robots.txt standard. Googlebot, for example, has an extension that makes Disallows like this legal:
the / at the end of each directory is not allowed as it is considered as second entry
Putting the / at the end of each directory is allowed and legal.
Mozilla is usually a human visitor, but there are also automated visitors that use it in their UA string.