Welcome to WebmasterWorld Guest from 184.108.40.206 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
Cannot find error Robots.txt syntax error webapache msg:1528281 1:00 pm on Mar 13, 2004 (gmt 0) I am getting errors in my robots.txt file.
Please help me to correct these errors if some one can .... I created this by using a tool.
Disallow: /cgi-bin/ Disallow: /pic/ Disallow: /images/ Disallow: /netforward/ Disallow: /reciprocal/ Disallow: /autorank/ Disallow: /members/ [ edited by: oilman at 11:55 pm (utc) on Mar. 13, 2004] [edit reason] no urls please [/edit]
closed msg:1528282 2:30 am on Mar 14, 2004 (gmt 0)
Doesn't look like there's anything wrong. What did you use to check it for errors, and what errors did you get? kevinpate msg:1528283 1:14 pm on Mar 14, 2004 (gmt 0)
First, leave off the self promoting url's (a mod will strip it out if you don't do it first)
Second, what you've posted doesn't match the actual robots file.
Instead of User-Agent: *
Delete this line:
I may be wrong, but I think you need to dump these three lines also:
Disallow: *.gif Disallow: *.jpg Disallow: *.bmp webapache msg:1528284 1:26 pm on Mar 14, 2004 (gmt 0)
Thanks for your help ...... the error was not with self promoting urls. As you said agent should be spelled with all lower case and second the / at the end of each directory is not allowed as it is considered as second entry ..... Now I have no errors .....
Question now rises .... is what is Mozilla? is this a robot or what?
closed msg:1528285 5:04 pm on Mar 14, 2004 (gmt 0)
What kevinpate means by the self promoting urls is that you're not supposed to post links to sites that you own because it's a violation of this site's TOS.
The capital "A" in
User-Agent was definitely a problem. I guess I missed that.
AFAIK, the use of wildcards in
Disallows are only allowed as extensions to the robots.txt standard. Googlebot, for example, has an extension that makes Disallows like this legal:
the / at the end of each directory is not allowed as it is considered as second entry
Putting the / at the end of each directory is allowed and legal.
Mozilla is usually a human visitor, but there are also automated visitors that use it in their UA string.