Welcome to WebmasterWorld Guest from 54.225.47.124

Forum Moderators: goodroi

Message Too Old, No Replies

Cannot find error

Robots.txt syntax error

     
1:00 pm on Mar 13, 2004 (gmt 0)

New User

10+ Year Member

joined:Mar 13, 2004
posts:5
votes: 0


I am getting errors in my robots.txt file.

Please help me to correct these errors if some one can .... I created this by using a tool.

User-Agent: *
Disallow: /cgi-bin/
Disallow: /pic/
Disallow: /images/
Disallow: /netforward/
Disallow: /reciprocal/
Disallow: /autorank/
Disallow: /members/

[edited by: oilman at 11:55 pm (utc) on Mar. 13, 2004]
[edit reason] no urls please [/edit]

2:30 am on Mar 14, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 20, 2003
posts:255
votes: 0


Doesn't look like there's anything wrong. What did you use to check it for errors, and what errors did you get?
1:14 pm on Mar 14, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 2, 2002
posts:1167
votes: 0


First, leave off the self promoting url's (a mod will strip it out if you don't do it first)

Second, what you've posted doesn't match the actual robots file.

Try User-agent*
Instead of User-Agent: *

Delete this line:
Disallow:User-agent: *

I may be wrong, but I think you need to dump these three lines also:
Disallow: *.gif
Disallow: *.jpg
Disallow: *.bmp

1:26 pm on Mar 14, 2004 (gmt 0)

New User

10+ Year Member

joined:Mar 13, 2004
posts:5
votes: 0


Thanks for your help ...... the error was not with self promoting urls. As you said agent should be spelled with all lower case and second the / at the end of each directory is not allowed as it is considered as second entry ..... Now I have no errors .....

Question now rises .... is what is Mozilla? is this a robot or what?

5:04 pm on Mar 14, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 20, 2003
posts:255
votes: 0


What kevinpate means by the self promoting urls is that you're not supposed to post links to sites that you own because it's a violation of this site's TOS.

The capital "A" in User-Agent was definitely a problem. I guess I missed that.

AFAIK, the use of wildcards in Disallows are only allowed as extensions to the robots.txt standard. Googlebot, for example, has an extension that makes Disallows like this legal:

Disallow: /*.gif$

webapache:

the / at the end of each directory is not allowed as it is considered as second entry

Putting the / at the end of each directory is allowed and legal.

Mozilla is usually a human visitor, but there are also automated visitors that use it in their UA string.