Welcome to WebmasterWorld Guest from 54.146.248.111

Forum Moderators: goodroi

Message Too Old, No Replies

Cannot find error

Robots.txt syntax error

     

webapache

1:00 pm on Mar 13, 2004 (gmt 0)

10+ Year Member



I am getting errors in my robots.txt file.

Please help me to correct these errors if some one can .... I created this by using a tool.

User-Agent: *
Disallow: /cgi-bin/
Disallow: /pic/
Disallow: /images/
Disallow: /netforward/
Disallow: /reciprocal/
Disallow: /autorank/
Disallow: /members/

[edited by: oilman at 11:55 pm (utc) on Mar. 13, 2004]
[edit reason] no urls please [/edit]

closed

2:30 am on Mar 14, 2004 (gmt 0)

10+ Year Member



Doesn't look like there's anything wrong. What did you use to check it for errors, and what errors did you get?

kevinpate

1:14 pm on Mar 14, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



First, leave off the self promoting url's (a mod will strip it out if you don't do it first)

Second, what you've posted doesn't match the actual robots file.

Try User-agent*
Instead of User-Agent: *

Delete this line:
Disallow:User-agent: *

I may be wrong, but I think you need to dump these three lines also:
Disallow: *.gif
Disallow: *.jpg
Disallow: *.bmp

webapache

1:26 pm on Mar 14, 2004 (gmt 0)

10+ Year Member



Thanks for your help ...... the error was not with self promoting urls. As you said agent should be spelled with all lower case and second the / at the end of each directory is not allowed as it is considered as second entry ..... Now I have no errors .....

Question now rises .... is what is Mozilla? is this a robot or what?

closed

5:04 pm on Mar 14, 2004 (gmt 0)

10+ Year Member



What kevinpate means by the self promoting urls is that you're not supposed to post links to sites that you own because it's a violation of this site's TOS.

The capital "A" in User-Agent was definitely a problem. I guess I missed that.

AFAIK, the use of wildcards in Disallows are only allowed as extensions to the robots.txt standard. Googlebot, for example, has an extension that makes Disallows like this legal:

Disallow: /*.gif$

webapache:

the / at the end of each directory is not allowed as it is considered as second entry

Putting the / at the end of each directory is allowed and legal.

Mozilla is usually a human visitor, but there are also automated visitors that use it in their UA string.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month