Welcome to WebmasterWorld Guest from 174.129.163.89

Forum Moderators: goodroi

Message Too Old, No Replies

Cannot find error

Robots.txt syntax error

     
1:00 pm on Mar 13, 2004 (gmt 0)

New User

10+ Year Member

joined:Mar 13, 2004
posts:5
votes: 0


I am getting errors in my robots.txt file.

Please help me to correct these errors if some one can .... I created this by using a tool.

User-Agent: *
Disallow: /cgi-bin/
Disallow: /pic/
Disallow: /images/
Disallow: /netforward/
Disallow: /reciprocal/
Disallow: /autorank/
Disallow: /members/

[edited by: oilman at 11:55 pm (utc) on Mar. 13, 2004]
[edit reason] no urls please [/edit]

2:30 am on Mar 14, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 20, 2003
posts:255
votes: 0


Doesn't look like there's anything wrong. What did you use to check it for errors, and what errors did you get?
1:14 pm on Mar 14, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 2, 2002
posts:1167
votes: 0


First, leave off the self promoting url's (a mod will strip it out if you don't do it first)

Second, what you've posted doesn't match the actual robots file.

Try User-agent*
Instead of User-Agent: *

Delete this line:
Disallow:User-agent: *

I may be wrong, but I think you need to dump these three lines also:
Disallow: *.gif
Disallow: *.jpg
Disallow: *.bmp

1:26 pm on Mar 14, 2004 (gmt 0)

New User

10+ Year Member

joined:Mar 13, 2004
posts:5
votes: 0


Thanks for your help ...... the error was not with self promoting urls. As you said agent should be spelled with all lower case and second the / at the end of each directory is not allowed as it is considered as second entry ..... Now I have no errors .....

Question now rises .... is what is Mozilla? is this a robot or what?

5:04 pm on Mar 14, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 20, 2003
posts:255
votes: 0


What kevinpate means by the self promoting urls is that you're not supposed to post links to sites that you own because it's a violation of this site's TOS.

The capital "A" in User-Agent was definitely a problem. I guess I missed that.

AFAIK, the use of wildcards in Disallows are only allowed as extensions to the robots.txt standard. Googlebot, for example, has an extension that makes Disallows like this legal:

Disallow: /*.gif$

webapache:

the / at the end of each directory is not allowed as it is considered as second entry

Putting the / at the end of each directory is allowed and legal.

Mozilla is usually a human visitor, but there are also automated visitors that use it in their UA string.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members