homepage Welcome to WebmasterWorld Guest from 54.205.189.156
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Cannot find error
Robots.txt syntax error
webapache




msg:1528281
 1:00 pm on Mar 13, 2004 (gmt 0)

I am getting errors in my robots.txt file.

Please help me to correct these errors if some one can .... I created this by using a tool.

User-Agent: *
Disallow: /cgi-bin/
Disallow: /pic/
Disallow: /images/
Disallow: /netforward/
Disallow: /reciprocal/
Disallow: /autorank/
Disallow: /members/

[edited by: oilman at 11:55 pm (utc) on Mar. 13, 2004]
[edit reason] no urls please [/edit]

 

closed




msg:1528282
 2:30 am on Mar 14, 2004 (gmt 0)

Doesn't look like there's anything wrong. What did you use to check it for errors, and what errors did you get?

kevinpate




msg:1528283
 1:14 pm on Mar 14, 2004 (gmt 0)

First, leave off the self promoting url's (a mod will strip it out if you don't do it first)

Second, what you've posted doesn't match the actual robots file.

Try User-agent*
Instead of User-Agent: *

Delete this line:
Disallow:User-agent: *

I may be wrong, but I think you need to dump these three lines also:
Disallow: *.gif
Disallow: *.jpg
Disallow: *.bmp

webapache




msg:1528284
 1:26 pm on Mar 14, 2004 (gmt 0)

Thanks for your help ...... the error was not with self promoting urls. As you said agent should be spelled with all lower case and second the / at the end of each directory is not allowed as it is considered as second entry ..... Now I have no errors .....

Question now rises .... is what is Mozilla? is this a robot or what?

closed




msg:1528285
 5:04 pm on Mar 14, 2004 (gmt 0)

What kevinpate means by the self promoting urls is that you're not supposed to post links to sites that you own because it's a violation of this site's TOS.

The capital "A" in User-Agent was definitely a problem. I guess I missed that.

AFAIK, the use of wildcards in Disallows are only allowed as extensions to the robots.txt standard. Googlebot, for example, has an extension that makes Disallows like this legal:

Disallow: /*.gif$

webapache:
the / at the end of each directory is not allowed as it is considered as second entry

Putting the / at the end of each directory is allowed and legal.

Mozilla is usually a human visitor, but there are also automated visitors that use it in their UA string.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved