Welcome to WebmasterWorld Guest from 54.196.77.14

Forum Moderators: goodroi

Message Too Old, No Replies

Yahoo robots.txt problem

what redirect?

     
6:37 am on Jul 24, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 30, 2003
posts:430
votes: 0


Slurp is coming to my site and getting a 301 for my robot.txt file and immediately after it is getting a 200. I have no idea why or how this is happening.

Any ideas?

6:36 pm on July 27, 2004 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9068
votes: 4


Are you doing any redirects from [example.com...] to [example.com...] for example?

If you are, that often explains why the 301 occurs when the bot tries the non-www version of robots.txt first, and gets redirected to the www version.

7:01 pm on July 27, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


I may be showing my lack of experience and/or picking at nits but I noticed you spelled the file robot.txt when in fact it is robots.txt.
9:53 pm on July 27, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 30, 2003
posts:430
votes: 0


Good eyes. I misspelled the name of the file. I have it spelled correctly on my website.

Encyclo, I think you are probably correct. I do have a 301 redirect from the www-less version to the www. version of the website. I didn't even think of that. It is wierd that it has only recently started to do this though. Maybe somebody linked to me without the www.

Thanks.