Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Yahoo robots.txt problem

what redirect?



6:37 am on Jul 24, 2004 (gmt 0)

10+ Year Member

Slurp is coming to my site and getting a 301 for my robot.txt file and immediately after it is getting a 200. I have no idea why or how this is happening.

Any ideas?


6:36 pm on Jul 27, 2004 (gmt 0)

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Are you doing any redirects from [example.com...] to [example.com...] for example?

If you are, that often explains why the 301 occurs when the bot tries the non-www version of robots.txt first, and gets redirected to the www version.


7:01 pm on Jul 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I may be showing my lack of experience and/or picking at nits but I noticed you spelled the file robot.txt when in fact it is robots.txt.


9:53 pm on Jul 27, 2004 (gmt 0)

10+ Year Member

Good eyes. I misspelled the name of the file. I have it spelled correctly on my website.

Encyclo, I think you are probably correct. I do have a 301 redirect from the www-less version to the www. version of the website. I didn't even think of that. It is wierd that it has only recently started to do this though. Maybe somebody linked to me without the www.



Featured Threads

Hot Threads This Week

Hot Threads This Month