Forum Moderators: goodroi

Message Too Old, No Replies

Yahoo robots.txt problem

what redirect?

         

yowza

6:37 am on Jul 24, 2004 (gmt 0)

10+ Year Member



Slurp is coming to my site and getting a 301 for my robot.txt file and immediately after it is getting a 200. I have no idea why or how this is happening.

Any ideas?

encyclo

6:36 pm on Jul 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Are you doing any redirects from [example.com...] to [example.com...] for example?

If you are, that often explains why the 301 occurs when the bot tries the non-www version of robots.txt first, and gets redirected to the www version.

GaryK

7:01 pm on Jul 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I may be showing my lack of experience and/or picking at nits but I noticed you spelled the file robot.txt when in fact it is robots.txt.

yowza

9:53 pm on Jul 27, 2004 (gmt 0)

10+ Year Member



Good eyes. I misspelled the name of the file. I have it spelled correctly on my website.

Encyclo, I think you are probably correct. I do have a 301 redirect from the www-less version to the www. version of the website. I didn't even think of that. It is wierd that it has only recently started to do this though. Maybe somebody linked to me without the www.

Thanks.