Welcome to WebmasterWorld Guest from 54.166.82.161

Forum Moderators: goodroi

Message Too Old, No Replies

Yahoo robots.txt problem

what redirect?

     
6:37 am on Jul 24, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 30, 2003
posts:430
votes: 0


Slurp is coming to my site and getting a 301 for my robot.txt file and immediately after it is getting a 200. I have no idea why or how this is happening.

Any ideas?

6:36 pm on July 27, 2004 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9074
votes: 6


Are you doing any redirects from [example.com...] to [example.com...] for example?

If you are, that often explains why the 301 occurs when the bot tries the non-www version of robots.txt first, and gets redirected to the www version.

7:01 pm on July 27, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


I may be showing my lack of experience and/or picking at nits but I noticed you spelled the file robot.txt when in fact it is robots.txt.
9:53 pm on July 27, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:July 30, 2003
posts:430
votes: 0


Good eyes. I misspelled the name of the file. I have it spelled correctly on my website.

Encyclo, I think you are probably correct. I do have a 301 redirect from the www-less version to the www. version of the website. I didn't even think of that. It is wierd that it has only recently started to do this though. Maybe somebody linked to me without the www.

Thanks.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members