homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Yahoo robots.txt problem
what redirect?

 6:37 am on Jul 24, 2004 (gmt 0)

Slurp is coming to my site and getting a 301 for my robot.txt file and immediately after it is getting a 200. I have no idea why or how this is happening.

Any ideas?



 6:36 pm on Jul 27, 2004 (gmt 0)

Are you doing any redirects from [example.com...] to [example.com...] for example?

If you are, that often explains why the 301 occurs when the bot tries the non-www version of robots.txt first, and gets redirected to the www version.


 7:01 pm on Jul 27, 2004 (gmt 0)

I may be showing my lack of experience and/or picking at nits but I noticed you spelled the file robot.txt when in fact it is robots.txt.


 9:53 pm on Jul 27, 2004 (gmt 0)

Good eyes. I misspelled the name of the file. I have it spelled correctly on my website.

Encyclo, I think you are probably correct. I do have a 301 redirect from the www-less version to the www. version of the website. I didn't even think of that. It is wierd that it has only recently started to do this though. Maybe somebody linked to me without the www.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved