Welcome to WebmasterWorld Guest from 18.104.22.168 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Yahoo robots.txt problem what redirect? yowza msg:1526147 6:37 am on Jul 24, 2004 (gmt 0) Slurp is coming to my site and getting a 301 for my robot.txt file and immediately after it is getting a 200. I have no idea why or how this is happening.
encyclo msg:1526148 6:36 pm on Jul 27, 2004 (gmt 0)
Are you doing any redirects from [ ...] to [ example.com ...] for example? example.com
If you are, that often explains why the 301 occurs when the bot tries the non-www version of robots.txt first, and gets redirected to the www version.
GaryK msg:1526149 7:01 pm on Jul 27, 2004 (gmt 0)
I may be showing my lack of experience and/or picking at nits but I noticed you spelled the file robot.txt when in fact it is robot s.txt. yowza msg:1526150 9:53 pm on Jul 27, 2004 (gmt 0)
Good eyes. I misspelled the name of the file. I have it spelled correctly on my website.
Encyclo, I think you are probably correct. I do have a 301 redirect from the www-less version to the www. version of the website. I didn't even think of that. It is wierd that it has only recently started to do this though. Maybe somebody linked to me without the www.