Welcome to WebmasterWorld Guest from 54.156.90.21

Forum Moderators: goodroi

Message Too Old, No Replies

yahoo slurping for www.file1.html/robots.txt

what's wrong that slurp can't figure out?

     
7:01 pm on May 12, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 1, 2004
posts:812
votes: 0


I've been seeing Yahoo slurp repatedly asking for
/
/robots.txt
/filename.html/robots.txt
/filename2.html/robots.txt

Anyone know why this might be?

11:43 pm on May 12, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 31, 2005
posts:1108
votes: 0


Well the first two are fairly standard request for the root page and the robots.txt file. However the other two requests are rather odd.

Are you doing any URL rewriting? That could possibly cause that problem.

9:08 am on May 14, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 1, 2004
posts:812
votes: 0


I thought so, too. But it seems ok. In the root, there is this in htaccess

RewriteEngine On
RewriteBase /
Options +FollowSymLinks
RewriteCond %{HTTP_HOST}!^www\.mysite\.com
RewriteRule ^(.*)$ [mysite.com...] [R=301,L]

In the folder, I now see there is the same in htaccess:

RewriteEngine On
RewriteBase /
Options +FollowSymLinks
RewriteCond %{HTTP_HOST}!^www\.mysite\.com
RewriteRule ^(.*)$ [mysite.com...] [R=301,L]

I think the rewrite in the folder was put there accidentally, but I don't see that it matters.

Yahoo is the only spider showing the odd behavior. Others find robots.txt in the root.