Welcome to WebmasterWorld Guest from 107.20.20.39

Forum Moderators: goodroi

Message Too Old, No Replies

yahoo slurping for www.file1.html/robots.txt

what's wrong that slurp can't figure out?

   
7:01 pm on May 12, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I've been seeing Yahoo slurp repatedly asking for
/
/robots.txt
/filename.html/robots.txt
/filename2.html/robots.txt

Anyone know why this might be?

11:43 pm on May 12, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Well the first two are fairly standard request for the root page and the robots.txt file. However the other two requests are rather odd.

Are you doing any URL rewriting? That could possibly cause that problem.

9:08 am on May 14, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I thought so, too. But it seems ok. In the root, there is this in htaccess

RewriteEngine On
RewriteBase /
Options +FollowSymLinks
RewriteCond %{HTTP_HOST}!^www\.mysite\.com
RewriteRule ^(.*)$ [mysite.com...] [R=301,L]

In the folder, I now see there is the same in htaccess:

RewriteEngine On
RewriteBase /
Options +FollowSymLinks
RewriteCond %{HTTP_HOST}!^www\.mysite\.com
RewriteRule ^(.*)$ [mysite.com...] [R=301,L]

I think the rewrite in the folder was put there accidentally, but I don't see that it matters.

Yahoo is the only spider showing the odd behavior. Others find robots.txt in the root.