yahoo slurping for www.file1.html/robots.txt

what's wrong that slurp can't figure out?

[paybacksa]

I've been seeing Yahoo slurp repatedly asking for
/
/robots.txt
/filename.html/robots.txt
/filename2.html/robots.txt

Anyone know why this might be?

[Dijkgraaf]

Well the first two are fairly standard request for the root page and the robots.txt file. However the other two requests are rather odd.

Are you doing any URL rewriting? That could possibly cause that problem.

[paybacksa]

I thought so, too. But it seems ok. In the root, there is this in htaccess

RewriteEngine On
RewriteBase /
Options +FollowSymLinks
RewriteCond %{HTTP_HOST}!^www\.mysite\.com
RewriteRule ^(.*)$ [mysite.com...] [R=301,L]

In the folder, I now see there is the same in htaccess:

RewriteEngine On
RewriteBase /
Options +FollowSymLinks
RewriteCond %{HTTP_HOST}!^www\.mysite\.com
RewriteRule ^(.*)$ [mysite.com...] [R=301,L]

I think the rewrite in the folder was put there accidentally, but I don't see that it matters.

Yahoo is the only spider showing the odd behavior. Others find robots.txt in the root.