homepage Welcome to WebmasterWorld Guest from 54.226.235.222
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
yahoo slurping for www.file1.html/robots.txt
what's wrong that slurp can't figure out?
paybacksa




msg:1528423
 7:01 pm on May 12, 2006 (gmt 0)

I've been seeing Yahoo slurp repatedly asking for
/
/robots.txt
/filename.html/robots.txt
/filename2.html/robots.txt

Anyone know why this might be?

 

Dijkgraaf




msg:1528424
 11:43 pm on May 12, 2006 (gmt 0)

Well the first two are fairly standard request for the root page and the robots.txt file. However the other two requests are rather odd.

Are you doing any URL rewriting? That could possibly cause that problem.

paybacksa




msg:1528425
 9:08 am on May 14, 2006 (gmt 0)

I thought so, too. But it seems ok. In the root, there is this in htaccess

RewriteEngine On
RewriteBase /
Options +FollowSymLinks
RewriteCond %{HTTP_HOST}!^www\.mysite\.com
RewriteRule ^(.*)$ [mysite.com...] [R=301,L]

In the folder, I now see there is the same in htaccess:

RewriteEngine On
RewriteBase /
Options +FollowSymLinks
RewriteCond %{HTTP_HOST}!^www\.mysite\.com
RewriteRule ^(.*)$ [mysite.com...] [R=301,L]

I think the rewrite in the folder was put there accidentally, but I don't see that it matters.

Yahoo is the only spider showing the odd behavior. Others find robots.txt in the root.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved