homepage Welcome to WebmasterWorld Guest from 54.163.70.249
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to ban YahooSeeker?
Used robots file, but it doesn't obey.
zf007




msg:1529154
 3:44 am on Dec 28, 2003 (gmt 0)

Hi,
I tried to ban yahooseeker from crawling my site,
I used robots.txt:
User-agent: YahooSeeker
Disallow: /
All other crawlers obey my robots file, except yahooseeker.

YahooSeeker is still crawling my site like crazy!
the ip is:
66.196.93.32 I checked this ip, it never read robots file.
the log file is like:
YahooSeeker/1.0+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://help.yahoo.com/help/us/shop/merchant/) - 200 335

but another ip: 66.196.93.5 reads the robots file then left.
the log file is like:
HTTP/1.0 YahooSeeker/1.0+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://search.yahoo.com/yahooseeker.html) - 200 233

Are they both Yahooseeker?
Anyone met this before and any suggestions is really appreciated.

BTW, since my site doesn't sell any products, there should be no negative effects on my site if I ban yahooseeker?

Thank you all, Happy new year!

Feng

 

sidyadav




msg:1529155
 3:59 am on Jan 1, 2004 (gmt 0)

All other crawlers obey my robots file, except yahooseeker.

Thats not true, there are more than a 1000 bots out there that don't obey robots.txt. Only the good ones do.

YahooSeeker is used for Yahoo's product search service: [products.yahoo.com...] , if your site doesn't sell any products, you should ban it.

Are they both Yahooseeker?

They both come from the same batch of IP 66.196.93 which resolve to yj1028.inktomisearch.com, and since Yahoo owns Inktomi, They indeed belong to Yahoo.

How to ban YahooSeeker?

If you want to block the whole YahooSeeker IPs, enter this code in your .htaccess file:
order allow,deny
allow from all
deny from 66.196.93.32

It should work.

Sid

BTW - Welcome to WebmasterWorld!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved