Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo! Slurp


Pfui - 12:48 am on Sep 11, 2011 (gmt 0)


A few minutes ago, another .yst + /3.0 combo tried to slurp graphics:

b5131265.yst.yahoo.net [74.6.8.89]
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)

robots.txt? NO

We shall see if 403'ing (gif|jpg|png) makes any difference...

P.S.
ALL of the .yst + /3.0 hits have full-URL referrers. The 'original' Slurp never does.

Anyone else seeing this bad behavior?


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4360952.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com