Welcome to WebmasterWorld Guest from 220.127.116.11 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Yahoo search is back "Slurp" has been visiting... iomfan Yahoo seems to be crawling again:
b100104.yse.yahoo.net - - [05/Feb/2014:20:26:20 +0000] "GET /robots.txt" 200 2551 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
Confirmed that b100104.yse.yahoo.net = 18.104.22.168 and vice versa. ;)
Slurp still crawls, but not for a general search engine. It crawls within specific niches for certain content areas within Yahoo! and it's partners. Mack. dstiles
No proper rDNS, no access, bot UA or not. iomfan
mack: Would you happen to know what those niches are? dstiles: Right - as with any other bot. :) Yahoo's host name format matches the one they used years ago, and - based on a sample of 1 ;) - I can say that rDNS appears to be working. If the past is any indication, Yahoo's spider has been way better behaved than Google's! Among rumours that Yahoo is considering returning to general crawling, we'll have to see how this pans out... dstiles
My several checks of yahoo bot IPs in DNS during the past couple of weeks shows they are no longer valid bot IPs. mack
mack: Would you happen to know what those niches are? Yahoo provide a little bit of this information on their Slurp web page. [ ...] help.yahoo.com Mack. iomfan
Yahoo provide a little bit of this information on their Slurp web page. help.yahoo.com... OK, thanks. It says there
Slurp collects content from partner sites Well, that makes searching my sites unnecessary, and slurp can be blocked like every other unwanted robot. ;)
Just a general note. It's probably not a good idea to block Slurp. Yahoo slices and dices Bing data. Slurps crawl is part of their in-house quality control and editorial process. https://help.yahoo.com/kb/search/slurp-crawling-page-sln22600.html [ help.yahoo.com]
It also accesses pages from sites across the Web to confirm accuracy and improve Yahoo's personalized content for our users. lucy24
How does it currently behave? I blocked Slurp some years ago because it seemed to pay no attention to robots.txt. But if it has begun conducting itself properly I'll let it back in-- at least from selected IP ranges. not2easy
I let it in one site and I have not seen it ignoring any robots disallows. Yet. I seldom see it requesting robots.txt, but it does. Keeping an eye on it.