Yahoo search is back

"Slurp" has been visiting...

[iomfan]

Yahoo seems to be crawling again:

b100104.yse.yahoo.net - - [05/Feb/2014:20:26:20 +0000] "GET /robots.txt" 200 2551 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

Confirmed that b100104.yse.yahoo.net = 68.180.224.228 and vice versa. ;)

[mack]

Slurp still crawls, but not for a general search engine. It crawls within specific niches for certain content areas within Yahoo! and it's partners.

Mack.

[dstiles]

No proper rDNS, no access, bot UA or not.

[iomfan]

mack:
Would you happen to know what those niches are?

dstiles:
Right - as with any other bot. :) Yahoo's host name format matches the one they used years ago, and - based on a sample of 1 ;) - I can say that rDNS appears to be working. If the past is any indication, Yahoo's spider has been way better behaved than Google's! Among rumours that Yahoo is considering returning to general crawling, we'll have to see how this pans out...

[dstiles]

My several checks of yahoo bot IPs in DNS during the past couple of weeks shows they are no longer valid bot IPs.

[mack]

mack:
Would you happen to know what those niches are?

Yahoo provide a little bit of this information on their Slurp web page.

[help.yahoo.com...]

Mack.

[iomfan]

mack:

Yahoo provide a little bit of this information on their Slurp web page. help.yahoo.com...

OK, thanks. It says there

Slurp collects content from partner sites

Well, that makes searching my sites unnecessary, and slurp can be blocked like every other unwanted robot. ;)

[martinibuster]

Just a general note. It's probably not a good idea to block Slurp. Yahoo slices and dices Bing data. Slurps crawl is part of their in-house quality control and editorial process.

https://help.yahoo.com/kb/search/slurp-crawling-page-sln22600.html [help.yahoo.com]

It also accesses pages from sites across the Web to confirm accuracy and improve Yahoo's personalized content for our users.

[lucy24]

How does it currently behave? I blocked Slurp some years ago because it seemed to pay no attention to robots.txt. But if it has begun conducting itself properly I'll let it back in-- at least from selected IP ranges.

[not2easy]

I let it in one site and I have not seen it ignoring any robots disallows. Yet. I seldom see it requesting robots.txt, but it does. Keeping an eye on it.