homepage Welcome to WebmasterWorld Guest from 23.21.9.44
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

    
Yahoo search is back
"Slurp" has been visiting...
iomfan




msg:4642834
 10:41 pm on Feb 5, 2014 (gmt 0)

Yahoo seems to be crawling again:

b100104.yse.yahoo.net - - [05/Feb/2014:20:26:20 +0000] "GET /robots.txt" 200 2551 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

Confirmed that b100104.yse.yahoo.net = 68.180.224.228 and vice versa. ;)

 

mack




msg:4644049
 12:59 am on Feb 11, 2014 (gmt 0)

Slurp still crawls, but not for a general search engine. It crawls within specific niches for certain content areas within Yahoo! and it's partners.

Mack.

dstiles




msg:4644318
 8:36 pm on Feb 11, 2014 (gmt 0)

No proper rDNS, no access, bot UA or not.

iomfan




msg:4644332
 9:26 pm on Feb 11, 2014 (gmt 0)

mack:
Would you happen to know what those niches are?

dstiles:
Right - as with any other bot. :) Yahoo's host name format matches the one they used years ago, and - based on a sample of 1 ;) - I can say that rDNS appears to be working. If the past is any indication, Yahoo's spider has been way better behaved than Google's! Among rumours that Yahoo is considering returning to general crawling, we'll have to see how this pans out...

dstiles




msg:4644570
 9:57 pm on Feb 12, 2014 (gmt 0)

My several checks of yahoo bot IPs in DNS during the past couple of weeks shows they are no longer valid bot IPs.

mack




msg:4644997
 3:18 am on Feb 14, 2014 (gmt 0)

mack:
Would you happen to know what those niches are?


Yahoo provide a little bit of this information on their Slurp web page.

[help.yahoo.com...]

Mack.

iomfan




msg:4664536
 11:45 am on Apr 20, 2014 (gmt 0)

mack:
Yahoo provide a little bit of this information on their Slurp web page. help.yahoo.com...

OK, thanks. It says there
Slurp collects content from partner sites

Well, that makes searching my sites unnecessary, and slurp can be blocked like every other unwanted robot. ;)

martinibuster




msg:4664545
 2:35 pm on Apr 20, 2014 (gmt 0)

Just a general note. It's probably not a good idea to block Slurp. Yahoo slices and dices Bing data. Slurps crawl is part of their in-house quality control and editorial process.

https://help.yahoo.com/kb/search/slurp-crawling-page-sln22600.html [help.yahoo.com]

It also accesses pages from sites across the Web to confirm accuracy and improve Yahoo's personalized content for our users.

lucy24




msg:4664567
 5:52 pm on Apr 20, 2014 (gmt 0)

How does it currently behave? I blocked Slurp some years ago because it seemed to pay no attention to robots.txt. But if it has begun conducting itself properly I'll let it back in-- at least from selected IP ranges.

not2easy




msg:4664575
 7:11 pm on Apr 20, 2014 (gmt 0)

I let it in one site and I have not seen it ignoring any robots disallows. Yet. I seldom see it requesting robots.txt, but it does. Keeping an eye on it.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved