homepage Welcome to WebmasterWorld Guest from 54.198.148.191
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

    
Slurp In Your Logs?
Is the Yahoo search bot slurp still crawling webpages
Drew




msg:4181110
 12:41 pm on Aug 3, 2010 (gmt 0)

I was wondering if Yahoo Slurp was still crawling websites. Actually I wanted to see if the Yahoo Site Explorer database was still getting updated.
I checked the [ysearchblog.com ] and they have not posted an article since October 19, 2009
Is anyone still seeing activity from slurp?

 

Drew




msg:4181163
 2:15 pm on Aug 3, 2010 (gmt 0)

Well, I was able to confirm that the bot is still alive:

2010-08-03 05:05:55 67.195.112.227 #*$!.#*$!.#*$!.#*$! 80 GET /robots.txt - 200 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)

So the question becomes, are they updating Yahoo Site Explorer and can we trust the back links found therein.

GeneVincent




msg:4182343
 10:02 am on Aug 5, 2010 (gmt 0)

Can we be reasonably sure the crawled pages aren't used inside Yahoo or Bind ?

If so, I would want to block it to save the bandwidth (which is still consibrably large for me).

blend27




msg:4182556
 4:13 pm on Aug 5, 2010 (gmt 0)

I got links from forum to one of the sites I run shown in Yahoo Site Explorer. The post where that page is ref'd is from July 22nd. The page itself got hit by Slurp next day and in Site Explorer as well, as well as in Yahoo index(Ranking as #1 :)). Other than that, the bot is on the site daily.

JAB Creations




msg:4182683
 6:52 pm on Aug 5, 2010 (gmt 0)

Slurp is still crawling because even the intentional and immoral monopolization of industries in order to reduce choice and screw over consumers takes time.

- John

incrediBILL




msg:4182951
 5:23 am on Aug 6, 2010 (gmt 0)

Not only is it still alive, it crawled over 18K pages on my site today.

tangor




msg:4183651
 5:12 pm on Aug 7, 2010 (gmt 0)

Bing and Yahoo are more active than Google... try harder? On a weekly basis these visits are < 2% of bandwidth so still happy to have them since > 22% of hits come from those two. And rising. Adopting a wait and see before inserting a crawl-delay.

tangor




msg:4183688
 8:12 pm on Aug 7, 2010 (gmt 0)

Sorry... to late to amend a typo, er, lack of text:

Should read POINT TWO PERCENT... that missing decimal puts things into perspective.

Azimuth




msg:4184530
 12:16 am on Aug 10, 2010 (gmt 0)

It could just be sending its info on to Bing. Its probably the same hardware and the same range of IP addresses so your server still recognizes it as the Yahoo bot.

The label didn't magically change when the Bing switchover took place.

incrediBILL




msg:4184541
 12:44 am on Aug 10, 2010 (gmt 0)

The label didn't magically change when the Bing switchover took place.


I don't think the switchover is complete because I still see old Yahoo SERPs, not Bing SERPs.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved