Welcome to WebmasterWorld Guest from 174.129.135.89

Forum Moderators: martinibuster

Message Too Old, No Replies

Slurp In Your Logs?

Is the Yahoo search bot slurp still crawling webpages

   
12:41 pm on Aug 3, 2010 (gmt 0)

10+ Year Member



I was wondering if Yahoo Slurp was still crawling websites. Actually I wanted to see if the Yahoo Site Explorer database was still getting updated.
I checked the [ysearchblog.com ] and they have not posted an article since October 19, 2009
Is anyone still seeing activity from slurp?
2:15 pm on Aug 3, 2010 (gmt 0)

10+ Year Member



Well, I was able to confirm that the bot is still alive:

2010-08-03 05:05:55 67.195.112.227 #*$!.#*$!.#*$!.#*$! 80 GET /robots.txt - 200 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)

So the question becomes, are they updating Yahoo Site Explorer and can we trust the back links found therein.
10:02 am on Aug 5, 2010 (gmt 0)

10+ Year Member



Can we be reasonably sure the crawled pages aren't used inside Yahoo or Bind ?

If so, I would want to block it to save the bandwidth (which is still consibrably large for me).
4:13 pm on Aug 5, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I got links from forum to one of the sites I run shown in Yahoo Site Explorer. The post where that page is ref'd is from July 22nd. The page itself got hit by Slurp next day and in Site Explorer as well, as well as in Yahoo index(Ranking as #1 :)). Other than that, the bot is on the site daily.
6:52 pm on Aug 5, 2010 (gmt 0)

WebmasterWorld Senior Member jab_creations is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Slurp is still crawling because even the intentional and immoral monopolization of industries in order to reduce choice and screw over consumers takes time.

- John
5:23 am on Aug 6, 2010 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Not only is it still alive, it crawled over 18K pages on my site today.
5:12 pm on Aug 7, 2010 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Bing and Yahoo are more active than Google... try harder? On a weekly basis these visits are < 2% of bandwidth so still happy to have them since > 22% of hits come from those two. And rising. Adopting a wait and see before inserting a crawl-delay.
8:12 pm on Aug 7, 2010 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Sorry... to late to amend a typo, er, lack of text:

Should read POINT TWO PERCENT... that missing decimal puts things into perspective.
12:16 am on Aug 10, 2010 (gmt 0)



It could just be sending its info on to Bing. Its probably the same hardware and the same range of IP addresses so your server still recognizes it as the Yahoo bot.

The label didn't magically change when the Bing switchover took place.
12:44 am on Aug 10, 2010 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The label didn't magically change when the Bing switchover took place.


I don't think the switchover is complete because I still see old Yahoo SERPs, not Bing SERPs.