Welcome to WebmasterWorld Guest from 54.162.172.144

Forum Moderators: martinibuster

Message Too Old, No Replies

Slurp In Your Logs?

Is the Yahoo search bot slurp still crawling webpages

     
12:41 pm on Aug 3, 2010 (gmt 0)

Junior Member from US 

10+ Year Member

joined:Dec 27, 2004
posts: 68
votes: 0


I was wondering if Yahoo Slurp was still crawling websites. Actually I wanted to see if the Yahoo Site Explorer database was still getting updated.
I checked the [ysearchblog.com ] and they have not posted an article since October 19, 2009
Is anyone still seeing activity from slurp?
2:15 pm on Aug 3, 2010 (gmt 0)

Junior Member from US 

10+ Year Member

joined:Dec 27, 2004
posts:68
votes: 0


Well, I was able to confirm that the bot is still alive:

2010-08-03 05:05:55 67.195.112.227 #*$!.#*$!.#*$!.#*$! 80 GET /robots.txt - 200 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)

So the question becomes, are they updating Yahoo Site Explorer and can we trust the back links found therein.
10:02 am on Aug 5, 2010 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 9, 2004
posts: 42
votes: 0


Can we be reasonably sure the crawled pages aren't used inside Yahoo or Bind ?

If so, I would want to block it to save the bandwidth (which is still consibrably large for me).
4:13 pm on Aug 5, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1761
votes: 43


I got links from forum to one of the sites I run shown in Yahoo Site Explorer. The post where that page is ref'd is from July 22nd. The page itself got hit by Slurp next day and in Site Explorer as well, as well as in Yahoo index(Ranking as #1 :)). Other than that, the bot is on the site daily.
6:52 pm on Aug 5, 2010 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member jab_creations is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 26, 2004
posts:3151
votes: 12


Slurp is still crawling because even the intentional and immoral monopolization of industries in order to reduce choice and screw over consumers takes time.

- John
5:23 am on Aug 6, 2010 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14648
votes: 94


Not only is it still alive, it crawled over 18K pages on my site today.
5:12 pm on Aug 7, 2010 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:6735
votes: 350


Bing and Yahoo are more active than Google... try harder? On a weekly basis these visits are < 2% of bandwidth so still happy to have them since > 22% of hits come from those two. And rising. Adopting a wait and see before inserting a crawl-delay.
8:12 pm on Aug 7, 2010 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:6735
votes: 350


Sorry... to late to amend a typo, er, lack of text:

Should read POINT TWO PERCENT... that missing decimal puts things into perspective.
12:16 am on Aug 10, 2010 (gmt 0)

New User

5+ Year Member

joined:July 22, 2010
posts:17
votes: 0


It could just be sending its info on to Bing. Its probably the same hardware and the same range of IP addresses so your server still recognizes it as the Yahoo bot.

The label didn't magically change when the Bing switchover took place.
12:44 am on Aug 10, 2010 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14648
votes: 94


The label didn't magically change when the Bing switchover took place.


I don't think the switchover is complete because I still see old Yahoo SERPs, not Bing SERPs.