homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Visit PubCon.com
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

Slurp In Your Logs?
Is the Yahoo search bot slurp still crawling webpages

 12:41 pm on Aug 3, 2010 (gmt 0)

I was wondering if Yahoo Slurp was still crawling websites. Actually I wanted to see if the Yahoo Site Explorer database was still getting updated.
I checked the [ysearchblog.com ] and they have not posted an article since October 19, 2009
Is anyone still seeing activity from slurp?



 2:15 pm on Aug 3, 2010 (gmt 0)

Well, I was able to confirm that the bot is still alive:

2010-08-03 05:05:55 #*$!.#*$!.#*$!.#*$! 80 GET /robots.txt - 200 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)

So the question becomes, are they updating Yahoo Site Explorer and can we trust the back links found therein.


 10:02 am on Aug 5, 2010 (gmt 0)

Can we be reasonably sure the crawled pages aren't used inside Yahoo or Bind ?

If so, I would want to block it to save the bandwidth (which is still consibrably large for me).


 4:13 pm on Aug 5, 2010 (gmt 0)

I got links from forum to one of the sites I run shown in Yahoo Site Explorer. The post where that page is ref'd is from July 22nd. The page itself got hit by Slurp next day and in Site Explorer as well, as well as in Yahoo index(Ranking as #1 :)). Other than that, the bot is on the site daily.

JAB Creations

 6:52 pm on Aug 5, 2010 (gmt 0)

Slurp is still crawling because even the intentional and immoral monopolization of industries in order to reduce choice and screw over consumers takes time.

- John


 5:23 am on Aug 6, 2010 (gmt 0)

Not only is it still alive, it crawled over 18K pages on my site today.


 5:12 pm on Aug 7, 2010 (gmt 0)

Bing and Yahoo are more active than Google... try harder? On a weekly basis these visits are < 2% of bandwidth so still happy to have them since > 22% of hits come from those two. And rising. Adopting a wait and see before inserting a crawl-delay.


 8:12 pm on Aug 7, 2010 (gmt 0)

Sorry... to late to amend a typo, er, lack of text:

Should read POINT TWO PERCENT... that missing decimal puts things into perspective.


 12:16 am on Aug 10, 2010 (gmt 0)

It could just be sending its info on to Bing. Its probably the same hardware and the same range of IP addresses so your server still recognizes it as the Yahoo bot.

The label didn't magically change when the Bing switchover took place.


 12:44 am on Aug 10, 2010 (gmt 0)

The label didn't magically change when the Bing switchover took place.

I don't think the switchover is complete because I still see old Yahoo SERPs, not Bing SERPs.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved