homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

Yahoo Reports "Major" Crawl Improvements

 5:23 pm on Aug 22, 2007 (gmt 0)

...we're reducing our crawling machines by a significant amount with this update, so you might have observed a significant reduction in crawl load from us along with fewer machine IP addresses hitting you over the last few days.

The new crawler continues to improve comprehensiveness and freshness of coverage while also enhancing crawl efficiency, which reduces spurious load on websites.

Yahoo Reports "Major" Crawl Improvements [ysearchblog.com]

Earlier story

It's great news to hear that Yahoo are making efforts on crawl efficiency.



 5:40 pm on Aug 22, 2007 (gmt 0)

That is good news, but can't say I have seen them reduce the crawling load on any of my web sites. Maybe after a week or so I'll see better results.


 5:58 pm on Aug 22, 2007 (gmt 0)

which reduces spurious load on websites

Well I like the sound of that, especially after receiving our bandwidth bill for the busiest part of our year.


 7:09 pm on Aug 22, 2007 (gmt 0)

Yahoo takes months to crawl and index a web site as it is - I don't see how this is good news to anyone!

It sounds like a step back to me... I'd rather see a press release from Yahoo that says "we will be indexing sites more quickly".

PS - I know the difference between crawling and being indexed... just wanted to clarify to save any educational attempts!


 7:42 pm on Aug 22, 2007 (gmt 0)

I don't know if it's significant... but my Yahoo traffic quadrupled today and now outstrips my Google traffic. I don't know if it's a blip but it appears to be an improvement in traffic from Yahoo across the board and not from a few specific keywords.

I don't normally keep much of an eye on what Yahoo is doing but of course this caught my attention. I can't think of anything I did recently that might account for it. I'm happy of course.

It also depends on what you call success. A year ago, I got maybe 30 or 40 referrals a day from all sources. Today I get around 450 per day from Google and about the same from Yahoo (as of this morning).
I get maybe 1 or 2 a day from MSN.
... Approx 30,000 unique IP's per month all told.

I made a custom 404 about 2 weeks ago and I started getting 404 hits form CGI pages in my forum last few days in Wembaster tools. I'm also moving to .xhtml extensions and MIME type but that's very gradual and only a few test pages just now, so I think that's irrelevant. I really can't think if any reason for the increase in Yahoo traffic, other than some kind of infrastructure change over there.

No particular reason for my post... make what you want of it.

[edited by: Asia_Expat at 7:44 pm (utc) on Aug. 22, 2007]


 8:04 pm on Aug 22, 2007 (gmt 0)

It seems as if they are getting a little better! I was so happy to see someone is alive at Yahoo that I seriously almost relieved myself where I sat. In the words of the Frankenstein's creator, "it's alive!"

I will be so happy for the Yahoo Slurp Spiders to put their beers down and quit partying on my site. HEHE.



 9:31 pm on Aug 22, 2007 (gmt 0)

It looks like their crawler update has helped out with our indexing issue as well. We were recently working on an issue with Yahoo indexing our articles properly. We couldn't figure out what was going on because both Google and MSN had indexed them fine. So it looks like all is well now.

Thanks Yahoo!


 10:17 pm on Aug 22, 2007 (gmt 0)

OK, this is completely bogus and helps nobody.

The number of IPs that Slurp uses? WHO CARES...

The fact that Yahoo has multiple crawlers for every division that crawl independently and don't share the common cache, now THAT's a problem that needs to be fixed.

This fix is served up with a major <yawn> garnished with a "why bother" and a big "BORING!" on the side.


 12:26 pm on Aug 23, 2007 (gmt 0)

We get 50% of our pages crawled every day.


 4:24 pm on Aug 24, 2007 (gmt 0)

how about major search improvments yahoo?


 6:39 pm on Aug 24, 2007 (gmt 0)

BTW, not trying to drift off topic, but thought I would add to the trivia of this trivial announcement....

Did anyone notice that it took Yahoo almost a full year just to implement DNS changes so that you could verify Slurp with round trip DNS checking?

The issue was raised for a method of 'bot verification in Aug '06 and Google was done in Sept '06 with both Ask and Live following suit by Nov '06. but it took our good buddies in Yahoo until June '07 to make the announcement that they had caught up with the pack.

Anyone ever implement reverse DNS entries?

It's just cut & paste IT grunt work, nothing special, no programming required, but it took almost a year.

Makes you go hmmmm....


 8:09 pm on Aug 24, 2007 (gmt 0)

Well Yahoo is a waste of bandwidth on one of my sites (a large directory). It hammers it almost every day and has only sent about 975 referrers this month. That is so low that I wonder if people even use Yahoo in Ireland for anything other than e-mail. I am strongly considering blocking it.



 10:36 pm on Aug 24, 2007 (gmt 0)

I hope Yahoo finds a way to convince more of it portal users to also use search,

The improvement is welcome, but I hope their indexing an caching speeds up


 12:58 pm on Aug 25, 2007 (gmt 0)

I am strongly considering blocking it.

Same here with massive crawls at full force with even various ips involved.

We have plenty page 1 positions at Google and loads of rich media content, but Yahoo does not send any traffic.


 1:29 pm on Aug 27, 2007 (gmt 0)

anyone have a list of Slurp IPs? We'd like to block them as well, complete waste of bandwidth.


 1:57 am on Aug 30, 2007 (gmt 0)

We will see if this makes any difference. Based on my calculations, Yahoo is due to announce an update.


 12:01 pm on Aug 30, 2007 (gmt 0)

They announced an update with this crawl improvement.


 9:02 am on Aug 31, 2007 (gmt 0)

Did they announce they were deleting a huge number of pages from their index?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved