Yahoo Reports "Major" Crawl Improvements

[engine]

...we're reducing our crawling machines by a significant amount with this update, so you might have observed a significant reduction in crawl load from us along with fewer machine IP addresses hitting you over the last few days.

The new crawler continues to improve comprehensiveness and freshness of coverage while also enhancing crawl efficiency, which reduces spurious load on websites.

Yahoo Reports "Major" Crawl Improvements [ysearchblog.com]

Earlier story
[webmasterworld.com...]

It's great news to hear that Yahoo are making efforts on crawl efficiency.

[robotsdobetter]

That is good news, but can't say I have seen them reduce the crawling load on any of my web sites. Maybe after a week or so I'll see better results.

[Rugles]

which reduces spurious load on websites

Well I like the sound of that, especially after receiving our bandwidth bill for the busiest part of our year.

[rehabguy]

Yahoo takes months to crawl and index a web site as it is - I don't see how this is good news to anyone!

It sounds like a step back to me... I'd rather see a press release from Yahoo that says "we will be indexing sites more quickly".

PS - I know the difference between crawling and being indexed... just wanted to clarify to save any educational attempts!

[Asia_Expat]

I don't know if it's significant... but my Yahoo traffic quadrupled today and now outstrips my Google traffic. I don't know if it's a blip but it appears to be an improvement in traffic from Yahoo across the board and not from a few specific keywords.

I don't normally keep much of an eye on what Yahoo is doing but of course this caught my attention. I can't think of anything I did recently that might account for it. I'm happy of course.

It also depends on what you call success. A year ago, I got maybe 30 or 40 referrals a day from all sources. Today I get around 450 per day from Google and about the same from Yahoo (as of this morning).
I get maybe 1 or 2 a day from MSN.
... Approx 30,000 unique IP's per month all told.

I made a custom 404 about 2 weeks ago and I started getting 404 hits form CGI pages in my forum last few days in Wembaster tools. I'm also moving to .xhtml extensions and MIME type but that's very gradual and only a few test pages just now, so I think that's irrelevant. I really can't think if any reason for the increase in Yahoo traffic, other than some kind of infrastructure change over there.

No particular reason for my post... make what you want of it.

[edited by: Asia_Expat at 7:44 pm (utc) on Aug. 22, 2007]

[WiseWebDude]

It seems as if they are getting a little better! I was so happy to see someone is alive at Yahoo that I seriously almost relieved myself where I sat. In the words of the Frankenstein's creator, "it's alive!"

I will be so happy for the Yahoo Slurp Spiders to put their beers down and quit partying on my site. HEHE.

:)

[SE_Magnets]

It looks like their crawler update has helped out with our indexing issue as well. We were recently working on an issue with Yahoo indexing our articles properly. We couldn't figure out what was going on because both Google and MSN had indexed them fine. So it looks like all is well now.

Thanks Yahoo!

[incrediBILL]

OK, this is completely bogus and helps nobody.

The number of IPs that Slurp uses? WHO CARES...

The fact that Yahoo has multiple crawlers for every division that crawl independently and don't share the common cache, now THAT's a problem that needs to be fixed.

This fix is served up with a major <yawn> garnished with a "why bother" and a big "BORING!" on the side.

[BillyS]

We get 50% of our pages crawled every day.

[marketingmagic]

how about major search improvments yahoo?

[incrediBILL]

BTW, not trying to drift off topic, but thought I would add to the trivia of this trivial announcement....

Did anyone notice that it took Yahoo almost a full year just to implement DNS changes so that you could verify Slurp with round trip DNS checking?

The issue was raised for a method of 'bot verification in Aug '06 and Google was done in Sept '06 with both Ask and Live following suit by Nov '06. but it took our good buddies in Yahoo until June '07 to make the announcement that they had caught up with the pack.

Anyone ever implement reverse DNS entries?

It's just cut & paste IT grunt work, nothing special, no programming required, but it took almost a year.

Makes you go hmmmm....

[jmccormac]

Well Yahoo is a waste of bandwidth on one of my sites (a large directory). It hammers it almost every day and has only sent about 975 referrers this month. That is so low that I wonder if people even use Yahoo in Ireland for anything other than e-mail. I am strongly considering blocking it.

Regards...jmcc

[centime]

I hope Yahoo finds a way to convince more of it portal users to also use search,

The improvement is welcome, but I hope their indexing an caching speeds up

[night707]

I am strongly considering blocking it.

Same here with massive crawls at full force with even various ips involved.

We have plenty page 1 positions at Google and loads of rich media content, but Yahoo does not send any traffic.

[marketingmagic]

anyone have a list of Slurp IPs? We'd like to block them as well, complete waste of bandwidth.

[BillyS]

We will see if this makes any difference. Based on my calculations, Yahoo is due to announce an update.

[rustybrick]

They announced an update with this crawl improvement.

[Jakpot]

Did they announce they were deleting a huge number of pages from their index?