Welcome to WebmasterWorld Guest from 54.163.168.15

Forum Moderators: martinibuster

Yahoo Reports "Major" Crawl Improvements

   
5:23 pm on Aug 22, 2007 (gmt 0)

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month



...we're reducing our crawling machines by a significant amount with this update, so you might have observed a significant reduction in crawl load from us along with fewer machine IP addresses hitting you over the last few days.

The new crawler continues to improve comprehensiveness and freshness of coverage while also enhancing crawl efficiency, which reduces spurious load on websites.

Yahoo Reports "Major" Crawl Improvements [ysearchblog.com]

Earlier story
[webmasterworld.com...]

It's great news to hear that Yahoo are making efforts on crawl efficiency.

5:40 pm on Aug 22, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



That is good news, but can't say I have seen them reduce the crawling load on any of my web sites. Maybe after a week or so I'll see better results.
5:58 pm on Aug 22, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



which reduces spurious load on websites

Well I like the sound of that, especially after receiving our bandwidth bill for the busiest part of our year.

7:09 pm on Aug 22, 2007 (gmt 0)

10+ Year Member



Yahoo takes months to crawl and index a web site as it is - I don't see how this is good news to anyone!

It sounds like a step back to me... I'd rather see a press release from Yahoo that says "we will be indexing sites more quickly".

PS - I know the difference between crawling and being indexed... just wanted to clarify to save any educational attempts!

7:42 pm on Aug 22, 2007 (gmt 0)

5+ Year Member



I don't know if it's significant... but my Yahoo traffic quadrupled today and now outstrips my Google traffic. I don't know if it's a blip but it appears to be an improvement in traffic from Yahoo across the board and not from a few specific keywords.

I don't normally keep much of an eye on what Yahoo is doing but of course this caught my attention. I can't think of anything I did recently that might account for it. I'm happy of course.

It also depends on what you call success. A year ago, I got maybe 30 or 40 referrals a day from all sources. Today I get around 450 per day from Google and about the same from Yahoo (as of this morning).
I get maybe 1 or 2 a day from MSN.
... Approx 30,000 unique IP's per month all told.

I made a custom 404 about 2 weeks ago and I started getting 404 hits form CGI pages in my forum last few days in Wembaster tools. I'm also moving to .xhtml extensions and MIME type but that's very gradual and only a few test pages just now, so I think that's irrelevant. I really can't think if any reason for the increase in Yahoo traffic, other than some kind of infrastructure change over there.

No particular reason for my post... make what you want of it.

[edited by: Asia_Expat at 7:44 pm (utc) on Aug. 22, 2007]

8:04 pm on Aug 22, 2007 (gmt 0)

5+ Year Member



It seems as if they are getting a little better! I was so happy to see someone is alive at Yahoo that I seriously almost relieved myself where I sat. In the words of the Frankenstein's creator, "it's alive!"

I will be so happy for the Yahoo Slurp Spiders to put their beers down and quit partying on my site. HEHE.

:)

9:31 pm on Aug 22, 2007 (gmt 0)

5+ Year Member



It looks like their crawler update has helped out with our indexing issue as well. We were recently working on an issue with Yahoo indexing our articles properly. We couldn't figure out what was going on because both Google and MSN had indexed them fine. So it looks like all is well now.

Thanks Yahoo!

10:17 pm on Aug 22, 2007 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



OK, this is completely bogus and helps nobody.

The number of IPs that Slurp uses? WHO CARES...

The fact that Yahoo has multiple crawlers for every division that crawl independently and don't share the common cache, now THAT's a problem that needs to be fixed.

This fix is served up with a major <yawn> garnished with a "why bother" and a big "BORING!" on the side.

12:26 pm on Aug 23, 2007 (gmt 0)

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member



We get 50% of our pages crawled every day.
4:24 pm on Aug 24, 2007 (gmt 0)

5+ Year Member



how about major search improvments yahoo?
6:39 pm on Aug 24, 2007 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



BTW, not trying to drift off topic, but thought I would add to the trivia of this trivial announcement....

Did anyone notice that it took Yahoo almost a full year just to implement DNS changes so that you could verify Slurp with round trip DNS checking?

The issue was raised for a method of 'bot verification in Aug '06 and Google was done in Sept '06 with both Ask and Live following suit by Nov '06. but it took our good buddies in Yahoo until June '07 to make the announcement that they had caught up with the pack.

Anyone ever implement reverse DNS entries?

It's just cut & paste IT grunt work, nothing special, no programming required, but it took almost a year.

Makes you go hmmmm....

8:09 pm on Aug 24, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Well Yahoo is a waste of bandwidth on one of my sites (a large directory). It hammers it almost every day and has only sent about 975 referrers this month. That is so low that I wonder if people even use Yahoo in Ireland for anything other than e-mail. I am strongly considering blocking it.

Regards...jmcc

10:36 pm on Aug 24, 2007 (gmt 0)

5+ Year Member



I hope Yahoo finds a way to convince more of it portal users to also use search,

The improvement is welcome, but I hope their indexing an caching speeds up

12:58 pm on Aug 25, 2007 (gmt 0)

10+ Year Member



I am strongly considering blocking it.

Same here with massive crawls at full force with even various ips involved.

We have plenty page 1 positions at Google and loads of rich media content, but Yahoo does not send any traffic.

1:29 pm on Aug 27, 2007 (gmt 0)

5+ Year Member



anyone have a list of Slurp IPs? We'd like to block them as well, complete waste of bandwidth.
1:57 am on Aug 30, 2007 (gmt 0)

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member



We will see if this makes any difference. Based on my calculations, Yahoo is due to announce an update.
12:01 pm on Aug 30, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



They announced an update with this crawl improvement.
9:02 am on Aug 31, 2007 (gmt 0)

10+ Year Member



Did they announce they were deleting a huge number of pages from their index?
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month