Welcome to WebmasterWorld Guest from 54.227.14.23

Forum Moderators: martinibuster

Message Too Old, No Replies

Yahoo Reports "Major" Crawl Improvements

     
5:23 pm on Aug 22, 2007 (gmt 0)

Administrator from GB 

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month

joined:May 9, 2000
posts:23275
votes: 359


...we're reducing our crawling machines by a significant amount with this update, so you might have observed a significant reduction in crawl load from us along with fewer machine IP addresses hitting you over the last few days.

The new crawler continues to improve comprehensiveness and freshness of coverage while also enhancing crawl efficiency, which reduces spurious load on websites.

Yahoo Reports "Major" Crawl Improvements [ysearchblog.com]

Earlier story
[webmasterworld.com...]

It's great news to hear that Yahoo are making efforts on crawl efficiency.

5:40 pm on Aug 22, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 31, 2004
posts:710
votes: 0


That is good news, but can't say I have seen them reduce the crawling load on any of my web sites. Maybe after a week or so I'll see better results.
5:58 pm on Aug 22, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2001
posts:2145
votes: 0


which reduces spurious load on websites

Well I like the sound of that, especially after receiving our bandwidth bill for the busiest part of our year.

7:09 pm on Aug 22, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 10, 2003
posts:351
votes: 0


Yahoo takes months to crawl and index a web site as it is - I don't see how this is good news to anyone!

It sounds like a step back to me... I'd rather see a press release from Yahoo that says "we will be indexing sites more quickly".

PS - I know the difference between crawling and being indexed... just wanted to clarify to save any educational attempts!

7:42 pm on Aug 22, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 7, 2005
posts:636
votes: 0


I don't know if it's significant... but my Yahoo traffic quadrupled today and now outstrips my Google traffic. I don't know if it's a blip but it appears to be an improvement in traffic from Yahoo across the board and not from a few specific keywords.

I don't normally keep much of an eye on what Yahoo is doing but of course this caught my attention. I can't think of anything I did recently that might account for it. I'm happy of course.

It also depends on what you call success. A year ago, I got maybe 30 or 40 referrals a day from all sources. Today I get around 450 per day from Google and about the same from Yahoo (as of this morning).
I get maybe 1 or 2 a day from MSN.
... Approx 30,000 unique IP's per month all told.

I made a custom 404 about 2 weeks ago and I started getting 404 hits form CGI pages in my forum last few days in Wembaster tools. I'm also moving to .xhtml extensions and MIME type but that's very gradual and only a few test pages just now, so I think that's irrelevant. I really can't think if any reason for the increase in Yahoo traffic, other than some kind of infrastructure change over there.

No particular reason for my post... make what you want of it.

[edited by: Asia_Expat at 7:44 pm (utc) on Aug. 22, 2007]

8:04 pm on Aug 22, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:June 8, 2006
posts:556
votes: 0


It seems as if they are getting a little better! I was so happy to see someone is alive at Yahoo that I seriously almost relieved myself where I sat. In the words of the Frankenstein's creator, "it's alive!"

I will be so happy for the Yahoo Slurp Spiders to put their beers down and quit partying on my site. HEHE.

:)

9:31 pm on Aug 22, 2007 (gmt 0)

New User

5+ Year Member

joined:Mar 22, 2007
posts:11
votes: 0


It looks like their crawler update has helped out with our indexing issue as well. We were recently working on an issue with Yahoo indexing our articles properly. We couldn't figure out what was going on because both Google and MSN had indexed them fine. So it looks like all is well now.

Thanks Yahoo!

10:17 pm on Aug 22, 2007 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


OK, this is completely bogus and helps nobody.

The number of IPs that Slurp uses? WHO CARES...

The fact that Yahoo has multiple crawlers for every division that crawl independently and don't share the common cache, now THAT's a problem that needs to be fixed.

This fix is served up with a major <yawn> garnished with a "why bother" and a big "BORING!" on the side.

12:26 pm on Aug 23, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 1, 2004
posts:3181
votes: 0


We get 50% of our pages crawled every day.
4:24 pm on Aug 24, 2007 (gmt 0)

Full Member

10+ Year Member

joined:Oct 25, 2005
posts:307
votes: 0


how about major search improvments yahoo?
6:39 pm on Aug 24, 2007 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


BTW, not trying to drift off topic, but thought I would add to the trivia of this trivial announcement....

Did anyone notice that it took Yahoo almost a full year just to implement DNS changes so that you could verify Slurp with round trip DNS checking?

The issue was raised for a method of 'bot verification in Aug '06 and Google was done in Sept '06 with both Ask and Live following suit by Nov '06. but it took our good buddies in Yahoo until June '07 to make the announcement that they had caught up with the pack.

Anyone ever implement reverse DNS entries?

It's just cut & paste IT grunt work, nothing special, no programming required, but it took almost a year.

Makes you go hmmmm....

8:09 pm on Aug 24, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 30, 2002
posts:2510
votes: 44


Well Yahoo is a waste of bandwidth on one of my sites (a large directory). It hammers it almost every day and has only sent about 975 referrers this month. That is so low that I wonder if people even use Yahoo in Ireland for anything other than e-mail. I am strongly considering blocking it.

Regards...jmcc

10:36 pm on Aug 24, 2007 (gmt 0)

Preferred Member

5+ Year Member

joined:Dec 26, 2006
posts:535
votes: 0


I hope Yahoo finds a way to convince more of it portal users to also use search,

The improvement is welcome, but I hope their indexing an caching speeds up

12:58 pm on Aug 25, 2007 (gmt 0)

Preferred Member

10+ Year Member Top Contributors Of The Month

joined:Feb 10, 2005
posts:489
votes: 0


I am strongly considering blocking it.

Same here with massive crawls at full force with even various ips involved.

We have plenty page 1 positions at Google and loads of rich media content, but Yahoo does not send any traffic.

1:29 pm on Aug 27, 2007 (gmt 0)

Full Member

10+ Year Member

joined:Oct 25, 2005
posts:307
votes: 0


anyone have a list of Slurp IPs? We'd like to block them as well, complete waste of bandwidth.
1:57 am on Aug 30, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 1, 2004
posts:3181
votes: 0


We will see if this makes any difference. Based on my calculations, Yahoo is due to announce an update.
12:01 pm on Aug 30, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member

joined:June 12, 2003
posts:710
votes: 14


They announced an update with this crawl improvement.
9:02 am on Aug 31, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 31, 2002
posts:353
votes: 0


Did they announce they were deleting a huge number of pages from their index?