Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: martinibuster
...we're reducing our crawling machines by a significant amount with this update, so you might have observed a significant reduction in crawl load from us along with fewer machine IP addresses hitting you over the last few days.
The new crawler continues to improve comprehensiveness and freshness of coverage while also enhancing crawl efficiency, which reduces spurious load on websites.
Yahoo Reports "Major" Crawl Improvements [ysearchblog.com]
It's great news to hear that Yahoo are making efforts on crawl efficiency.
It sounds like a step back to me... I'd rather see a press release from Yahoo that says "we will be indexing sites more quickly".
PS - I know the difference between crawling and being indexed... just wanted to clarify to save any educational attempts!
I don't normally keep much of an eye on what Yahoo is doing but of course this caught my attention. I can't think of anything I did recently that might account for it. I'm happy of course.
It also depends on what you call success. A year ago, I got maybe 30 or 40 referrals a day from all sources. Today I get around 450 per day from Google and about the same from Yahoo (as of this morning).
I get maybe 1 or 2 a day from MSN.
... Approx 30,000 unique IP's per month all told.
I made a custom 404 about 2 weeks ago and I started getting 404 hits form CGI pages in my forum last few days in Wembaster tools. I'm also moving to .xhtml extensions and MIME type but that's very gradual and only a few test pages just now, so I think that's irrelevant. I really can't think if any reason for the increase in Yahoo traffic, other than some kind of infrastructure change over there.
No particular reason for my post... make what you want of it.
[edited by: Asia_Expat at 7:44 pm (utc) on Aug. 22, 2007]
I will be so happy for the Yahoo Slurp Spiders to put their beers down and quit partying on my site. HEHE.
The number of IPs that Slurp uses? WHO CARES...
The fact that Yahoo has multiple crawlers for every division that crawl independently and don't share the common cache, now THAT's a problem that needs to be fixed.
This fix is served up with a major <yawn> garnished with a "why bother" and a big "BORING!" on the side.
Did anyone notice that it took Yahoo almost a full year just to implement DNS changes so that you could verify Slurp with round trip DNS checking?
The issue was raised for a method of 'bot verification in Aug '06 and Google was done in Sept '06 with both Ask and Live following suit by Nov '06. but it took our good buddies in Yahoo until June '07 to make the announcement that they had caught up with the pack.
Anyone ever implement reverse DNS entries?
It's just cut & paste IT grunt work, nothing special, no programming required, but it took almost a year.
Makes you go hmmmm....