homepage Welcome to WebmasterWorld Guest from 54.166.122.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

    
Yahoo Reports "Major" Crawl Improvements
engine

WebmasterWorld Administrator engine us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month Best Post Of The Month



 
Msg#: 3429088 posted 5:23 pm on Aug 22, 2007 (gmt 0)

...we're reducing our crawling machines by a significant amount with this update, so you might have observed a significant reduction in crawl load from us along with fewer machine IP addresses hitting you over the last few days.

The new crawler continues to improve comprehensiveness and freshness of coverage while also enhancing crawl efficiency, which reduces spurious load on websites.

Yahoo Reports "Major" Crawl Improvements [ysearchblog.com]

Earlier story
[webmasterworld.com...]

It's great news to hear that Yahoo are making efforts on crawl efficiency.

 

robotsdobetter

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3429088 posted 5:40 pm on Aug 22, 2007 (gmt 0)

That is good news, but can't say I have seen them reduce the crawling load on any of my web sites. Maybe after a week or so I'll see better results.

Rugles

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3429088 posted 5:58 pm on Aug 22, 2007 (gmt 0)

which reduces spurious load on websites

Well I like the sound of that, especially after receiving our bandwidth bill for the busiest part of our year.

rehabguy

10+ Year Member



 
Msg#: 3429088 posted 7:09 pm on Aug 22, 2007 (gmt 0)

Yahoo takes months to crawl and index a web site as it is - I don't see how this is good news to anyone!

It sounds like a step back to me... I'd rather see a press release from Yahoo that says "we will be indexing sites more quickly".

PS - I know the difference between crawling and being indexed... just wanted to clarify to save any educational attempts!

Asia_Expat

5+ Year Member



 
Msg#: 3429088 posted 7:42 pm on Aug 22, 2007 (gmt 0)

I don't know if it's significant... but my Yahoo traffic quadrupled today and now outstrips my Google traffic. I don't know if it's a blip but it appears to be an improvement in traffic from Yahoo across the board and not from a few specific keywords.

I don't normally keep much of an eye on what Yahoo is doing but of course this caught my attention. I can't think of anything I did recently that might account for it. I'm happy of course.

It also depends on what you call success. A year ago, I got maybe 30 or 40 referrals a day from all sources. Today I get around 450 per day from Google and about the same from Yahoo (as of this morning).
I get maybe 1 or 2 a day from MSN.
... Approx 30,000 unique IP's per month all told.

I made a custom 404 about 2 weeks ago and I started getting 404 hits form CGI pages in my forum last few days in Wembaster tools. I'm also moving to .xhtml extensions and MIME type but that's very gradual and only a few test pages just now, so I think that's irrelevant. I really can't think if any reason for the increase in Yahoo traffic, other than some kind of infrastructure change over there.

No particular reason for my post... make what you want of it.

[edited by: Asia_Expat at 7:44 pm (utc) on Aug. 22, 2007]

WiseWebDude

5+ Year Member



 
Msg#: 3429088 posted 8:04 pm on Aug 22, 2007 (gmt 0)

It seems as if they are getting a little better! I was so happy to see someone is alive at Yahoo that I seriously almost relieved myself where I sat. In the words of the Frankenstein's creator, "it's alive!"

I will be so happy for the Yahoo Slurp Spiders to put their beers down and quit partying on my site. HEHE.

:)

SE_Magnets

5+ Year Member



 
Msg#: 3429088 posted 9:31 pm on Aug 22, 2007 (gmt 0)

It looks like their crawler update has helped out with our indexing issue as well. We were recently working on an issue with Yahoo indexing our articles properly. We couldn't figure out what was going on because both Google and MSN had indexed them fine. So it looks like all is well now.

Thanks Yahoo!

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3429088 posted 10:17 pm on Aug 22, 2007 (gmt 0)

OK, this is completely bogus and helps nobody.

The number of IPs that Slurp uses? WHO CARES...

The fact that Yahoo has multiple crawlers for every division that crawl independently and don't share the common cache, now THAT's a problem that needs to be fixed.

This fix is served up with a major <yawn> garnished with a "why bother" and a big "BORING!" on the side.

BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3429088 posted 12:26 pm on Aug 23, 2007 (gmt 0)

We get 50% of our pages crawled every day.

marketingmagic

5+ Year Member



 
Msg#: 3429088 posted 4:24 pm on Aug 24, 2007 (gmt 0)

how about major search improvments yahoo?

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3429088 posted 6:39 pm on Aug 24, 2007 (gmt 0)

BTW, not trying to drift off topic, but thought I would add to the trivia of this trivial announcement....

Did anyone notice that it took Yahoo almost a full year just to implement DNS changes so that you could verify Slurp with round trip DNS checking?

The issue was raised for a method of 'bot verification in Aug '06 and Google was done in Sept '06 with both Ask and Live following suit by Nov '06. but it took our good buddies in Yahoo until June '07 to make the announcement that they had caught up with the pack.

Anyone ever implement reverse DNS entries?

It's just cut & paste IT grunt work, nothing special, no programming required, but it took almost a year.

Makes you go hmmmm....

jmccormac

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



 
Msg#: 3429088 posted 8:09 pm on Aug 24, 2007 (gmt 0)

Well Yahoo is a waste of bandwidth on one of my sites (a large directory). It hammers it almost every day and has only sent about 975 referrers this month. That is so low that I wonder if people even use Yahoo in Ireland for anything other than e-mail. I am strongly considering blocking it.

Regards...jmcc

centime

5+ Year Member



 
Msg#: 3429088 posted 10:36 pm on Aug 24, 2007 (gmt 0)

I hope Yahoo finds a way to convince more of it portal users to also use search,

The improvement is welcome, but I hope their indexing an caching speeds up

night707

5+ Year Member



 
Msg#: 3429088 posted 12:58 pm on Aug 25, 2007 (gmt 0)

I am strongly considering blocking it.

Same here with massive crawls at full force with even various ips involved.

We have plenty page 1 positions at Google and loads of rich media content, but Yahoo does not send any traffic.

marketingmagic

5+ Year Member



 
Msg#: 3429088 posted 1:29 pm on Aug 27, 2007 (gmt 0)

anyone have a list of Slurp IPs? We'd like to block them as well, complete waste of bandwidth.

BillyS

WebmasterWorld Senior Member billys us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3429088 posted 1:57 am on Aug 30, 2007 (gmt 0)

We will see if this makes any difference. Based on my calculations, Yahoo is due to announce an update.

rustybrick

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3429088 posted 12:01 pm on Aug 30, 2007 (gmt 0)

They announced an update with this crawl improvement.

Jakpot

10+ Year Member



 
Msg#: 3429088 posted 9:02 am on Aug 31, 2007 (gmt 0)

Did they announce they were deleting a huge number of pages from their index?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved