homepage Welcome to WebmasterWorld Guest from 50.16.36.153
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Visit PubCon.com
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library : Charter : Moderators: martinibuster

Yahoo Search Engine and Directory Forum

    
Yahoo! Slurp is slurping up my site and a massive amount of bandwidth
endomorph1




msg:835167		 7:37 pm on Nov 17, 2005 (gmt 0)

Hi people,

Not posted in this section before, I normally stick to Google and Adsense forums.

I have been having a good look at my sites stats tonightand I think I should be concerned.

Yahoo! Slurp is coming to my site and spidering my site like crazy.

So far this month -

Page Views: 56,566
Visits: 51,290
Bandwidth: 2,715,409 Kb

Is this normal?

The worrying thing is they are taking all my bandwidth. At the going rate they will eat nearly 5 gig.

By a long long long way they are the busiest spider on my site. Google come in at 1,128 visits and MSN at 794 visits so far.

And does anyone know anything more about "OmniExplorer"?

They say on their site ... "Omni-Explorer is a venture-backed startup based in Silicon Valley. Stay tuned to this site; we plan on launching shortly".

Who are they?

Thanks for all replies.

 

orbiter




msg:835168		 1:27 am on Nov 18, 2005 (gmt 0)

endomorph1 --

I'm seeing the same thing. I was dropped from Yahoo back in early July. They've been crawling me to the tune of 30,000+ pages daily ever since. (Yes, 30,000 pages daily!) I have multiple dedicated servers, and they're tuned well enough to handle the load. Otherwise, I'd probably have to ban Slurp.

I keep hoping that all of this activity from Slurp is a good sign and that they will eventually reinclude my site. But I'm not showing up at all, even after filing a reinclusion request and after Yahoo's most recent index update. I don't know what to think anymore...

It's as if Yahoo and Google have both gone insane. But I suppose that's a topic for another thread.

-- Orbiter

jdMorgan




msg:835169		 2:25 am on Nov 18, 2005 (gmt 0)

Look into using robots.txt to control what directories and pages are crawled, and use the Crawl-delay parameter to slow down Slurp to a tolerable level.

Yahoo robots help page [help.yahoo.com]

Jim

endomorph1




msg:835170		 6:58 am on Nov 18, 2005 (gmt 0)

JD,

Thanks for that. My robots file now looks like -

User-agent: *
Disallow: /images/

User-agent: Slurp
Crawl-delay: 20

Do you think that is OK? I have not used robots file much so not sure of syntax.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved