homepage Welcome to WebmasterWorld Guest from 54.166.105.24
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Ask - Teoma
Forum Library, Charter, Moderator: open

Ask - Teoma Forum

    
Ask Jeeves hit pretty hard
carfac




msg:32294
 5:24 pm on Oct 9, 2002 (gmt 0)

Got hit (2k+) pretty hard by Ask Jeeves yesterday...

Bad timing- they hit me at about noon... and very excessive... 3-5 hits per second with NO LET UP for a couple hours.

Is this normal for ask jeeves? Is this excessive for a bot?

dave

 

fiestagirl




msg:32295
 6:37 pm on Oct 9, 2002 (gmt 0)

That seems pretty excessive for the ask jeeves bot. The only time I've ever seen them do that is when they got caught up in a loop (our fault). Most of the time they plod along at about one request every 5 minutes.

As for the other robots - I've seen googlebot make requests that fast. If you can dish it up they will take it. But the bot seems smart enough to slow down if you can't serve the requests that quickly.

jeremy goodrich




msg:32296
 8:26 pm on Oct 9, 2002 (gmt 0)

That is very excessive. Can you give us any more information on the pages? Are they static, or dynamic? And what % of your site did Ask spider?

Ask Jeeves spider came 'round a few sites yesterday for me, but mostly grabbed robots.txt and left. To date, even on larger sites, I haven't seen their bot do this, though I haven't watched a larger scale dynamic site when the bot struck to know that it wouldn't happen in that situation...

carfac




msg:32297
 11:01 pm on Oct 9, 2002 (gmt 0)

Jeremy:

So I do not risk posting any "personal" info, I will sticky you seperately...

I would say they grabbed 5% or so of the site... it is a REALLY big site. Some is dynamic, some static...

dave

jeremy goodrich




msg:32298
 11:04 pm on Oct 9, 2002 (gmt 0)

That's surprising they spidered static stuff that way...hm. Odd.

Wonder if it means they are testing something new with their bot? Have you contacted them to let them know about the problem?

The one time or two I've done similar, it was with good results...engines can be responsive when it makes their product better.

(feel free to sticky if you like, but that was all I really wanted to know.) :)

And if you do contact them, please let us know what the response is!

carfac




msg:32299
 11:14 pm on Oct 9, 2002 (gmt 0)

Jeremy:

OK, whoops! I sticked you some logs- just ignore if you want.

I will contact them, but I do not want to run the risk of getting banned. Is that possible? I conrtacted FAST once about a robot ignoring robots.txt instructions, and they said, "OK, we will put your site into our exclusions list." Noooooo!

I did also find this robot go into a loop looking for something that it kept requesting 5 times a second for at least two minutes.... and the file is SPECIFICALLY EXCLUDED in robots.txt.

Perhaps they do need an e-mail!

dave

carfac




msg:32300
 11:32 pm on Oct 9, 2002 (gmt 0)

Hi:

OK, I send a nicely worded letter on their web feedback form... I guess I gotta hope it gets to someone who cares, and who understands what I meant!

Will update when and if they reply!

dave

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Ask - Teoma
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved