Welcome to WebmasterWorld Guest from 107.20.5.156

Forum Moderators: open

Message Too Old, No Replies

Ask Jeeves hit pretty hard

     

carfac

5:24 pm on Oct 9, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Got hit (2k+) pretty hard by Ask Jeeves yesterday...

Bad timing- they hit me at about noon... and very excessive... 3-5 hits per second with NO LET UP for a couple hours.

Is this normal for ask jeeves? Is this excessive for a bot?

dave

fiestagirl

6:37 pm on Oct 9, 2002 (gmt 0)

10+ Year Member



That seems pretty excessive for the ask jeeves bot. The only time I've ever seen them do that is when they got caught up in a loop (our fault). Most of the time they plod along at about one request every 5 minutes.

As for the other robots - I've seen googlebot make requests that fast. If you can dish it up they will take it. But the bot seems smart enough to slow down if you can't serve the requests that quickly.

jeremy goodrich

8:26 pm on Oct 9, 2002 (gmt 0)

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member



That is very excessive. Can you give us any more information on the pages? Are they static, or dynamic? And what % of your site did Ask spider?

Ask Jeeves spider came 'round a few sites yesterday for me, but mostly grabbed robots.txt and left. To date, even on larger sites, I haven't seen their bot do this, though I haven't watched a larger scale dynamic site when the bot struck to know that it wouldn't happen in that situation...

carfac

11:01 pm on Oct 9, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Jeremy:

So I do not risk posting any "personal" info, I will sticky you seperately...

I would say they grabbed 5% or so of the site... it is a REALLY big site. Some is dynamic, some static...

dave

jeremy goodrich

11:04 pm on Oct 9, 2002 (gmt 0)

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member



That's surprising they spidered static stuff that way...hm. Odd.

Wonder if it means they are testing something new with their bot? Have you contacted them to let them know about the problem?

The one time or two I've done similar, it was with good results...engines can be responsive when it makes their product better.

(feel free to sticky if you like, but that was all I really wanted to know.) :)

And if you do contact them, please let us know what the response is!

carfac

11:14 pm on Oct 9, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Jeremy:

OK, whoops! I sticked you some logs- just ignore if you want.

I will contact them, but I do not want to run the risk of getting banned. Is that possible? I conrtacted FAST once about a robot ignoring robots.txt instructions, and they said, "OK, we will put your site into our exclusions list." Noooooo!

I did also find this robot go into a loop looking for something that it kept requesting 5 times a second for at least two minutes.... and the file is SPECIFICALLY EXCLUDED in robots.txt.

Perhaps they do need an e-mail!

dave

carfac

11:32 pm on Oct 9, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi:

OK, I send a nicely worded letter on their web feedback form... I guess I gotta hope it gets to someone who cares, and who understands what I meant!

Will update when and if they reply!

dave

 

Featured Threads

Hot Threads This Week

Hot Threads This Month