Welcome to WebmasterWorld Guest from 184.72.138.255

Forum Moderators: open

Message Too Old, No Replies

Ask Jeeves hit pretty hard

     
5:24 pm on Oct 9, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:774
votes: 0


Got hit (2k+) pretty hard by Ask Jeeves yesterday...

Bad timing- they hit me at about noon... and very excessive... 3-5 hits per second with NO LET UP for a couple hours.

Is this normal for ask jeeves? Is this excessive for a bot?

dave

6:37 pm on Oct 9, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:May 14, 2002
posts:378
votes: 0


That seems pretty excessive for the ask jeeves bot. The only time I've ever seen them do that is when they got caught up in a loop (our fault). Most of the time they plod along at about one request every 5 minutes.

As for the other robots - I've seen googlebot make requests that fast. If you can dish it up they will take it. But the bot seems smart enough to slow down if you can't serve the requests that quickly.

8:26 pm on Oct 9, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 4, 2000
posts:3468
votes: 0


That is very excessive. Can you give us any more information on the pages? Are they static, or dynamic? And what % of your site did Ask spider?

Ask Jeeves spider came 'round a few sites yesterday for me, but mostly grabbed robots.txt and left. To date, even on larger sites, I haven't seen their bot do this, though I haven't watched a larger scale dynamic site when the bot struck to know that it wouldn't happen in that situation...

11:01 pm on Oct 9, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:774
votes: 0


Jeremy:

So I do not risk posting any "personal" info, I will sticky you seperately...

I would say they grabbed 5% or so of the site... it is a REALLY big site. Some is dynamic, some static...

dave

11:04 pm on Oct 9, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 4, 2000
posts:3468
votes: 0


That's surprising they spidered static stuff that way...hm. Odd.

Wonder if it means they are testing something new with their bot? Have you contacted them to let them know about the problem?

The one time or two I've done similar, it was with good results...engines can be responsive when it makes their product better.

(feel free to sticky if you like, but that was all I really wanted to know.) :)

And if you do contact them, please let us know what the response is!

11:14 pm on Oct 9, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:774
votes: 0


Jeremy:

OK, whoops! I sticked you some logs- just ignore if you want.

I will contact them, but I do not want to run the risk of getting banned. Is that possible? I conrtacted FAST once about a robot ignoring robots.txt instructions, and they said, "OK, we will put your site into our exclusions list." Noooooo!

I did also find this robot go into a loop looking for something that it kept requesting 5 times a second for at least two minutes.... and the file is SPECIFICALLY EXCLUDED in robots.txt.

Perhaps they do need an e-mail!

dave

11:32 pm on Oct 9, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:774
votes: 0


Hi:

OK, I send a nicely worded letter on their web feedback form... I guess I gotta hope it gets to someone who cares, and who understands what I meant!

Will update when and if they reply!

dave

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members