Welcome to WebmasterWorld Guest from 54.196.232.162

Forum Moderators: open

Message Too Old, No Replies

Everything crawled twice (at almost the same time)

Is Ask Jeeves' bot stupid?

     
8:26 pm on Jun 28, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 20, 2002
posts:194
votes: 0


On 26/Jun/2002 some of my pages were crawled by Ask Jeeves. But why do they get everything twice within two minutes? I'd rather they'd index all of my site once instead of half of my site twice. Okay, same load but half the value. ;)
2:03 pm on July 1, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 21, 2001
posts:2488
votes: 0


not sure why they would:

1. crawl a selection of pages
2. crawl them twice

was it exactly the same bot ?

10:35 am on July 11, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 20, 2002
posts:194
votes: 0


It was exaclty the same bot ("Mozilla/2.0 (compatible; Ask Jeeves)"). It took 54 pages (27 pages times 2).

They got exaclty the same 27 unique pages on July, 5th, so this seems fixed. Still, I wouldn't mind them getting all of my site.