Welcome to WebmasterWorld Guest from 184.108.40.206 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
Everything crawled twice (at almost the same time) Is Ask Jeeves' bot stupid? luma msg:31660 8:26 pm on Jun 28, 2002 (gmt 0) On 26/Jun/2002 some of my pages were crawled by Ask Jeeves. But why do they get everything twice within two minutes? I'd rather they'd index all of my site once instead of half of my site twice. Okay, same load but half the value. ;)
caine msg:31661 2:03 pm on Jul 1, 2002 (gmt 0)
not sure why they would:
1. crawl a selection of pages
2. crawl them twice
was it exactly the same bot ?
luma msg:31662 10:35 am on Jul 11, 2002 (gmt 0)
It was exaclty the same bot ("Mozilla/2.0 (compatible; Ask Jeeves)"). It took 54 pages (27 pages times 2).
They got exaclty the same 27 unique pages on July, 5th, so this seems fixed. Still, I wouldn't mind them getting all of my site.