Welcome to WebmasterWorld Guest from 54.159.250.110

Forum Moderators: mack

Message Too Old, No Replies

Number of indexed pages changes all the time

site:www.domain.com fluctuate widely from day to day

   
10:27 pm on Mar 22, 2005 (gmt 0)

10+ Year Member



This has been our situation = MSN had been hitting the site aggressively prior to their search engine launch. When the launch happened the site had about 8,000 URLs in the index. While this seems like a lot - they actually spidered a lot more URLs than that (our site has 50,000 URLs). From then on the number of indexed pages changes quite frequently both UP and DOWN constantly. Looks like MSN does not have any 'memory' of the URLs it spiders and must re-spider pages constantly to have a representation of a site in its index.
10:54 pm on Mar 22, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



That could be a good idea for a rapidly changing web, everytime you respider drop all previously cached pages, on a site by site basis rather than a database by database. I've seen some evidence of this on some sites I used to monitor, everytime MSN respidered by old pages were vanishing from the index. (Superficial testing only I hasten to add).

Stops all the clutter of old, orphaned, dead or vanished pages oi suppose!