Welcome to WebmasterWorld Guest from 54.159.50.111

Forum Moderators: mack

Message Too Old, No Replies

Number of indexed pages changes all the time

site:www.domain.com fluctuate widely from day to day

     
10:27 pm on Mar 22, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 25, 2004
posts:81
votes: 0


This has been our situation = MSN had been hitting the site aggressively prior to their search engine launch. When the launch happened the site had about 8,000 URLs in the index. While this seems like a lot - they actually spidered a lot more URLs than that (our site has 50,000 URLs). From then on the number of indexed pages changes quite frequently both UP and DOWN constantly. Looks like MSN does not have any 'memory' of the URLs it spiders and must re-spider pages constantly to have a representation of a site in its index.
10:54 pm on Mar 22, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
posts:1001
votes: 0


That could be a good idea for a rapidly changing web, everytime you respider drop all previously cached pages, on a site by site basis rather than a database by database. I've seen some evidence of this on some sites I used to monitor, everytime MSN respidered by old pages were vanishing from the index. (Superficial testing only I hasten to add).

Stops all the clutter of old, orphaned, dead or vanished pages oi suppose!