Andylew - 1:22 pm on May 26, 2010 (gmt 0)
Just an update on the route I am taking ref earlier post, convinced there is nothing wrong with IBLs and sitemaps still reporting a substantial number more indexed pages than site: (930k v 80k) indicating there is an index being serverd which is different to the one wmt is reporting we took a hardware route.
The reasoning is if google is reindexing or taking more account of speed or perhaps sites per server = authority we wanted to make sure that they could reindex our entire site in the fastest time possible.
So a new server later and promissing news, within hours of switching we have more than trebbled the gbot activity on the site. Although the old server was not struggling google was clearly throttling the speed that it crawled considerably, perhaps an equation between crawl speed and no pages resulting in a drop - perhaps they are thinking a site with massive amount of pages that cant handle intense gbot traffic is clearly not very good. We used to see 4-5 distinct ips crawling at any time in the day, we are now seeing between 12 and 20 distinct concurrent gbot ips.