Forum Moderators: open
Is anyone seeing the results of this crawling appearing in the SERPs? It's pushing me toward my bandwidth limit as it is, so was wondering if it might be worth putting a delay into the robots.txt to temper the beast...
I have just noticed today that:
Slowly, google is releasing the pages crawled by the crazy bot to the index. For me the result of site:www.mysite.com goes from 437 to 709 without the normal bot hitted.
We have about new 20,000 'no-title, no-description' URLs showing in the index. This, after Google spidered over 100,000 pages in early Nov.
Has anyone seen new pages included in the index after the massive crawl?
Yes. Despite (or because of) message 87 in this thread, I am now seeing an extra 20,000 pages from the site in the Google Index.
The site is a mainly forum. Before it was only half indexed. Now they have pretty much every page.
All fully indexed, as far as I can see.
A lot of them don't come up on many searches unless you click
If you like, you can repeat the search with the omitted results included.
But that's not so unusual. You'd see the same if looking for common words on all webmasterworld pages, eg:
webmasterworld control panel site search glossary subscribe help library conference
However, since September the reverse has become true. Pages steadily fell out of the index, our position sliped and customers vanished. We had changed nothing and were totally white hat in everything we did.
It seems that somtime around September our site map stopped being cached fully (too big?) and the spider was too lazy to follow other links outside the site map.
We changed the site map and restructured our site to no avail as the bot just does not seem to want to know. It randomly visits a handfull of pages and randomly caches a fraction of those.
This so frustrating because we are a start-up, google is our sole marketing tool and source of income. Sales are virtually nil now and it seems that all we can do is wait. What are the rules? What changed?
