Forum Moderators: Robert Charlton & goodroi
But now, I have moved to my own CMS. I wonder if anything has changed from the Googlebot's standpoint, since I keep showing the website links in pretty much the same layout (latest ten articles on the home page moving deeper with pagination, etc.)
But now, Google has last indexed my page more than a week back though I have been adding content. Any idea why this is happening?
Also, double check your new RSS feed functionality to make sure it is sound.
My new CMS does not have an RSS feed. Since it was built in-house and since it is no longer a blog, rather a user generated content website, I decided to do away with RSS feeds.
Do you think RSS could be the reason for slower spidering? I am confused because many websites I know that get spidered quite often do not have an RSS feed.
Since your site is not a blog now, your backlink profile and PageRank will be important factors in how Google assigns crawl budget.
Also, many sites that are not blogs still have RSS feeds - even for user generated content. I think this might help in spidering - but there are pitfalls, too, especially in other sites easily republishing your content. If you do this, make sure everything entry that goes into your feed contains a link pointing back to the original page on your site.
RSS doesn't seem to have helped. The title of this thread should read 'Google spidering STOPPED' rather than slowed down :(
When you say 'spidering stopped' do you mean you are tracking GBot visits and they are not present or do you mean the pages are not appearing in the index, so you think GBot is not requesting them?
The reason I ask is: as anallawalla stated above others have noted a slowdown in spidering and/or indexing, but in tracking GBot visits personally on a few sites I know the pages I have recently added to sites are spidered quickly although they have not been indexed quickly.
I've personally noticed a delay ranging from 3 to 14 days between initial spidering and index inclusion, and sometimes they are tough to find when initially indexed. EG I have to search using site:example.com/directory/ to find them.
In either case, I would suggest thinking about putting up a dynamic XML Sitemap if you don't already have one, then every time you add a page it's automatically added to the XML Sitemap...
If you mean you are tracking GBot visits and they completely stopped, then I would look at the site very closely, because it could be there is a header issue or some other coding issue causing GBot to stay away. Personally, I would check all server headers immediately if the spidering has completely stopped, then I would start looking at less obvious coding issues.
Did you 301 all changed URLs from the WordPress URLs to the new locations if they were changed in order to transfer as much inbound link weight as possible to the new pages and the site in general?
Some of my top sites which used to have their new pages indexed within hours are now taking several hours to appear and sometimes even days. Some new content on less popular sites are not being picked up for weeks ... which is hugely strange for me.
New pages are also going in and out of the index. I think Google bot is being held back a little right now whilst caffeine changes are being made.
@TheMadScientist
I realize I didn't differentiate between the two. The GWT page kept showing the website getting crawled every 20 hours or so, however the page did not get indexed. Just yesterday, my home page got indexed what was crawled on the 23rd of November.
@Pavlovapete
Thanks for reminding! I did ping once manually but later found it futile since the indexing was not happening.
Some of my top sites which used to have their new pages indexed within hours are now taking several hours to appear and sometimes even days. Some new content on less popular sites are not being picked up for weeks ... which is hugely strange for me.
