homepage Welcome to WebmasterWorld Guest from 50.16.130.188
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Google crawling reduced drastically
gzoo




msg:4452502
 7:44 am on May 12, 2012 (gmt 0)

Two weeks back I modified the structure of my website and implemented over 200 sub-domains for different countries and sections.

Initially, everything went OK but since last 3-4 days crawling of my website is reduced drastically by Google. Not just crawl, website traffic also effected 30-40% with compare to last week.

I checked everything like robots.txt, fetch by google but no clue has been found so far.

What can be the possible reasons for this problem?

 

jemois




msg:4452551
 11:21 am on May 12, 2012 (gmt 0)

There could be many reasons for this. How old is your site ? Have you been penalized/filtered by any reason ? Are the subdomains new with new data or redirected from old url's by moving the old data through 301 ? where do you see the frequency of the Google Bot crawl ? check with the site: command if the new/modified pages are already cached and indexed. The best way to see if the new version is indexed is by checking the preview of the page in google search. Providing more information will help us getting a better image of what's going on. I also modified one of my sites about 2 weeks ago. Google WMT tools shows almost no crawling of my website, but the whole design was changed. I noticed that actually all of the pages are indexed in the new version but are not yet ranking for the new content.

deadsea




msg:4452567
 12:35 pm on May 12, 2012 (gmt 0)

I had a similar experience. Two months ago I launched my website in 30 new languages on subdomains (in addition to the 10 languages I already had). Google crawl rate on my main English site dropped dramatically when I did this. For me there was no corresponding drop in traffic. I did not to the analysis to see if total crawl volume was up or down. I suspect that it is about the same, maybe a bit higher. Because Googlebot has to spider all this new content, it doesn't have as much time for the content it already knows about.

Interestingly, the crawl rate to another unrelated site of mine on the same server also decreased dramatically at the same time. It could be that Google doesn't want to hit the server itself too hard and that is the limiting factor in the crawl rate.

gzoo




msg:4452830
 10:47 am on May 13, 2012 (gmt 0)

Thanks jemois & deadsea for your contribution.

@ jemois, my website is approx. 3 years old. Website doesn't seems blacklisted and neither it is get penalized ever in past.

All sub-domains are new but the data is old and yes I have applied 301 to move old URLs to new one. Initially Google crawling was good on main website as well as on top sub-domains for approx. 6-7 days but later on (since last 4-5 days) suddenly it gets dropped up to 90%, means ratio dropped from 100 to just 10-12.

Using "site:" I can see good number of index pages of top sub-domains but I can't say that all pages are indexed.

One problem I just noticed in WMT is increased number of not found or 404 pages, while reviewing those pages I found that to increased the search quality we tighten the criteria of landing pages which result in generating "0" results pages which are reporting as not found or 404. Could this be the reason of this issue?

I would appreciate the contribution of experts in this thread?

Errioxa




msg:4452835
 10:49 am on May 13, 2012 (gmt 0)

In my site always googlebot crawls around 200.000 - 250,000 urls with redirects per day. I track it for long time ago from my access logs.


06/May/ - 252.963
07/May/ - 213.955
08/May/ - 231.638
09/May/ - 169.253
10/May/ - 24.891

jemois




msg:4452944
 7:01 pm on May 13, 2012 (gmt 0)

@gzoo
Google crawls the pages but it takes some time to index and serve them, this might be the reason why the Google Bot isn't crawling your site so often now. It is also possible that Google found that the "0" pages have no value for you site and for that reason not to show them in the index. Check the header of the not found pages, it might be 410 or 404 so even if there is content on the pages they will look to Google as deleted (gone) or not existing.

gzoo




msg:4453163
 9:52 am on May 14, 2012 (gmt 0)

@jemois : the status of "0" records pages is 404.

Today, Googlebot has further slower down crawling as well as traffic is decreased drastically. In "Crawl Errors" 8000+ errors are reporting by WMT for 11th to 13th but in 1000 sample error pages all detected records are of before 11th.

Can anyone help me out to figure-out the problem with my website and share a reliable resource from where I can check whether my website is blacklisted or not?

Tedster, I would appreciate few words from you about the problem I am facing.

Andylew




msg:4453257
 1:47 pm on May 14, 2012 (gmt 0)

interesting....

Also seen a massive decrease in googlebot crawling the past weekish and no refresh in index data.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved