Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Google stopped crawling many sites Jun 15 AM


manof - 11:09 pm on Jun 16, 2010 (gmt 0)


Me too.

I've post here: [webmasterworld.com...]

My site has released for 4 months,in the last 15 days,Googlebot crawling 300,000-500,000 pages every day.

But,From 08/06/2010 I find Googlebot usually only crawling my homepage every 10-20 minutes:

"GET / HTTP/1.1" 200

Occasional,Googlebot will repeatedly crawling other 3-4 pages,such as:

............
"GET / HTTP/1.1" 200
"GET /my_*_*_*_mori HTTP/1.1" 200
"GET / HTTP/1.1" 200
............
............
............
"GET / HTTP/1.1" 200
"GET /my_*_*_*_mori HTTP/1.1" 200
"GET / HTTP/1.1" 200
............
............
............
"GET / HTTP/1.1" 200
"GET /my_*_*_*_mori HTTP/1.1" 200
"GET / HTTP/1.1" 200
............


Googlebot has been continuously crawling some pages that obviously does not belong to my site:

"GET /include/setup.exe HTTP/1.1" 404
"GET /author/*_klayiv_*/pisma_*/download.*.prc.zip HTTP/1.1" 404
"GET /usenext/1213093/*+*+5.0.15+*+Full+Version.exe.html HTTP/1.1" 404
"GET /download/projects/vcpp/*_screen.zip HTTP/1.1" 404

Then I changed my site's IP,Googlebot get the robots.txt,but it continues to crawl my site's home page only:

"GET /robots.txt HTTP/1.1" 200
"GET / HTTP/1.1" 200
............
............
............


I've checked google webmaster tools,did not find any abnormality.

Google indexing has dropped some,24 hours indexing is zero,and my site's traffic has dropped 1/3.

I search my site name in google,is first.

I search "site:example.com" in google, and the domain root, example.com, is not the first.


Thread source:: http://www.webmasterworld.com/google/4152972.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com