Forum Moderators: phranque

Message Too Old, No Replies

help, google stops crawling my site for some reason

google and other search engine stop crawling my site

         

westlakedi

6:47 pm on Feb 4, 2005 (gmt 0)

10+ Year Member



I updated my URLs. But google still cache my old URLs (over six month old). Google thinks my old URLs have same contents as my new URLs and penalized us. Here's what happen on my site. Google stop crawling my site after I use google auto remove sub directories (). I still can find my page in google. But they just not crawling my site any more.

Here's my robot.txt file.
User-agent: *
Disallow: /modules/
Disallow: /cache/
Disallow: /class/
Disallow: /images/
Disallow: /include/
Disallow: /install/
Disallow: /kernel/
Disallow: /language/
Disallow: /templates_c/
Disallow: /themes/
Disallow: /uploads/
Disallow: /news/
Disallow: /polls/

Here's my traffic report.
Jeeves1397+39.66 MB04 Feb 2005 - 09:40
Inktomi Slurp833+1068.83 MB04 Feb 2005 - 09:37
MSNBot800+175.33 MB04 Feb 2005 - 08:51
GigaBot140+351.18 MB03 Feb 2005 - 12:16
Unknown robot (identified by 'crawl')160+131.45 MB04 Feb 2005 - 01:59
BaiDuSpider53495.18 KB04 Feb 2005 - 09:17
Alexa (IA Archiver)14+2120.68 KB04 Feb 2005 - 05:25
Googlebot (Google)10+697.44 KB04 Feb 2005 - 08:28

indodpendo

7:41 pm on Feb 4, 2005 (gmt 0)

10+ Year Member



try
Disallow: index,follow