Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Control crawling through code

Is it possible to control the crawling through google?

         

shail2005

1:29 pm on Sep 30, 2005 (gmt 0)

10+ Year Member



Hi ,
Though its true that google crawls the site in a fixed interval of time .
But i heard that , through code also we can control the
crawling of Google ..
what the code is , if we wish like that ..

tedster

6:16 pm on Oct 3, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Well, you cannot "control" Google crawls. But certain factors that you can control seem to make their crawls better, so you can help. Here are a few things that I think help

1. Have your server employ If-Modified-Since HTTP headers
2. Rewrite dynamic urls
3. Have only one version of any url, and redirect all the alternatives -- including "www" and "no-www".
4. Be scrupulous about download speed for your pages in every way you can -- that includes minimizing server and database latency.