Forum Moderators: open
Googlebot/2.1 is visiting my site on a regular basis (every 2 days), but it is only fetching the top level page. ie not following the links. I'm wondering why.
I have the line:
<meta name="robots" content="index,follow">
in the head section of the page (and all other pages).
I also have a robots.txt:
User-agent: *
Disallow: /blah/
Disallow: /blah1/
Google has indexed the whole site in the past. And other search engines are indexing the whole site (yesterday for example).
CLUES:
This happened after I changed all file names on site. Initially files were .php, then changed site structure and now all are .html. I used .htaccess to do this:
Redirect permanent /page-name.php [my.domain.com...]
ALSO:
The date on my PC had been accidently set to be a month in advance when I uploaded the new site (this was less than a month ago). But I can't see that this would make any difference. I've checked the dates on the files on the server and they make sense.
Have I misunderstood *Redirect permanent*?
Could the date thing have had some effect?
Any ideas anyone please?
mink.
Refresh bot is not normally a deep crawler (although it can), and usually gobbles up top level pages.
Welcome to WebmasterWorld freemink! ;)
Googlebot "deep-crawling" is usually continuous throughout the month.
The more authoritative your site is (to your topics) the more it will get crawled and this can be close to continuous through the month, as well.
The actual update for this though occurs once a month (month-end) and googlebot stops crawling completely just before.
I presume that my meta-tags and robots.txt are fine. I guess I'll wait and see what happens by the end of this month, and if I'm still confused I'll post again in december.
Until then I'll check out recent posts in this forum and see what I can learn...
