Top Page Only

[freemink]

Hi,

Posted this a week or so ago, but will try again...

Googlebot/2.1 is visiting my site every day, but it is only fetching the top level page. ie not following the links. I'm wondering why. This had been happening for the past month now. Google has indexed my whole site before, and other bots are crawling my site.

** What are the possible reasons for this? **

I have the line:
<meta name="robots" content="index,follow">
in the head section of the page (and all other pages).

I also have a robots.txt:
User-agent: *
Disallow: /blah/
Disallow: /blah1/

Thanks,
freemink

[troels nybo nielsen]

There's no way to force or trick freshbot to follow a link. You can only try to make the links as easy to find and follow as possible. And then hope.

[freemink]

Hope? Trick? Web crawling sounds a rather mysterious process!

My only question is: why has googlebot only fetched my top page for the last month? Is this normal? Is it strange? If not normal, then does anyone know of any reasons for this behaviour?

[ukgimp]

Have you checked the links. Are they viewable to googlebot. Are they Java. Perhaps the page structure is too complicated and in effect making the links invisible.

You could try passing the page through somethinglike like Links Viewer to see it that can see the links.

Cheers

[troels nybo nielsen]

I don't think that anybody outside Google really understands how freshbot works. It's a rather new feature and they are still experimenting with it. It seems to have taken an interest in some of my pages while ignoring others. I am unable to understand the reasons why.

[freemink]

My links are the standard:
<a href="link.html" class="sidenavlink">LINK</a>

Getting my top page indexed is most important to me, so I'm not too worried about this. But it has me puzzled, especially as other searchbots crawl site okay.

Seeing as little is known about freshbot, I guess I should wait and see for a while...

Thanks for comments.

[rfgdxm1]

It looks to me like having decent PR (say 5) increases the interest of the freshbot. Also, the freshbot seems to like pages at root. The further down and farther away from root, the less likely freshbot cares.

[troels nybo nielsen]

I agree. But yesterday freshbot surprised me by visiting some pages on the 4th level. Today they are in the serps with fresh 24th tags.

[kstprod]

freemink,

I'm not sure if this is relevant to your situation, but when Freshbot 1st started visting me, it only took the root page. After a couple of weeks, it gobbled all of my pages up. I got so excited that I created another page, and linked it from the root. As soon as Freshbot saw that page, it slowly quit the deep crawling. My guess is that somehow inadvertantly, I did something wrong on that extra page that Freshy didn't like. I'm not very savvy when it comes to optimizing, so my guess is that I went a little overboard on that one page. Anyway, I removed it and all the links, hoping Freshy will start going deep again.

Just a thought for you! Good luck!

Karen

[freemink]

Thanks Karen. Sounds similar to my situation I did add new pages and links. Just had a look at the html, and it has changed since google last crawled full site - become a little more messy in fact. Maybe that's the reason. Anyway, I'll tidy the html and see what happens...

[troels nybo nielsen]

One of the golden general rules with Google seems to be: Don't overdo it.