Google Fetches Home Page and Leaves

Comes one a day, never spiders.

[notbob]

Seems that google bot comes to one of my websites, at least once a day, sometime two or three times within just a few minutes of each other.

Each time, googlbot only seems to spider the homepage then leave.

Just wondering if anyone had any thoughts on what the point of this is? The domain was a few months ago, but only put a website up about 2 weeks ago. Few incoming links.

Thank you

[Reid]

sounds like your homepage may be hanging googlebot somehow.

run it through an HTML validator
WW has a bot simulator too
also google 'poodle predictor' check that little toy out.
I'll bet you find something wrong.

[notbob]

Thanks for the tips. I'm curious about this now and will have to check out if this might be the cause.

[Bluey]

Hi,

Notbob - when did you submit your sites to Google (or how long have they been around?)

I launched my site in April and it has taken until today (yes I am very happy) for it to appear in Google - before only the index page was listed. I now have 39 pages indexed out of around 100.

SO going on my experience it is taking 3 months - though from what I have read it varies.

Things I did which may have helped:
- robots.txt
- google Sitemap
- fresh page of content daily (though bizarrely Google has indexed the pages with the least text so far - so all this hype about Google.bot loving text I have yet to witness).
- inbound links
- Metatags
- Lots of text links
- Check and validate everything.

Though of course the above MAY have helped - it was probably just a waiting game.

Cheers.

[webdude]

It's the google Sitemap that did it for you, Bluey.

[theBear]

Bluey, my bet would be that Google finally found a real inbound link or two.

That normally does the trick.

Good news and good luck.

[Reid]

it's not a case of googlebot 'finding' the site or the site finally coming up the list.
The complaint is that googlebot is continually asking for the homepage. Why does it not spider the links on the homepage?
It may be doing different things with the pages each time or it may be several googlebots all asking for the homepage for a little bit but I know that if it can't access the page but knows it exists, it will keep requesting it (and robots.txt) for weeks and then it will go away and come back in a month or 2 and repeat the cycle. This could go on for years.

[moftary]

I have a very strong high PR linking campaigns that I use to link to my newly created sites (think devshed!). However it appears that googlebot will NOT go into a deep crawl before the new website retains its own PR. I have three new websites, two months old or so and they are still PR0 (strong inbound links, but just missed the PR update). These three websites, not sandboxed btw, do have their homepages indexed only. Other websites under my control have the same hierarchy, searching engine optimisation techniques and everything and they are %100 indexed. The only difference between those websites and the other non-indexed three is the PR shown in google toolbar. Yes, there might be many factors for having a deep crawl/index, but at least in my case it's the PR shown in google toolbar that matters. Any confirmation, arguments or same/other experiences are welcomed :)

Cheers,
mOftary

[theBear]

notbob,

What return code did googlebot (which googlebot) receive when it asked for your homepage?

What if anything is in your robots.txt file?

We just launched a new site in June and the bots have been busy.

[sit2510]

>>> Each time, googlbot only seems to spider the homepage then leave...but only put a website up about 2 weeks ago. Few incoming links.

-------------------------

That is quite normal for new site with few incoming links. Get more incoming links and bots will crawl your site more deeply soon.

[Reid]

when i launched my site (14 months ago) the bot came right away and deep crawled everything - been coming steady ever since - over 500 hits this month from gbots.
I did do a directory campaign (free submit) though. That may have got things going.

here is a little secret how to get googlebot to come to a new site right away with no IBL's at all. it works according to someone who 'does it all the time with clients' and 'has tried it on a test site with absolutely no IBL's'.

ready?

submit a simple robots.txt to the removal tool.

User-agent: *
Disallow:

a little off-topic here though since googlebot is already coming.

[notbob]

This is interesting.

Prior to creating a sitemap, I had the index page on the google SE.

Since submitting the sitemap, even the index page has been dropped from google.

ummm

[Marcia]

I have two small sites that went up a few weeks ago, about 20 pages each and they're all indexed - one had 7 indexed, another 10 just were included since this past weekend. Only one mediocre link to each site.

Run the homepage through the html validator and bot simulator, and also validate the robots.txt

[activeco]

notbob, sounds very much like detected duplicate content on all your pages.
Can you check raw stats files from the very beginning?
There is a high probability the googlebot visited all/most of your internal links and simply deleted them after detecting duplicate content.
The same for your home page. Many reasons could lead to it.

[notbob]

Well, it's taken one extra backlink from one of my pr0 sites to get the ball rolling.

This evening googlebot came, spidered 20 pages and left. I'm curious if anything will show up in google in the next 24 hours or so.

No other changes were necessary.