Search Engine Problems...not crawling entire site

searchbots visit my site index the homepage and leave - help!

[Bluey]

Hello,

I've built my own site with Meta tags, built a Google sitemap.xml (only yesterday so can't expect too much).

Problem is I have been listed in search engines for weeks now. The bots come along - ask for robots.txt (which I created to make sure all ok - here's what I have:

User-agent: *
Disallow: /cgi-bin/
Disallow: /Tellafriend/
Disallow: /Library/
Disallow: /infoemail.htm
Disallow: /Templates/

...) and then leave after only indexing the index page! This is happening with Yahoo and Google and nuerous other SE so I think it must be something I am doing wrong? Any advice gratefully received.

If anyone wants to have a look I can give the URL (am I allowed to give with spaces so it isn't promotional?)

Thanks

[gdawg]

Did you submit your sitemap.xml file to Google and put it on your webserver? As long as you have a good linking structure (text links) and a real sitemap that links to all your pages I am sure the search engines will index all your pages. It may just be a matter of time.

[Bluey]

Thanks, yes I submitted sitemap to Google and said ok - by putting on server I added it to my files in Dreamweaver and "put" to publish it (I hope that's all I need).

What's odd is that around 6 spiders have visited my site yet don;t go anywhere after the index page...is anyone else having this problem? Site was born in early May...

Thanks

[gdawg]

Do you have static text links to all of your interior pages?

[shtirlitz]

If you want google to visit your site more often add this to the header.

<meta name="revisit-after" content="2 days" />
<snip>

[edited by: Brett_Tabke at 9:46 pm (utc) on July 2, 2005]
[edit reason] no urls please. [/edit]

[Bluey]

Good news! (I think)

Googlebot has crawled most of the site so at least I now know that there is nothing stopping the spider.

Of course it is now just a waiting game to see if and when they will appear in the listings or if it is an 18 month sandbox as I have read on other posts.

Thank you for replying to this topic. Yes, I have static text links to some (I have a sitemap) of my interior pages.

I have set the <META NAME="REVISIT-AFTER" CONTENT="1 days"> but it doesn't seem to have made much difference.

I presume that no one knows how long it might be until the pages our listed...?

Thanks again