Robots Meta Tag

Robots problems

[paul_sully]

I have several websites that use dhtml. I only want googlebot to
spider the index page and nothing else. I have description and content meta tags, i also have this one on the index page:

<meta name="robots" content="index, nofollow">

This should work but googlebot seems to be ignoring the site completely?

What am i doing wrong? help!

Paul Sully
UK

[Giacomo]

Cheers Paul, and welcome to WebmasterWorld [webmasterworld.com]!

Spiders can't see DHTML, which is HTML output by the client, so maybe that's the problem with your index page.

Another reason for it not being indexed might be too few external links (links from other web sites pointing to your index page).

[paul_sully]

Hi Thanks for the reply.
it is possible that is is because of not enough external links,

I don't want the spider to go through all my pages anyway, shouldn't it just index the first page if i put this tag in:

<meta name="robots" content="index, nofollow">

Paul?

[Giacomo]

Yes, the meta tag is OK.

You should also make sure that your robots.txt file, if present, does not disallow robots from your index page.

Check your access logs for Googlebot. If Googlebot has crawled the page, then you should get a few external links in order to have it included in Google's index.

[paul_sully]

Sorry to keep on hassling but ive another question

I have a website that gets regularly crawled by googlebot, i need links to some of my other sites to get them crawled but i dont want a visible link on the page.
Does it matter if the links are text href links or images as links?
can i have a hidden table with its visibility set to 'display=none' full of links?

Thanks for your help

Paul Sully

[edited by: WebGuerrilla at 7:45 pm (utc) on Mar. 12, 2003]
[edit reason] TOS#21 [/edit]

[Giacomo]

Paul,

Googlebot follows any URL inside an

<a href>

tag.

Be warned, however, that hidden links are the quickest way to get your site banned. Please check out Google's information for webmasters [google.com] and guidelines [google.com].

I also suggest you read Brett Tabke's Successful Site in 12 Months with Google Alone [webmasterworld.com] for a list of very good advices of proven effectiveness.

[paul_sully]

Hi again,
i am sorry for getting my posts in the wrong places on your forum,

I use Dreamweaver and use fireworks for my menu's, fireworks makes
image links like this

<td colspan="10"><a href="adopt.htm" onMouseOut="MM_swapImgRestore()" onMouseOver="MM_swapImage('menu3_r2_c7','','pics/menu3_r2_c7_f2.gif',1)" target="_self" ><img name="menu3_r2_c7" src="pics/menu3_r2_c7.gif" width="108" height="19" border="0"></a></td>

Is this google friendly? i mean will it follow the links?

Sorry to ask you all these questions but i think getting spidered correctly is very important.

Also I have seen many posts about robot.txt files, if you want google to index all of a site can you simply not have a robots text file? or do you need one anyway?

Paul Sully

[Giacomo]

Paul,

Googlebot will follow any link as long as it has a valid URL in the

href

attribute. The JavaScript in the line you posted is just for the image rollover effect and is irrelevant: in this case, Googlebot will follow the link to "adopt.htm".

About robots.txt, you don't need to have one if you want all of the pages in your site to be indexed. Please note, however, that Googlebot can only find pages through links, so check you web site's link structure ("site map" in Dreamweaver) and make sure that each of the pages that you want to have indexed is linked from at least another page in your site.