Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Indexed pages that are disalowed by robots.txt


jdMorgan - 6:52 pm on Jan 6, 2004 (gmt 0)


Defining the terms here would help...

When you say these disallowed pages are indexed, do you mean that the page is shown in G search results with a title, snippet, description, etc., or is it just listed as a URL?

If the latter case, G is simply listing the information it can find from links to the page (and the associated link-text), therefore no title, snippet, and description appear in the results. If you want to remove this type of listing, the solution seems counter-intuitive; Remove the disallow in robots.txt, and add an on-page meta robots noindex tag. This applies to AJ/Teoma as well.

If the former case - if you're seeing a "full listing" - then there is indeed some problem with robots.txt. Make sure that your records are in the right order (Spiders accept the first User-agent directive that matches their name or "*", whichever comes first, and won't look further).

Jim


Thread source:: http://www.webmasterworld.com/robots_txt/229.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com