Page is a not externally linkable
jdMorgan - 6:52 pm on Jan 6, 2004 (gmt 0)
When you say these disallowed pages are indexed, do you mean that the page is shown in G search results with a title, snippet, description, etc., or is it just listed as a URL? If the latter case, G is simply listing the information it can find from links to the page (and the associated link-text), therefore no title, snippet, and description appear in the results. If you want to remove this type of listing, the solution seems counter-intuitive; Remove the disallow in robots.txt, and add an on-page meta robots noindex tag. This applies to AJ/Teoma as well. If the former case - if you're seeing a "full listing" - then there is indeed some problem with robots.txt. Make sure that your records are in the right order (Spiders accept the first User-agent directive that matches their name or "*", whichever comes first, and won't look further). Jim
Defining the terms here would help...