Page is a not externally linkable
bobriggs - 5:13 am on Mar 15, 2001 (gmt 0)
But I did notice the NBCi (Snap) posts. Here is my site's robots.txt file: User-agent: * The reason I'm giving this is that when I clicked "more results from this site" on google, it gave: as a listing. It has since been deleted (from google's index) I checked my logs, and sure enough, it tried for that page, but received a 401 response header. (Authorization) I only noticed the listing after the site had first been listed in google's index. The point is, that Snap's robots.txt excludes everything. But is that really going to keep google out? And by the way, is my robots.txt formatted properly? I see a many with a blank line after the User-agents line.
Well I saw the Robots.txt in the title of this thread, but it started as Link points. Anyway, if this is off topic I apologize.
Disallow: /dealers/
Disallow: /cgi-bin/
[mydomain.com...]
www.mydomain.com/dealers/