Google indexing ftp log

How to stope certain file types being indexed

[jsnow]

How do I stop Google and any other spiders from indexing files that aren't linked to from the site. I thought that if they didn't have a link they wouldn't be indexed but I was wrong!

[dmorison]

Just assume that anything in your public web tree can and probably will be indexed, unless prevented (at least in an advisory capacity) by robots.txt.

Esp. with various toolbars feeding URLs back to search engine companies etc.; everywhere you go is likely to end up being requested by a robot at some point.

If you have the option; keep anything you don't want indexed out of the document root. If you don't, robots.txt is the best you can do.

See [robotstxt.org...] for more details.