LifeinAsia
msg:3892534 | 11:58 pm on Apr 14, 2009 (gmt 0) |
There are a number of ways:
- robots.txt
- <meta name="Robots" content="NOINDEX"> on every page
- block user agents with .htaccess Either of the first 2 should block most well-behaved bots. The 3rd will require updates as you find more bots to block.
|
Hutch
msg:3892776 | 10:16 am on Apr 15, 2009 (gmt 0) |
Thanks for the help :) Hutch
|
g1smd
msg:3905681 | 10:05 pm on May 2, 2009 (gmt 0) |
The robots.txt disallow stops bots spidering the site but will still allow URLs from your site to be listed in the SERPs. These will be listed as URL-only entries if anyone ever links to those URLs from some other place. The meta noindex tag will prevent even the URLs from being listed in the SERPs, but bots will need to access those pages in order to actually see the meta tag.
|
|
