Is there any way to make search engines ignore your web site? I need to make up an unsecure site that's available on the web but I'd rather the search engines ignore it so we don't get a slew of freeloaders looking for that fast buck. Does anyone know if this is possible?
Msg#: 3892516 posted 10:05 pm on May 2, 2009 (gmt 0)
The robots.txt disallow stops bots spidering the site but will still allow URLs from your site to be listed in the SERPs. These will be listed as URL-only entries if anyone ever links to those URLs from some other place.
The meta noindex tag will prevent even the URLs from being listed in the SERPs, but bots will need to access those pages in order to actually see the meta tag.