homepage Welcome to WebmasterWorld Guest from 54.211.113.223
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How NOT to be seen.
What if you don't want search engines to find you?
Hutch

10+ Year Member



 
Msg#: 3892516 posted 11:36 pm on Apr 14, 2009 (gmt 0)

Strange question?

Is there any way to make search engines ignore your web site? I need to make up an unsecure site that's available on the web but I'd rather the search engines ignore it so we don't get a slew of freeloaders looking for that fast buck. Does anyone know if this is possible?

Thanks.
Hutch

 

LifeinAsia

WebmasterWorld Administrator lifeinasia us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3892516 posted 11:58 pm on Apr 14, 2009 (gmt 0)

There are a number of ways:
- robots.txt
- <meta name="Robots" content="NOINDEX"> on every page
- block user agents with .htaccess

Either of the first 2 should block most well-behaved bots. The 3rd will require updates as you find more bots to block.

Hutch

10+ Year Member



 
Msg#: 3892516 posted 10:16 am on Apr 15, 2009 (gmt 0)

Thanks for the help :)

Hutch

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3892516 posted 10:05 pm on May 2, 2009 (gmt 0)

The robots.txt disallow stops bots spidering the site but will still allow URLs from your site to be listed in the SERPs. These will be listed as URL-only entries if anyone ever links to those URLs from some other place.

The meta noindex tag will prevent even the URLs from being listed in the SERPs, but bots will need to access those pages in order to actually see the meta tag.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved