Hi! I was hoping that someone can help me with understanding a <meta name="robots" content="all". I wanted to know that if it is set to all does that mean that all spiders have access to my pages? I have read that rogue spiders are horrible for your website, and that they clogged loading time.
Can anyone tell me what it should be set to so that not every spider runs through the website?
I hope I am clear on what I am asking.. I am very new to the SEO world..
Kim, your best best is to create an htaccess file. I know trying to ban a robot via your meta tags won't work. I'm not real familiar with htaccess syntax but I know many members here are. Try a site search for htaccess here at webmasterworld as this is what I would do first. Or ask for assistance in the Website Technology Issues Forum [webmasterworld.com]