Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
Can anyone tell me what it should be set to so that not every spider runs through the website?
I hope I am clear on what I am asking.. I am very new to the SEO world..
joined:Jan 30, 2002
You're OK with what you have just now if you want any spider to freely crawl through your site. You might want to read this page for more on robots and the robots.txt protocol
"Rogue" spiders usually ignore your robots.txt and robots meta tags, nonetheless there are still ways to ban them.
The website technology issues [webmasterworld.com] forum and scripting forums talk more about these, reading up on ".htaccess" is your best bet if you want to deal with these rogue spiders.
The forum library has some good threads about bannging bad bots I believe. Here's one of them