homepage Welcome to WebmasterWorld Guest from 50.17.176.149
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
"robots"
What should it be set at?
kimderella86




msg:1525979
 5:53 pm on Jun 18, 2003 (gmt 0)

Hi! I was hoping that someone can help me with understanding a <meta name="robots" content="all". I wanted to know that if it is set to all does that mean that all spiders have access to my pages? I have read that rogue spiders are horrible for your website, and that they clogged loading time.

Can anyone tell me what it should be set to so that not every spider runs through the website?

I hope I am clear on what I am asking.. I am very new to the SEO world..

Thank you!
Kim

 

brotherhood of LAN




msg:1525980
 6:00 pm on Jun 18, 2003 (gmt 0)

Hi Kim, welcome to Webmasterworld!

You're OK with what you have just now if you want any spider to freely crawl through your site. You might want to read this page for more on robots and the robots.txt protocol

www.robotstxt.org/wc/meta-user.html

"Rogue" spiders usually ignore your robots.txt and robots meta tags, nonetheless there are still ways to ban them.

The website technology issues [webmasterworld.com] forum and scripting forums talk more about these, reading up on ".htaccess" is your best bet if you want to deal with these rogue spiders.

The forum library has some good threads about bannging bad bots I believe. Here's one of them
[webmasterworld.com...]

roscoepico




msg:1525981
 6:04 pm on Jun 18, 2003 (gmt 0)

Kim, your best best is to create an htaccess file. I know trying to ban a robot via your meta tags won't work. I'm not real familiar with htaccess syntax but I know many members here are. Try a site search for htaccess here at webmasterworld as this is what I would do first. Or ask for assistance in the
Website Technology Issues Forum [webmasterworld.com]

kimderella86




msg:1525982
 6:22 pm on Jun 18, 2003 (gmt 0)

Thank you very much for your help! This site is great for a rookie like myself!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved