homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How important is having a properly configured robots?

10+ Year Member

Msg#: 285 posted 8:37 pm on Feb 11, 2004 (gmt 0)

I am using OSCommerce, I haven't changed robots.txt from the way it comes default. There are MANY things I need to research still to help the site place better with search engines, so I'm just trying get an idea of how much I'd benefit from tweaking/configuring robots.txt. I want to put a priority on everything since I have other things to do with the company other then just internet-marketing related things, therefore I want to do what is most important first. Thanks!



10+ Year Member

Msg#: 285 posted 3:42 pm on Feb 12, 2004 (gmt 0)

OK, after researching some more I've come to learn that you can disallow specific bots and/or disallow them to access specific areas of your site...but I don't understand why you would want to not give access to certain areas. Can someone explain this to me please?


10+ Year Member

Msg#: 285 posted 5:06 pm on Feb 13, 2004 (gmt 0)

Some reasons I could think of why you might want to disallow to certain areas would be....
1. Security issues. If you have an area of your site with sensitive information that would be of no real SEO use.
2. SEO itself. A person might have a site with seperate folders, for say navigation bars or images that would not be of any benefit to you to have them crawled and put on a search engine index somewhere.
3. Bandwidth issues. If you had a very large site with lots of links. Bandwidth utilized by these crawlers could be an issue and effect performance of your site. Or could cost you alot of money if you pay for bandwidth utilization. Especially if you have a site that would be for regional purposes and an overseas crawler keeps hitting your site. Its not going to benefit you and could effect performance and cost.

Im kinda new to SEO and robots.txt , etc also. But these were just a couple of thoughts I had on it.
Hope it helps...


10+ Year Member

Msg#: 285 posted 3:45 pm on Feb 14, 2004 (gmt 0)

Those are some very good points, thanks for the insight.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved