Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

How important is having a properly configured robots?



8:37 pm on Feb 11, 2004 (gmt 0)

10+ Year Member

I am using OSCommerce, I haven't changed robots.txt from the way it comes default. There are MANY things I need to research still to help the site place better with search engines, so I'm just trying get an idea of how much I'd benefit from tweaking/configuring robots.txt. I want to put a priority on everything since I have other things to do with the company other then just internet-marketing related things, therefore I want to do what is most important first. Thanks!


3:42 pm on Feb 12, 2004 (gmt 0)

10+ Year Member

OK, after researching some more I've come to learn that you can disallow specific bots and/or disallow them to access specific areas of your site...but I don't understand why you would want to not give access to certain areas. Can someone explain this to me please?


5:06 pm on Feb 13, 2004 (gmt 0)

10+ Year Member

Some reasons I could think of why you might want to disallow to certain areas would be....
1. Security issues. If you have an area of your site with sensitive information that would be of no real SEO use.
2. SEO itself. A person might have a site with seperate folders, for say navigation bars or images that would not be of any benefit to you to have them crawled and put on a search engine index somewhere.
3. Bandwidth issues. If you had a very large site with lots of links. Bandwidth utilized by these crawlers could be an issue and effect performance of your site. Or could cost you alot of money if you pay for bandwidth utilization. Especially if you have a site that would be for regional purposes and an overseas crawler keeps hitting your site. Its not going to benefit you and could effect performance and cost.

Im kinda new to SEO and robots.txt , etc also. But these were just a couple of thoughts I had on it.
Hope it helps...


3:45 pm on Feb 14, 2004 (gmt 0)

10+ Year Member

Those are some very good points, thanks for the insight.

Featured Threads

Hot Threads This Week

Hot Threads This Month