How important is having a properly configured robots?

[budbiss]

I am using OSCommerce, I haven't changed robots.txt from the way it comes default. There are MANY things I need to research still to help the site place better with search engines, so I'm just trying get an idea of how much I'd benefit from tweaking/configuring robots.txt. I want to put a priority on everything since I have other things to do with the company other then just internet-marketing related things, therefore I want to do what is most important first. Thanks!

[budbiss]

OK, after researching some more I've come to learn that you can disallow specific bots and/or disallow them to access specific areas of your site...but I don't understand why you would want to not give access to certain areas. Can someone explain this to me please?

[DavidAtWork]

Some reasons I could think of why you might want to disallow to certain areas would be....
1. Security issues. If you have an area of your site with sensitive information that would be of no real SEO use.
2. SEO itself. A person might have a site with seperate folders, for say navigation bars or images that would not be of any benefit to you to have them crawled and put on a search engine index somewhere.
3. Bandwidth issues. If you had a very large site with lots of links. Bandwidth utilized by these crawlers could be an issue and effect performance of your site. Or could cost you alot of money if you pay for bandwidth utilization. Especially if you have a site that would be for regional purposes and an overseas crawler keeps hitting your site. Its not going to benefit you and could effect performance and cost.

Im kinda new to SEO and robots.txt , etc also. But these were just a couple of thoughts I had on it.
Hope it helps...

[budbiss]

Those are some very good points, thanks for the insight.