Does adding robot text help your site get crawled more often?

Advantages of robot text?

[moneyangel13]

Hello

I was just wondering what are the benefits of adding robot text to your site? For examle will Googlebot crawl your site more often? Will you get indexed faster? I dont have it on my website and I would like to know what are the advantages. I am trying to get indexed on google and maybe that will help me. Also any suggestions on where to get information to make a site map? Thanks to all

[Sanenet]

robots.txt is just a directive that tells bots which pages they aren't allowed to look at.

www.robotstxt.org

A sitemap is just a standard page that contains links and a description of each of your important pages. Easy enough to whip up.

[jdMorgan]

A major advantage to having *any* robots.txt file on your site is that all Standard-compliant robots will ask for it. If it is missing, they will receive a 404-Not Found error, and you will receive an entry in your server error log. This makes their job harder, and pollutes your error log with useless entries, making it harder to identify *real* server errors.

If you don't need robots.txt to control robots on your site, then upload a blank robots.txt file. This will eliminate all those 404 errors in your server error log, and make it easier for you to use it to identify actual problems with your site.

Jim

[Sanenet]

Good point Jim, and something that I would recommend to all. Thanks for reminding us of it.

[Reid]

moneyangel - no robots.txt will not get you listed in google faster -or any other search engine.
It shouldn't have any affect in the frequency of bot visits either.
You want to get listed in google fast?
Try the new 'google sitemaps' feature. Submit your sitemap to google and I bet the bot will come around shortly after your sitemap is approved.
Incoming links from other sites is the tried and true method to get bots coming.
Also submit your site to the search engines submit URL pages.

[moneyangel13]

Hello all thank you for your advice I will check out google site maps. Thanks. I am just frustrated I am indexed on Yahoo and not Google and its driving me nuts! Thanks to all. You are awesome!