Newbie here require help

robots.txt

[sim20]

Hi,

I'm newbie here.

I hv a website and my hosting bandwidth limit is very limited. Recently when i checked my log file at my server i found below given data.

11 different robots* Hits Bandwidth Last visit

MSNBot 3191 66.75 MB 30 Nov 2005 - 02:39
Unknown robot (identified by 'crawl') 1008 380.99 MB 27 Nov 2005 - 07:54
Googlebot 296 106.83 MB 30 Nov 2005 - 01:04
Inktomi Slurp 167 2.33 MB 29 Nov 2005 - 23:22
Unknown robot (identified by 'spider') 154 3.16 MB 28 Nov 2005 - 19:47
AskJeeves 111 3.61 MB 30 Nov 2005 - 01:58
Alexa (IA Archiver) 60 1.42 MB 27 Nov 2005 - 22:25
LinkWalker 19 252.01 KB 25 Nov 2005 - 07:45
WISENutbot 17 368.72 KB 26 Nov 2005 - 07:11
Walhello appie 8 148.45 KB 23 Nov 2005 - 20:32
Unknown robot (identified by 'robot') 1 24.19 KB 19 Nov 2005 - 00:45

Now come to the point that how i save my bandwidth because due to this last month my website goes down for bandiwitdh limit exceeded problem.

I hv never added robots.txt to my server so need help from members regarding same.

If i disallow all spiders then it affect my website's popularity.

And i'm using google adsense code (I'm google adsnese publisher) so which thing i hv to add in robots.txt for that to continue display google ads at my website?

Senior Members reply appreciate,

Regards.

[Leosghost]

Either you are really being ripped ( hard to tell cos your log data doesnt say what got hit ..just who "says" they did it ) or your hoster is real stingy with your bandwidth ..that said your site appears to be reasonable size so maybe you should change hosts ..

One item makes me wonder "Unknown robot (identified by 'crawl') 1008 380.99 MB 27 Nov 2005 - 07:54"

that is how many megs?..over how much time?