newbie question

About robots.txt

[dicken288]

hi. i'm newbie, I have some question,
I look statistics warez, the follow Robots/Crawler visit my site.

MSIECrawler
Inktomi Slurp
MSNBot
Googlebot
Unknown robot (identified by 'crawl')
Alexa (IA Archiver)
Unknown robot (identified by hit on 'robots.txt')
arks
Grub.org
GigaBot

I have a robots.txt, I need add something in my site let more Robots/Crawler know about my site? (e.g Channel Definition Format (CDF) etc...)

Thank you

[SEOMike]

I need add something in my site let more Robots/Crawler know about my site?

There's a lot more to it than that. Welcome to the world of SEO.

[Dijkgraaf]

If spiders/crawlers/bots are requesting your robots.txt file, then they allready know about your site, so adding anything to encourage the ones that don't is not needed, nor is there anything you can add that would achieve that anyway.
Most spiders/crawlers/bots find your site by finding a link on another web site pointing to your site. There is a whole forum about links, so thats probably where you want to go and look for your answers.

[dicken288]

thanks all~
Is it necessary to modify or add something on my robots.txt for other spiders / bots / crawlers? Actually is there any mean to improve the site?

Thanks~

[edited by: dicken288 at 7:44 am (utc) on Sep. 21, 2005]

[marcs]

Bad bots (email harvesters and the like) won't bother to read your robots.txt file.