I've been a web developer for years, but am pretty new to the concept of robots.txt. Perhaps somebody can point me to a general thread that explains it, or just answer my simple questions.
As I understand it, robots.txt limits the indexing of your site for whichever searchbots or spambots you mention. Does it also facilitate indexing in some way? If I do not put a robots.txt in my site at all, will my page ranking with some search engines suffer?