I just have a question about robots. I read that robots will only search through certain file types. Is that correct? I have a forum on my website which uses php. I don't mind if the bot goes into the forum to search through the posts, but can I prevent it from searching other areas?
I currently have it limiting my photos, cgi-bin, and templates.
TIA
