Quick help with robots.txt please?

where to put the file?

[LadyTi]

Hi all,

I did look at the other recent threads on this but didn't want to hijack them, so starting a new one.

I do have my own hosting and haven't added this file yet. Can someone explain just a few things to me please?

1. Why is this file really needed
2. What do you mean by put in 'root' directory
3. Do these spiders crawl all the html type pages AND all of the mysql databases?
4. If, I do NOT want certain pages crawled do I have to disallow each one?

Yes, obviously I'm new here and still trying to grasp all of these concepts, lol

Thanks for your responses.

Ti

[Brett_Tabke]

1) file is only needed if you wish to block indexing of certain content.

2) www.example.com/ is your root directory. So www.example.com/robots.txt is where your bots.txt goes.

3) no they do not "crawl mysql". Only certain file types.

4) yes.