Forum Moderators: goodroi
I am a new member and quite new to optimisation in general.
My first question for this section is -
Do we need robot.txt commands? Do search engines such as the big 3 or 4 actually use them and follow them or am I best not adding anything at all about robots?
Look forward to receiving some help and I will no doubt be asking more questions soon as I try and get my head round all this optimisation stuff. Thanks again.
PlenusVita.
Without a robots.txt file at the root of your site, your just producing a 404 error every day to the bots, as they always look for it, even if it's not there.
[edited by: Asia_Expat at 5:20 am (utc) on Nov. 27, 2006]
Welcome to WebmasterWorld!
All the big engines follow robots.txt. Robots.txt is a good way to prevent the search engines from indexing pages that may have sensitive information like your admin pages, development area, your stats etc. Also you probably don't want the engines causing trouble in your programs & code directories like /cgi-bin/.
If your site is very simple and has nothing to block from the engines you do not need a robots.txt. Because the engines will always ask for a robots.txt file and in its absense they will index and follow everything. I would suggest using a robots.txt since most sites do have some area on their site they do not want the engines poking around.
ps make sure to use a robots.txt validator to ensure your robots.txt is doing what you want it to do
cheers,
goodroi
i guess it's ok to mention [robotstxt.org...] here as well...
Sorry for the delay in getting back to my own thread. Thanks very much for all of the information - very much appreciated.
I have done a couple of sites that have no sensitive information in them and also 1 large e-commerce site, so I will look into that url that has been suggested and do some reading and research - thanks again, no doubt I will see you guys in the forums soon as I have lots of questions.
Take care.
Plenusvita
