Howdy,
I just need a tip on how to properly setup a Robots.txt file. I don't want to exclude any spiders, I just want to maybe exclude some directories. I basically am just trying out to see how a robots.txt file works for a new site I'm building and deciding if I should go back and add it to older ones I have.
Ok, so it should start like this:
User-agent:*
because I'm not targeting any specific agent right?
then I thought I'd disallow something because I read that if you don't give it something to do when it reads your file that the spiders/robots can get confused and may ignore your site. So the next line would be:
Disallow:/foldername/
Disallow:/foldername2/
Is this right? that's all I need to include?
I'm basically just trying to bait some spiders here into digging in and taking all they can with my site. I know that some do read it and some don't, that's no problem. I am just trying to cover all the bases with this. Trying to see how it works. I've got the meta robots tag in place and it seems maybe to be helping I'd just like to try this and see if it nudges my listing up a little.
so, am I on the right track? any other tips or tricks anyone can offer?
thanks!
