Howdy, I just need a tip on how to properly setup a Robots.txt file. I don't want to exclude any spiders, I just want to maybe exclude some directories. I basically am just trying out to see how a robots.txt file works for a new site I'm building and deciding if I should go back and add it to older ones I have. Ok, so it should start like this: User-agent:* because I'm not targeting any specific agent right? then I thought I'd disallow something because I read that if you don't give it something to do when it reads your file that the spiders/robots can get confused and may ignore your site. So the next line would be: Disallow:/foldername/ Disallow:/foldername2/ Is this right? that's all I need to include? I'm basically just trying to bait some spiders here into digging in and taking all they can with my site. I know that some do read it and some don't, that's no problem. I am just trying to cover all the bases with this. Trying to see how it works. I've got the meta robots tag in place and it seems maybe to be helping I'd just like to try this and see if it nudges my listing up a little. so, am I on the right track? any other tips or tricks anyone can offer? thanks!
Well, test site #1 was submitted to AV yesterday, it's in the rest fine, it was stalling in AV so hence my interest in the robots file. Now all they have to do is crawl and I'll be ok. I'll post if I come across any revalations with this.