Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Mr. Roboto

How do I set up a Robots.txt file?



4:41 pm on Jan 25, 2001 (gmt 0)

Inactive Member
Account Expired


I just need a tip on how to properly setup a Robots.txt file. I don't want to exclude any spiders, I just want to maybe exclude some directories. I basically am just trying out to see how a robots.txt file works for a new site I'm building and deciding if I should go back and add it to older ones I have.
Ok, so it should start like this:
because I'm not targeting any specific agent right?
then I thought I'd disallow something because I read that if you don't give it something to do when it reads your file that the spiders/robots can get confused and may ignore your site. So the next line would be:
Is this right? that's all I need to include?
I'm basically just trying to bait some spiders here into digging in and taking all they can with my site. I know that some do read it and some don't, that's no problem. I am just trying to cover all the bases with this. Trying to see how it works. I've got the meta robots tag in place and it seems maybe to be helping I'd just like to try this and see if it nudges my listing up a little.
so, am I on the right track? any other tips or tricks anyone can offer?
2:47 am on Jan 26, 2001 (gmt 0)


WebmasterWorld Administrator 10+ Year Member

joined:July 2, 2000
votes: 1

Hi Cave Toad,

Here's a couple of excellent resouces that you can use as reference. The first one spells out what to do(with examples towards the bottom) while the 2nd checks for syntax errors.



9:15 pm on Jan 29, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 62

CaveToad, you are on the right track. It might help to check some bigger sites for robots.txt to see how they are formated.



3:31 pm on Jan 30, 2001 (gmt 0)

Inactive Member
Account Expired


Yeah, I've got one built. It looks right
and passes the syntax test so I think I'm just about ready to take the plunge with it. The guinea pig site is just about ready.
Thanks for the help!
3:40 pm on Jan 30, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 25, 2000
votes: 0

Just an observation:

When I started using robots.txt file I began to get deeper crawls.


3:57 pm on Jan 31, 2001 (gmt 0)

Inactive Member
Account Expired


Well, test site #1 was submitted to AV yesterday, it's in the rest fine, it was stalling in AV so hence my interest in the robots file. Now all they have to do is crawl and I'll be ok. I'll post if I come across any revalations with this.