homepage Welcome to WebmasterWorld Guest from 54.211.235.255
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Mr. Roboto
How do I set up a Robots.txt file?
CaveToad




msg:1527979
 4:41 pm on Jan 25, 2001 (gmt 0)

Howdy,
I just need a tip on how to properly setup a Robots.txt file. I don't want to exclude any spiders, I just want to maybe exclude some directories. I basically am just trying out to see how a robots.txt file works for a new site I'm building and deciding if I should go back and add it to older ones I have.
Ok, so it should start like this:
User-agent:*
because I'm not targeting any specific agent right?
then I thought I'd disallow something because I read that if you don't give it something to do when it reads your file that the spiders/robots can get confused and may ignore your site. So the next line would be:
Disallow:/foldername/
Disallow:/foldername2/
Is this right? that's all I need to include?
I'm basically just trying to bait some spiders here into digging in and taking all they can with my site. I know that some do read it and some don't, that's no problem. I am just trying to cover all the bases with this. Trying to see how it works. I've got the meta robots tag in place and it seems maybe to be helping I'd just like to try this and see if it nudges my listing up a little.
so, am I on the right track? any other tips or tricks anyone can offer?
thanks!

 

eljefe3




msg:1527980
 2:47 am on Jan 26, 2001 (gmt 0)

Hi Cave Toad,

Here's a couple of excellent resouces that you can use as reference. The first one spells out what to do(with examples towards the bottom) while the 2nd checks for syntax errors.

[info.webcrawler.com...]

[tardis.ed.ac.uk...]

Brett_Tabke




msg:1527981
 9:15 pm on Jan 29, 2001 (gmt 0)

CaveToad, you are on the right track. It might help to check some bigger sites for robots.txt to see how they are formated.

[google.com...]

CaveToad




msg:1527982
 3:31 pm on Jan 30, 2001 (gmt 0)

Thanks.
Yeah, I've got one built. It looks right
and passes the syntax test so I think I'm just about ready to take the plunge with it. The guinea pig site is just about ready.
Thanks for the help!

BoneHeadicus




msg:1527983
 3:40 pm on Jan 30, 2001 (gmt 0)

Just an observation:

When I started using robots.txt file I began to get deeper crawls.

CaveToad




msg:1527984
 3:57 pm on Jan 31, 2001 (gmt 0)

Well, test site #1 was submitted to AV yesterday, it's in the rest fine, it was stalling in AV so hence my interest in the robots file. Now all they have to do is crawl and I'll be ok. I'll post if I come across any revalations with this.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved