homepage Welcome to WebmasterWorld Guest from 50.17.86.12
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Why would I use a Robots.txt file?
BoneHeadicus

10+ Year Member



 
Msg#: 158 posted 10:15 pm on Jan 11, 2001 (gmt 0)

Is there a good set of rules to follow as far as keeping spidey off certain pages???
IP delivery, cgi, etc.

 

scberg

10+ Year Member



 
Msg#: 158 posted 4:20 am on Jan 12, 2001 (gmt 0)

What exactly is the robots.txt file used for?

--Shawn

skirril

10+ Year Member



 
Msg#: 158 posted 12:36 am on Jan 13, 2001 (gmt 0)

The robots.txt file (and the robots) META tag are used to
tell robots what they should and should not access.

Most robots follow the robots exclusion standard, which can be found at
[info.webcrawler.com...]

The idea of limiting a robots crawling is that you might not want all pages indexed (any inexed page can be an entrypoint to your site). You wouldn't want to start your visit at the feedback page, or perhaps you dont want robots to index your pdf docs which you keep in a certain dir. Or there might be a section that's password protected, eg.

Just ideas

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 158 posted 3:42 pm on Jan 13, 2001 (gmt 0)

Robots is good for blocking off high profile directories. If it specific pages you want to block, I'd use a meta tag No Index tag - far more effective. Engines (like Google) will check for a robots once a month, whereas the meta tag is read every page read.

BoneHeadicus

10+ Year Member



 
Msg#: 158 posted 5:04 pm on Jan 13, 2001 (gmt 0)

Thanks Brett,
High profile meaning what ?

Forgive me..sometimes I'm retarded.

grnidone



 
Msg#: 158 posted 11:58 pm on Jan 13, 2001 (gmt 0)

Bonehead,

We used a robots.txt file to keep the privacy policy pages from being the first listed in the search results when the site was spidered. It was embarrasing to type in the keyword for the site, and the first listing was "This our dry legal privacy policy". When people saw that page pop up, they wouldn't even click a nav link on the page to go somewhere else, they would just hit back and avoid the site all together.

*shrug* That's why WE used it.

-G

BoneHeadicus

10+ Year Member



 
Msg#: 158 posted 12:40 am on Jan 14, 2001 (gmt 0)

Thanx G-

I guess I better rename my IP delivery folder something other than "Super-Secret Search Engine Redirection Script" before I write that robots.txt file, huh?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved