homepage Welcome to WebmasterWorld Guest from 54.205.247.203
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robot.txt File
What is it's importance?
Enigmatic




msg:1525642
 10:26 pm on Feb 24, 2004 (gmt 0)

Can anyone tell me the importance of the robot.txt file? And is it robot.txt or robots.txt?

I e-mailed my web host asking where mine was located on the server and they told me they never use one. Should I be trying to make one for the SE's and where should I upload it? Thanks,

Enigmatic

 

closed




msg:1525643
 8:12 pm on Feb 25, 2004 (gmt 0)

robots.txt is usually used for restricting robots from indexing parts of a site.

pr0purgatory




msg:1525644
 12:50 pm on Mar 15, 2004 (gmt 0)

If you have an META tag instruction for robots, do you need a robots.txt?

Alternative Future




msg:1525645
 12:55 pm on Mar 15, 2004 (gmt 0)

>>If you have an META tag instruction for robots, do you need a robots.txt?

Most SE's will/should obey a robots.txt file while there is a lesser amount will read the META tag.

Am open for correction on this, but I think that is what i have came to understand.

-george

closed




msg:1525646
 5:38 pm on Mar 16, 2004 (gmt 0)

If you have an META tag instruction for robots, do you need a robots.txt?

It depends.

At the very least, you should put an empty robots.txt to cut down on 404s on your site.

Using META tags allows you to control how robots index or follow links on your pages.

Using robots.txt restricts access to files (text and binary), as well as directories.

MarkHutch




msg:1525647
 5:46 pm on Mar 16, 2004 (gmt 0)

I'm probably opening up a can of worms here, but here we go.

I have wondered for the past year if there may come a time when spiders will not visit sites without a robots.txt file. The reasons are the increasing amount of privacy laws around the world. With email it's opt in, with any information we gather on our sites it's a published privacy policy. In the future, new laws may require web spiders to get an opt in (robots.txt permission slip) before they visit and list pages. I'm just commenting on the way the wind is blowing these days. I'm sure many people disagree with me, but I don't think it hurts to tell all robots it's ok to spider their site. We've added this information to all our domains and sub domains over the past few month. I hope to hear some more comments on this.

User-agent: *
Disallow:

hannamyluv




msg:1525648
 8:52 pm on Mar 16, 2004 (gmt 0)

To tell you the truth, I don't think it will be an issue. It just isn't something the average user (aka politician) understands or even knows about. Heck, I think a survey of websites will show that 50% of webmasters don't know about them.

If nobody knows, nobody will make a law about it.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved