homepage Welcome to WebmasterWorld Guest from 54.197.110.151
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Pros and cons of having
a robots.txt file?
ideavirus

10+ Year Member



 
Msg#: 193 posted 8:47 am on Nov 11, 2003 (gmt 0)

hello,

Though, i run a few sites, i do not have robots.txt for any of my sites. not sure, why i never did bother to have one. But now, i wam considering it, if it benefits a website by having it.

Towarsd that end, what are the pros and cons of having a robots.txt file for a website.

Thank you for your time.
Cheers

 

Mohamed_E

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 193 posted 12:29 pm on Nov 11, 2003 (gmt 0)

The only real function of a robots.txt file is to keep robots out of part of your web site. If there is no area of the site which you want to keep unspidered you have no real need for one.

A very minor advantage of having an empty robots.txt file (which bans nothing) is that it will keep some "error" messages out of your logs. Every time a robot visits your site it asks for the robots.txt file. If there is none the robot will get a 404 error, which will show in your logs.

WebmasterFisherman



 
Msg#: 193 posted 5:32 pm on Nov 13, 2003 (gmt 0)

Hi,

I want to use robots.txt to keep robots out of a few pages on my site.
I do this to allocate more page rank to notional pages of the site not garbish like disclaimer, sitemap etc.

If I put those pages in the robots.txt but leave simple href links on the page will Google transfer pagerank to them anyway?

If there's strong suspicion it will, what do you think will be a better idea?

Javascript navigation to those pages only?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved