Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Pros and cons of having

a robots.txt file?

8:47 am on Nov 11, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 16, 2001
votes: 0


Though, i run a few sites, i do not have robots.txt for any of my sites. not sure, why i never did bother to have one. But now, i wam considering it, if it benefits a website by having it.

Towarsd that end, what are the pros and cons of having a robots.txt file for a website.

Thank you for your time.

12:29 pm on Nov 11, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 14, 2002
votes: 0

The only real function of a robots.txt file is to keep robots out of part of your web site. If there is no area of the site which you want to keep unspidered you have no real need for one.

A very minor advantage of having an empty robots.txt file (which bans nothing) is that it will keep some "error" messages out of your logs. Every time a robot visits your site it asks for the robots.txt file. If there is none the robot will get a 404 error, which will show in your logs.

5:32 pm on Nov 13, 2003 (gmt 0)

Junior Member

joined:Oct 17, 2003
votes: 0


I want to use robots.txt to keep robots out of a few pages on my site.
I do this to allocate more page rank to notional pages of the site not garbish like disclaimer, sitemap etc.

If I put those pages in the robots.txt but leave simple href links on the page will Google transfer pagerank to them anyway?

If there's strong suspicion it will, what do you think will be a better idea?

Javascript navigation to those pages only?