Welcome to WebmasterWorld Guest from 54.162.227.136

Forum Moderators: goodroi

Message Too Old, No Replies

Pros and cons of having

a robots.txt file?

     
8:47 am on Nov 11, 2003 (gmt 0)

10+ Year Member



hello,

Though, i run a few sites, i do not have robots.txt for any of my sites. not sure, why i never did bother to have one. But now, i wam considering it, if it benefits a website by having it.

Towarsd that end, what are the pros and cons of having a robots.txt file for a website.

Thank you for your time.
Cheers

12:29 pm on Nov 11, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The only real function of a robots.txt file is to keep robots out of part of your web site. If there is no area of the site which you want to keep unspidered you have no real need for one.

A very minor advantage of having an empty robots.txt file (which bans nothing) is that it will keep some "error" messages out of your logs. Every time a robot visits your site it asks for the robots.txt file. If there is none the robot will get a 404 error, which will show in your logs.

5:32 pm on Nov 13, 2003 (gmt 0)



Hi,

I want to use robots.txt to keep robots out of a few pages on my site.
I do this to allocate more page rank to notional pages of the site not garbish like disclaimer, sitemap etc.

If I put those pages in the robots.txt but leave simple href links on the page will Google transfer pagerank to them anyway?

If there's strong suspicion it will, what do you think will be a better idea?

Javascript navigation to those pages only?

 

Featured Threads

Hot Threads This Week

Hot Threads This Month