Pros and cons of having

a robots.txt file?

[ideavirus]

hello,

Though, i run a few sites, i do not have robots.txt for any of my sites. not sure, why i never did bother to have one. But now, i wam considering it, if it benefits a website by having it.

Towarsd that end, what are the pros and cons of having a robots.txt file for a website.

Thank you for your time.
Cheers

[Mohamed_E]

The only real function of a robots.txt file is to keep robots out of part of your web site. If there is no area of the site which you want to keep unspidered you have no real need for one.

A very minor advantage of having an empty robots.txt file (which bans nothing) is that it will keep some "error" messages out of your logs. Every time a robot visits your site it asks for the robots.txt file. If there is none the robot will get a 404 error, which will show in your logs.

[WebmasterFisherman]

Hi,

I want to use robots.txt to keep robots out of a few pages on my site.
I do this to allocate more page rank to notional pages of the site not garbish like disclaimer, sitemap etc.

If I put those pages in the robots.txt but leave simple href links on the page will Google transfer pagerank to them anyway?

If there's strong suspicion it will, what do you think will be a better idea?

Javascript navigation to those pages only?