- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Robots.txt - Block all but Most Popular?
g1smd - 6:32 pm on Aug 26, 2010 (gmt 0)
Directives in the robots.txt file do not 'block' anything at all.
They are merely a polite request, which many rogue bots will ignore.
If you want to physically block stuff, you will need allow/deny directives or mod_rewrite code.
Thread source: http://www.webmasterworld.com/robots_txt/4192676.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com